From 2019-11-11 17:11 to 2019-11-12 16:40 UTC, our primary reporting table was missing data for the range 2019-09-23 00:00 UTC and 2019-09-29 00:00 UTC.
During the incident window, clients could not pull data for data hour between 2019-09-23 00:00 & 2019-09-29 00:00. No data was lost; clients who pulled data during the incident need to re-pull data ranging from 2019-09-23 00:00 to 2019-09-29 00:00
2019-11-11 15:10: Internal job containing incorrect configuration deployed.
2019-11-12 16:14: Incident escalated: Xandr Internal Product Support Team escalated issue to engineering.
2019-11-12 16:40: Incident resolved: missing data reverted from internal table.
A new internal job aimed to optimize disk storage was deployed on 2019-11-11 at 15:10 UTC. The job contained incorrect configuration that acted on the wrong data partitions post-optimization. All missing data was also stored on an internal table so no data was lost.
We reverted the original data from our internal table, correcting the issue.
Maintain 24 hour gaps between deployments for different datacenters; this gap will help ensure integrity of client-facing data in our primary reporting table.
The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:
Data hours 2019-09-23 00:00:00 to 2019-09-29 23:00:00 UTC were unavailable from from approximately 2019-11-11 17:11:01 to 2019-11-12 16:40 UTC. No data was lost. We apologize for the inconvenience, and thank you for your continued support.