Missing Data in Analytics Reports
Incident Report for Xandr
Postmortem

Incident Summary

From 2019-11-11 17:11 to 2019-11-12 16:40 UTC, our primary reporting table was missing data for the range 2019-09-23 00:00 UTC and 2019-09-29 00:00 UTC.

Scope of Impact

During the incident window, clients could not pull data for data hour between 2019-09-23 00:00 & 2019-09-29 00:00. No data was lost; clients who pulled data during the incident need to re-pull data ranging from 2019-09-23 00:00 to 2019-09-29 00:00

Timeline (UTC)

2019-11-11 15:10: Internal job containing incorrect configuration deployed.

2019-11-12 16:14: Incident escalated: Xandr Internal Product Support Team escalated issue to engineering.

2019-11-12 16:40: Incident resolved: missing data reverted from internal table.

Cause Analysis

A new internal job aimed to optimize disk storage was deployed on 2019-11-11 at 15:10 UTC. The job contained incorrect configuration that acted on the wrong data partitions post-optimization. All missing data was also stored on an internal table so no data was lost.

Resolution Steps

We reverted the original data from our internal table, correcting the issue.

Next Steps

Maintain 24 hour gaps between deployments for different datacenters; this gap will help ensure integrity of client-facing data in our primary reporting table.

Posted Nov 19, 2019 - 17:29 UTC

Resolved

The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Analytics reports
  • Impact(s):
    • Some data incomplete or incorrect until reprocessed (please repull data as necessary)
  • Severity: Major Outage
  • Datacenter(s): Global

Data hours 2019-09-23 00:00:00 to 2019-09-29 23:00:00 UTC were unavailable from from approximately 2019-11-11 17:11:01 to 2019-11-12 16:40 UTC. No data was lost. We apologize for the inconvenience, and thank you for your continued support.

Posted Nov 12, 2019 - 17:17 UTC