Reporting Data Delays
Incident Report for Xandr
Postmortem

Incident Summary
Missing up to date data in some reports - reporting data was available but only up to a particular point in the past. That particular point for each report was different.

Scope of Impact
During 2020-09-17 to 2020-09-21, the following reports contained contiguous, yet not up to date data: Platform Seller Analytics, Prebid Analytics, YX Report, Platform Video Analytics, Video Analytics, Mobile Carrier Analytics, KV Analytics, Platform Domain Detection, Platform Inventory Analytics, Client error reports.

Timeline (UTC)
2020-09-17 15:00 : Incident Started.
2020-09-18 07:42 : Incident was reported. initially seen via “Seller Bid Error Report” missing data.
2020-09-18 14:32 : Issue resolved.
2020-09-18 13:59 : Our internal team decided to reopen the issue, as a part of a broader database fix.
2020-09-21 16:00 : Issue resolved.

Cause Analysis
Some of the data syncing jobs failed, which occasionally occur, and no alert was generated as expected. This caused a delay in the issue resolution.

Resolution Steps
The jobs were re-run to restore the missing data.

Next Steps
We are working on an improved model for dealing with operational support issues.

Posted Oct 20, 2020 - 15:32 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Sep 18, 2020 - 18:20 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Reporting
  • Impact(s):
    • Stale reporting data for some reports
  • Severity: Minor Outage
  • Datacenter(s): LAX1

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Sep 18, 2020 - 16:49 UTC