Reporting and Log Level Data Delays

Incident Report for Xandr

Postmortem

Incident Summary

From approximately 18:21 to 22:15 UTC on Tuesday, July 27, 2020 reporting data was delayed due to duplicate auction IDs being used.

Scope of Impact

During the incident window, clients experienced longer than normal reporting delays for Network Analytics reports.

Timeline (UTC)

2020-07-27 18:21: Incident started
2020-07-27 21:01: Incident reported
2020-07-27 19:30: Fix deployed and data reprocessing initiated
2020-07-27 10:30: Data verification
2020-07-27 22:15: Incident resolved: Fresh data available to customers

Cause Analysis

Duplicate auction IDs were accidentally used in live auction triggering internal data warnings. These flags prevented the data from being released to reporting until resolution, resulting in the six hour reporting SLA window being exceeded.

Resolution Steps

Engineering released a fix and reprocessed the affected data.

Next Steps

Ensure duplicated records are not passed through in the future
Identify additional detection opportunities further up the data stream

Posted Aug 13, 2020 - 20:14 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jul 28, 2020 - 00:31 UTC

Identified

We have identified the following issue:

Component(s): Log Level Data, Analytics reports
Impact(s):
- Stale reporting data
Severity: Partially Degraded
Datacenter(s): Global

Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.

Posted Jul 27, 2020 - 21:28 UTC