Log Level Data and Network Analytics reporting is delayed
Incident Report for Xandr
Postmortem

Incident Summary
From approximately 18:31 UTC Wednesday, June 10th, 2020 to 13:41 UTC Thursday, June 11th, 2020 a system upgrade caused delays in Log Level Data and Network Analytics reporting.

Scope of Impact
During the incident window, clients experienced longer than normal reporting delays for Log Level Data and Network Analytics reports.

Timeline (UTC)
2020-06-10 18:31 UTC: Incident Started: System upgrade started
2020-06-10 22:55 UTC: Incident Reported: Log Level Data and Network Analytics delays were reported
2020-06-11 13:41 UTC: Incident Resolved: Reporting delays fell under threshold

Cause Analysis
The root cause of the incident was a system upgrade that caused delays in the execution of jobs.

Resolution Steps
Our engineers resolved the issue by cleaning up the queue and rolling back changes.

Next Steps
• Improve speed of issue detection
• Streamline the recovery process

Posted Jun 15, 2020 - 21:56 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 11, 2020 - 13:50 UTC
Monitoring

We have identified and patched the following issue:

  • Component(s): Log Level Data, Analytics reports
  • Impact(s):
    • Reporting data is delayed
  • Severity: Major Outage
  • Datacenter(s): Global

We are monitoring our systems closely, and will provide an update as soon as the issue has been fully resolved.

Posted Jun 10, 2020 - 22:56 UTC