Delayed delivery of reports and log level data for 2020-05-26 21:00 UTC
Incident Report for Xandr

Incident Summary:

At approximately 20:20 UTC on May 26, 2020, emergency server maintenance in the NYM2 datacenter caused a backup in log processing which led to delayed log level data feeds.

Scope of Impact:

Customers experienced delayed delivery of standard feed and engineered features feed for hours 2020-05-26 21:00 and 22:00 UTC.

Timeline (UTC):

2020-05-26 21:45: First alert of delayed data
2020-05-26 22:05: Backup in other servers identified
2020-05-26 23:35: Incident ticket created
2020-05-26 23:37: Begin installation of extra servers
2020-05-27 00:15: Transfer of backlog data completed
2020-05-27 03:59: Extra servers brought online
2020-05-27 06:34: Incident resolved

Cause Analysis:

The NYM2 data center had been operating with fewer machines than normal for maintenance when an additional server failed during peak hours.

Resolution Steps:

Our engineers resolved the issue by adding permanent extra capacity in the NYM2 data center.

Next Steps:

  • Add more capacity
  • Modify process and timing of server maintenance
Posted Jun 03, 2020 - 03:31 UTC


The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted May 27, 2020 - 06:34 UTC

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted May 27, 2020 - 03:12 UTC

We have identified the following issue:

  • Component(s): Log Level Data, Analytics reports
  • Impact(s):
    • Some data incomplete or incorrect until reprocessed (please repull data as necessary)
    • Expect delayed delivery of reports and log-level data for for 2020-05-26 21:00 UTC time
  • Severity: Partially Degraded
  • Datacenter(s): Global

Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.

Posted May 27, 2020 - 00:36 UTC