Data missing for some deals in Deal Metrics UI, hour 18 UTC data missing from certain reports
Incident Report for Xandr
Postmortem

Incident Summary
From approximately 2020-07-15 16:00 UTC to 2020-07-16 22:15 UTC, failure of a storage system in NYM2 datacenter caused corrupt data to enter our data streams which caused some deal related reports and UI screens fail to load properly and show incomplete data.

Scope of Impact
During the incident window, partner center screen was not loading depending on the time window selected and data for UTC hours 16 and 17 was missing from certain buyer/seller deal reports.

Timeline (UTC)
2020-07-15 16:00: Incident Started - corrupted data caused deal metrics API to error out

2020-07-16 02:10: Incident Reported - escalated to engineering team

2020-07-16 22:15: Incident Resolved: corrupted data was purged and all systems were back online

Cause Analysis
Failure of a storage system in NYM2 caused corrupted data in some deal-metrics related data streams.

Resolution Steps
Deal data was cut over to LAX1 datacenter which did not have the same fault in storage system and after purging bad data, records were copied from LAX1 to NYM2.

Next Steps

Improve detection, monitoring and alerts for prevent bad data from entering the data pipeline.

Posted Aug 07, 2020 - 07:04 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jul 17, 2020 - 14:59 UTC
Identified

We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.

Posted Jul 16, 2020 - 17:30 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Partner/Deal Console pages
  • Impact(s):
    • Data missing from Deal Metrics UI for some deals, hour 18 UTC missing from Seller/Buyer Deal Metrics
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jul 16, 2020 - 02:28 UTC