Deal Metric Reporting Down in NYM2
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 15:30 UTC on September 9 to 17:30 UTC on September 9, 2020, some deal related reports and UI screens were failing to load properly or showing incomplete data.

Scope of Impact

Some partner center deal screens did not have any data for the 10 min and 1 hour views and the deal metrics buyers/sellers reports had data processing delay for 3 hours between 15:00 and 17:00 UTC on September 9, 2020.

Timeline (UTC)

2020-09-09 15:30: Incident began
2020-09-09 16:00: Issue flagged and affected data stream temporarily transferred to alternate datacenter
2020-09-09 17:30: Issue resolved and data stream returned to home datacenter

Cause Analysis

The root cause of the incident was a scheduled update change.

Resolution Steps

Our engineers resolved the issue by reverting the change and restarting the affected system.

Next Steps

Create an incident protocol for the Deal Metrics data stream similar to that of other systems.

Posted Sep 16, 2020 - 18:24 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Sep 09, 2020 - 18:52 UTC
Identified

We have identified the following issue:

  • Component(s): Partner/Deal Console pages
  • Impact(s):
    • Starting at 15:30 UTC no reporting data in deal metrics reporting and UI
  • Severity: Minor Outage
  • Datacenter(s): NYM2

Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.

Posted Sep 09, 2020 - 16:51 UTC