Reporting and Possibly LLD Delayed
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 11:00 PM UTC on Friday, June 24th to 2:29 UTC on Saturday, June 25th, a hardware issue caused a delay of up to one hour beyond SLA for the Standard and Network Analytics feeds.

Incident Impact

Nature of Impact: Data delivery was delayed for the Standard feed and Network Analytics reports 

Timeframe: 11:00 PM on Friday, June 24th to 2:29 UTC on Saturday, June 25th

Customers Impacted: Monetize, Invest, and Curate users

Scope: Global

Magnitude: Customers using the Standard feed and Network Analytics reporting

Timeline (UTC)

2022-06-24 23:00: Xandr teams identified and escalated the delay.

2022-06-25 01:21: Xandr teams decommissioned the affected hardware and restarted the database.

2022-06-25 02:29: Data delivery was restored to SLA levels.

Cause Analysis

A hardware issue with one of the servers slowed performance.

Resolution Steps

Xandr teams removed the affected server and restarted the delayed operations.

Next Steps

Xandr teams will replace the affected hardware and enhance alerting procedures to expedite recovery.

Posted Jun 28, 2022 - 20:45 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 25, 2022 - 08:33 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Log Level Data, Analytics reports
  • Impact(s):
    • Stale reporting data
  • Severity: Partially Degraded
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jun 25, 2022 - 01:21 UTC