Incident Summary
From approximately 20:00 UTC on Tuesday, June 28th, 2022 to 04:34 UTC on Wednesday, June 29th, 2022, a degraded third-party hardware component impacted some servers, causing the delivery SLA for log-level data to be breached by varying amounts between 11 minutes and the maximum breach of four hours and 56 minutes.
Incident Impact
Nature of Impact: Data delivery was delayed beyond SLA.
Timeframe: 2022-6-28 at 20:00 UTC to 2022-6-29 at 4:34 UTC
Customers Impacted: Users of the Log-Level Data feed
Scope: Global
Magnitude: All customers
Timeline (UTC)
2022-06-28 16:04: Xandr teams were alerted to a potential hardware issue and began monitoring.
2022-06-28 6:28: Xandr teams isolated the issue to a specific drive and initiated troubleshooting.
2022-06-28 17:22: Xandr teams began remediating the issue.
2022-06-29 13:48: Data delivery was restored to SLA levels.
Cause Analysis
A third-party hardware component experienced data corruption and affected servers.
Resolution Steps
Engineers restarted the affected servers and replaced the hardware that caused the incident.
Next Steps
Xandr teams will continue to monitor hardware.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the following issue:
Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.