Partially delayed Analytics Report data

Incident Report for Xandr

Postmortem

Incident Summary

Xandr reporting data delayed 14 hours between September 1st 06:33 PM UTC to September 2nd 01:41 PM UTC, and reporting was delayed until the reporting data delaying caught up.

Scope of Impact

Clients would have experienced 14 hours reporting delay until reporting caught up.

Timeline ( in UTC )

2021-09-01 06:33 PM - Analytics reports start delaying
2021-09-02 07:22 AM - Incident is reported 2
021-09-02 12:30 PM - Indident is escalated internally
2021-09-02 01:41 PM - Incident resolved and reports back within SLA

Cause Analysis

New version of internal service was released on September 1st. This program changed configuration incorrectly and caused internal traffic to be unable to reach this service from other services.

Resolution Steps

Our Engineer fixed incorrect configuration.

Next Steps

Added additional monitoring for this service as prevention

Posted Oct 13, 2021 - 00:38 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Sep 03, 2021 - 09:35 UTC

Investigating

We are currently investigating the following issue:

Component(s): API, Reporting
Impact(s):
- Stale reporting data
- Latency, timeouts and errors in API
- Potentially some data incomplete or incorrect until reprocessed (please repull data as necessary)
Severity: Partially Degraded
Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Sep 02, 2021 - 11:06 UTC