Reporting is Unavailable in UIs & API

Incident Report for Xandr

Postmortem

Incident Summary

From 15:00:29 UTC to 16:46:00 UTC on Thursday, July 9th 2020, clients were unable to run reporting via the Xandr UIs and API Report Service.

Scope of Impact

During the incident window, globally, customers were unable to run reporting.

Timeline (UTC)

2020-07-09 15:22:06: Incident Started: Users unable to run reporting

2020-07-09 15:24:00: Incident Reported: Issue was escalated to Engineering

2020-07-09 16:46:00: Incident Resolved

Cause Analysis

This incident occurred from a misconfiguration in the current reporting database version

Resolution Steps

Our engineers resolved the issue by pointing reporting away from the affected reporting cluster, to another cluster.

Next Steps

Initiate process to upgrade our reporting database version
Review steps to improve internal monitoring

Posted Jul 17, 2020 - 14:58 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jul 09, 2020 - 17:06 UTC

Identified

We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.

Posted Jul 09, 2020 - 16:40 UTC

Investigating

We are currently investigating the following issue:

Component(s): Reporting Console pages, Reporting
Impact(s):
- Latency, timeouts and errors for affected service
- Data unavailable through affected service(s)
Severity: Major Outage
Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jul 09, 2020 - 16:27 UTC