Deal reporting, data delay

Incident Report for Xandr

Postmortem

Incident Summary
From approximately 20:30 UTC to 22:41 UTC, on Tuesday, June 22, 2021 deals-metrics reporting was impacted for about two hours.

Scope of Impact
During the incident some clients may have seen stale data reporting in Seller & buyer deal metrics and UI screens.Timeline (UTC)
2021-06-22 20:49: Incident Started: Scheduled change made to host caused a failure
2021-06-22 20:52: Internal detection by engineer of negative affect of change
2021-06-22 21:45: Incident escalated to Engineers
2021-06-22 22:04: Testing of fix began
2021-06-22 22:30: Client communication sent
2021-06-22 22:41: Incident Resolved and client notification sent out

Cause Analysis
A configuration update on Ubuntu host caused an unintentional networking failure.

Resolution Steps
Reverting the change allowed clusters to clear up the networking issue and allowed pods to restart and communicate normally.

Next Steps
Improve testing and release procedures

Posted Jun 28, 2021 - 15:15 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 22, 2021 - 22:58 UTC

Investigating

We are currently investigating the following issue:

Component(s): Reporting pages, Deal reporting, data delays
Impact(s):
- Seller & buyer deal metrics reports are affected. UI screens will display stale deal metrics data.
Severity: Minor Outage
Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jun 22, 2021 - 22:29 UTC