Bidder Notification Loss
Incident Report for Xandr

Incident Summary
From approximately 15:55 UTC on Saturday, January 25th to 17:45 UTC on Monday, January 27nd bidders, both external and internal, stopped receiving notifications on some transactions.

Scope of Impact
During the incident window, transaction notifications were not sent to bidders for 0.2% of traffic. As a result of this incident buyer's may see a lower number of transactions recorded and, as a result, discrepant transaction data with third party systems. Sellers were not impacted by this incident.

Timeline (UTC)
2020-01-25 15:55 UTC: Incident Started: Release deployed to four impression bus instances
2020-01-27 10:30 UTC: Incident Identified: Reporting reveals discrepant data
2020-01-27 17:26 UTC: Incident escalated to Engineering
2020-01-27 18:50 UTC: Release rolled back
2020-01-27 17:45 UTC: Incident Resolved

Cause Analysis
The root cause of the incident was due to a very limited impression bus release that cut off affected impressions buses from sending notifications to bidders.

Resolution Steps
The engineering team rolled back the impression bus release that caused the issue.

Next Steps

  • Improve communication of impression bus releases to downstream applications
  • Tighten alert threshold on discrepant data
  • Implement new monitoring on streaming for earlier detection
Posted Feb 06, 2020 - 22:36 UTC


The following incident has been identified and fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Reporting
  • Impact(s):
    • Some transactional data loss between 16 UTC 01/25 and 18 UTC 01/27 resulting in some incomplete data
  • Severity: Major Outage
  • Datacenter(s): NYM2, AMS1

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jan 27, 2020 - 20:02 UTC