Incident Summary
From approximately 15:55 UTC on Saturday, January 25th to 17:45 UTC on Monday, January 27nd bidders, both external and internal, stopped receiving notifications on some transactions.

Scope of Impact
During the incident window, transaction notifications were not sent to bidders for 0.2% of traffic. As a result of this incident buyer's may see a lower number of transactions recorded and, as a result, discrepant transaction data with third party systems. Sellers were not impacted by this incident.

Timeline (UTC)
2020-01-25 15:55 UTC: Incident Started: Release deployed to four impression bus instances
2020-01-27 10:30 UTC: Incident Identified: Reporting reveals discrepant data
2020-01-27 17:26 UTC: Incident escalated to Engineering
2020-01-27 18:50 UTC: Release rolled back
2020-01-27 17:45 UTC: Incident Resolved

Cause Analysis
The root cause of the incident was due to a very limited impression bus release that cut off affected impressions buses from sending notifications to bidders.

Resolution Steps
The engineering team rolled back the impression bus release that caused the issue.

Next Steps

• Improve communication of impression bus releases to downstream applications
• Tighten alert threshold on discrepant data
• Implement new monitoring on streaming for earlier detection