Incident Summary
From approximately 16:48 UTC to 23:21 UTC on Monday, October 21st, a Console Bidder release caused a small percentage of missing bid rows.

Scope of Impact
During the incident window, some bid rows were dropped, resulting in a discrepancy between Analytics and Billing reports. Some buyers will not be charged for a small number of ads. Sellers will still be compensated for these transacted impressions by Xandr.

Timeline (UTC)
2019-10-21 15:29: Bidder release begins
2019-10-21 16:48: Incident Started
2019-10-21 23:21: Incident Resolved: Bidder release ended
2019-10-22 14:01: Escalated to Engineering

Cause Analysis
A rolling release of a patch to the system that disseminates object changes to our ad server resulted in temporarily mismatched in-memory currency rate snapshots across versions of Console Bidder. Bid messages that referenced non-existent objects were dropped.

Resolution Steps
The bidder release was completed, resulting in uniform currency snapshots across all Console Bidder instances.

Next Steps
* Prevent bid rows from being dropped
* Add alerting to improve detection speed