Bidder Issue Causing Missing Impressions
Incident Report for Xandr

Incident Summary
From approximately 16:48 UTC to 23:21 UTC on Monday, October 21st, a Console Bidder release caused a small percentage of missing bid rows.

Scope of Impact
During the incident window, some bid rows were dropped, resulting in a discrepancy between Analytics and Billing reports. Some buyers will not be charged for a small number of ads. Sellers will still be compensated for these transacted impressions by Xandr.

Timeline (UTC)
2019-10-21 15:29: Bidder release begins
2019-10-21 16:48: Incident Started
2019-10-21 23:21: Incident Resolved: Bidder release ended
2019-10-22 14:01: Escalated to Engineering

Cause Analysis
A rolling release of a patch to the system that disseminates object changes to our ad server resulted in temporarily mismatched in-memory currency rate snapshots across versions of Console Bidder. Bid messages that referenced non-existent objects were dropped.

Resolution Steps
The bidder release was completed, resulting in uniform currency snapshots across all Console Bidder instances.

Next Steps
* Prevent bid rows from being dropped
* Add alerting to improve detection speed

Posted Oct 29, 2019 - 14:07 UTC


We are aware of an issue that caused a small percentage of missing bid rows from 2019-10-21 16 to 21 UTC. Our team is investigating available follow-up actions, which will be detailed in the post mortem. Please see below details:

  • Component(s): Bidding, Billing reports, Analytics reports
  • Impact(s):
    • Data unavailable through affected service(s)
  • Severity: Partially Degraded
  • Datacenter(s): Global

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Oct 22, 2019 - 14:34 UTC