Incident Summary

From approximately 16:47 UTC on Thursday, June 2nd, 2022 to 23:55 UTC on Friday, June 3rd, 2022, line item pacing was impacted when the primary data was unavailable and a third-party log aggregator caused an issue with fallback data.

Incident Impact

Nature of Impact: Line item underspend or overspend occurred, and Analytics tab data was intermittently unavailable. 

Timeframe: 16:47 UTC on Thursday, June 2nd, 2022 to 23:55 UTC on Friday, June 3rd, 2022

Customers Impacted: All DSP customers

Scope: Global

Magnitude: All customers

Timeline (UTC)

2022-06-02 15:58: Xandr teams deployed a repair to a third-party log aggregator. 

2022-06-03 04:45: Budget pacing data stopped being published.

2022-06-03 11:47:00: Xandr teams rolled back the change.

2022-06-03 23:55: Budget pacing history resumed publishing as expected.

Cause Analysis

An issue with budget pacing data records was compounded when the third-party log aggregator that handles fallback data failed, resulting in missing data. The database itself was decommissioned, so the specific root cause could not be determined.

Resolution Steps

A database change suspected to be the root cause was reverted. 

Next Steps

Xandr teams will enhance monitoring and procedures, and will continue to investigate to ensure greater database resilience.