Incident Summary

From approximately 17:52 on Tuesday, June 21, 2022 to 15:27 on Friday, June 24, a release meant to deprecate a legacy database table and improve our budgeting and delivery system caused some guaranteed delivery line items (GDLIs), standard line items, and augmented line Items using roadblocks or guaranteed outcomes not to deliver. 

Incident Impact

Nature of Impact: A subset of customers saw line items that still had budget, but did not continue to deliver. 

Timeframe: 17:52 on Tuesday, June 21, 2022 to 15:27 on Friday, June 24,

Customers Impacted: Invest and Monetize

Scope: Global

Magnitude: Some customers

Timeline (UTC)

2022-06-21 17:52: A change to a budgeting and delivery data job was released, causing some objects to stop spending.

2022-06-24 12:19: Xandr teams became aware of the issue and escalated it.

2020-06-24 12:40: Xandr teams stopped the initial job and re-ran it using the previous configuration.

2022-06-24 5:27: The reverted job completed, and the affected line items began spending as normal.

Cause Analysis

A change to a budgeting and delivery data job generated erroneous data indicating that certain line items had exhausted their lifetime budget, when they had not. 

Resolution Steps

Xandr teams reverted and re-ran the initial job to provide budget pacing with the correct values. 

Next Steps

Xandr teams will enhance change management processes to improve monitoring around lifetime delivery data.