Incident Summary

From approximately 15:20 UTC on Wednesday, April 14 to 05:02:08 UTC on Thursday, April 15, 2021, newly created/activated budgets were not being activated and some clients may have experienced non-delivery of Insertion Order and Line Item objects.

Scope of Impact

During the incident window, some clients may have seen some Insertion Order and Line Item objects fail to activate and, therefore, not deliver until the incident was resolved.

Timeline (UTC)

2021-04-14 15:20:00: Incident started
2021-04-15 05:02:08: Incident resolved

Cause Analysis

The incident was caused by a failure within the budget data pipeline which determined the data that was considered for budget state changes. This in turn caused delays to the budget state data, resulting in some buy-side objects failing to activate.

Resolution Steps

The issue resolved on its own once the internal failure was resolved, which initiated an update in the budget data pipeline. This allowed for the activation of the affected buy-side objects.

Next Steps

• Remove reliance on legacy budget data pipelines.
  
• Improve detection of budget data defects.
  
• Accelerate time to incident escalation.