Spike in Delivery on Line Items

Incident Summary

From approximately 21:15 UTC on Monday, Sep 11 to 05:00 UTC on Tuesday, Sep 12, 2023, users experienced an unexpected increase in spend, during the impact window

Incident Impact

• Nature of Impact(s): Increase in delivery across only a few selected line items
• Timeframe: ~7.75 Hours. 21:15 UTC on Monday, Sep 11 to 05:00 UTC on Tuesday, Sep 12, 2023.
• Scope: Global
• Components: Budget Delivery

Timeframe (UTC)

• 2023-09-11 21:15: Incident Started.
• 2023-09-12 00:25: Escalated to Engineer.
• 2023-09-12 00:25: Changes rolled back.
• 2023-09-12 05:00 Incident Resolved.

Cause Analysis

The issue was caused due to a planned release that was implemented to enhance memory resources to optimize the performance of relevant endpoints. However, during the implementation of this improvement, the endpoints failed to handle the concurrent requests resulting in traffic drops and inaccurate daily budget calculations. This caused users to experience an unexpected increase in spend, during the impact window.

Resolution Steps

The issue was resolved by reverting the changes to a stable version and by triggering the relevant jobs that were necessary to reprocess the data to update budgets for all the Line Items.

Follow-up Items

• To mitigate potential incidents and prevent unintended failures, engineers revisited and implemented additional unit and integration test coverages
• Inspected relevant alerts and metrics to add relevant threshold parameters.
• Created a staging instance that replicates production traffic, enabling us to minimize potential future disruptions while testing new features. This would ensure to avert occurrences in the future that would potentially result in causing a similar IM