Delays in Daily Budget Update on Lifetime pacing ALI
Incident Report for Xandr
Postmortem

Incident Summary

On Thursday, February 18 2021, a new version of the Budget Distribution Service was released. This release exposed a bug that caused delayed and missing daily budget for clients that have lifetime_pacing enabled on their ALI's.

Scope of Impact

There are two buckets of impact:

ALI's with lifetime pacing enabled that were in flight after the release. Due to the delayed update to the daily budgets, the spend curve for the remainder of the flight may not be ideal. If daily budget was ahead of schedule, there may be less spend in the remaining days of the flight. If the daily budget was behind schedule there will be more spend than expected in the remaining days of the flight.

ALI's with lifetime pacing enabled that ended during the release - daily budget would stay at the last daily budget that need to be updated. There's a chance of underspend or overspend but, ultimately, the lifetime budget should've been enforced.

Timeline (UTC)

2021-02-18 10:49: New version of Budget Distribution Service released
2021-02-22 13:41:00: Engineer notified of Internal Server Error on graphs on budget overview
2021-02-22 16:22:00: Incident Resolved: Budget Distribution rolled back to old version
2021-02-22 17:14:00: Bug fixed and merged.

Cause Analysis

This incident was caused by a refactor of daily budget calculation from Budget Distribution Service where we broke down into different calculation. When calculating project_spend for the current date, it doesn't account for when there's no previous day flight information.

Resolution Steps

Budget Distribution Service was rolled back and fix was made to cover the missing case.

Next Steps

Work on meaningful monitory metrics and test coverage for all edge cases.

Posted Mar 05, 2021 - 15:14 UTC

Resolved

The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Bidding
  • Impact(s):
    • Some objects may spend under budgets
    • Some objects may spend over budgets
  • Severity: Minor Outage
  • Datacenter(s): Global

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Feb 24, 2021 - 15:31 UTC