Partially blocked delivery in Line Items with manually weighted creative distribution
Incident Report for Xandr

Incident Summary

From approximately 15:29 UTC on Monday, October 21st to 14:36 UTC on Thursday, October 24th, there was an issue with creative distribution that caused some LIs to stop spending.

Scope of Impact

During the incident window, creatives without a distribution weight under line items with manually weighted or CTR-optimized distribution types did not serve.

Timeline (UTC)

2019-10-21 15:29: Bidder release begins

2019-10-21 16:48: First line items impacted

2019-10-24 08:38: Incident escalated to engineering

2019-10-24 14:36: Partial roll back to reduce the impact of the issue

2019-10-24 16:06: Final fix released

Cause Analysis

There are many line items that rely on the default value of the creatives distribution weights. On Monday 2019-10-21 at 15:29 a release changed the default distribution weight values from 10 to 0 which caused these creatives to stop delivering.

Resolution Steps

The system was rolled back to a version before the change to the default values was introduced.

Next Steps

Add alerting to identify the issue quicker

Posted Oct 29, 2019 - 10:10 UTC


The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Oct 25, 2019 - 01:38 UTC

We have identified the cause of the issue, and our engineers are actively working towards a resolution. In the meantime, making any changes to the weights of the creatives and saving the LI will unblock spend. Thank you for your patience.

Posted Oct 24, 2019 - 11:48 UTC

We are currently investigating the following issue:

  • Component(s): Bidding
  • Impact(s):
    • Updates not reflected for some objects, especially buy-side
    • Increase in defaults and PSAs for Console customers
    • Some objects may spend under budgets
  • Severity: Partially Degraded
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Oct 24, 2019 - 10:50 UTC