From approximately 15:29 UTC on Monday, October 21st to 14:36 UTC on Thursday, October 24th, there was an issue with creative distribution that caused some LIs to stop spending.
During the incident window, creatives without a distribution weight under line items with manually weighted or CTR-optimized distribution types did not serve.
2019-10-21 15:29: Bidder release begins
2019-10-21 16:48: First line items impacted
2019-10-24 08:38: Incident escalated to engineering
2019-10-24 14:36: Partial roll back to reduce the impact of the issue
2019-10-24 16:06: Final fix released
There are many line items that rely on the default value of the creatives distribution weights. On Monday 2019-10-21 at 15:29 a release changed the default distribution weight values from 10 to 0 which caused these creatives to stop delivering.
The system was rolled back to a version before the change to the default values was introduced.
Add alerting to identify the issue quicker
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. In the meantime, making any changes to the weights of the creatives and saving the LI will unblock spend. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.