Incident Summary

From approximately 20:04 UTC on Monday, November 15, 2021 to 00:30 UTC on Tuesday, November 16, 2021, data updates to the platform were delayed.

Scope of Impact

During the incident window, some customers experienced difficulties editing objects (such as creatives, Line Items, and Programmable Splits) and other client updates made via API/UI were saved but did not take immediate affect during ad serving as expected. Database changes made during that period would slowly start to take effect during 00:30 UTC to 1:22 UTC, after which point database changes would propagate to ad server at regular intervals.

Timeline (UTC)

2021-11-15 20:04 UTC  - Data age for console database started to rise

2021-11-15 20:26 UTC  - Data team alerted and started investigating

2021-11-15 20:50 UTC  - Issue Escalated to Engineering for further investigation

2021-11-15 21:25 UTC  - Incident Ticket Created: Increased data age and delays to data updates

2021-11-15 22:53 UTC  - Issue Identified: App was stuck while waiting for query results from console database follower

2021-11-16 00:23 UTC  - Engineers updated stats and reran analyze table to fix query index

2021-11-16 00:30 UTC  - App started to catch up on data updates

2021-11-16 01:22 UTC  - Incident Resolved: App finished data updates and engineers sampled data

Cause Analysis

Some internal servers which process changed data from the Console database got stuck while waiting for query results. Due to non-progression of that application, data age started to increase for applications like Impbus, Bidder, etc.‌

Resolution Steps

Our engineering team resolved the issue by updating configuration to increase database table sampling for execution plan to utilize the ideal query index and resume timely processing.‌

Next Steps

• Analyze index statistics and plan to monitor/alert/prevent the issue in future for optimal performance.