Incident Summary

A routine update meant to restart a single server restarted multiple servers, resulting in blank ads being served from approximately 21:20 UTC to 21:40 UTC on Thursday, June 23, 2022.

Incident Impact

Nature of Impact: Creatives couldn't be rendered and blank ads were served.

Timeframe:  21:20 UTC to 21:40 UTC Thursday June 23, 2022

Customers Impacted: All customers

Scope: Global

Magnitude: All customers

Timeline (UTC)

2022-06-23 21:10: A routine update was initiated, causing a cascade of server restarts.
2020-06-23 21:20: Creative rendering became unavailable.2
020-06-23 20:40: The affected servers were restarted, restoring normal performance.
2020-06-23 21:43: Xandr teams identified the cause of the issue and verified it was no longer affecting customers.
2020-06-23 22:00: Xandr teams terminated the update that initially caused the problem. 

Cause Analysis
When a server update was rolled out, a minor misconfiguration caused all servers to restart.

Resolution Steps
Xandr teams terminated the update that caused the issue so it could be implemented correctly. 

Next Steps
Xandr teams will enhance change management processes to prevent future incidents.