Incident Summary

Between 2020-01-07 16:00 UTC and 2020-01-08 02:10 UTC, Prebid availability in LAX was impacted.

Scope of Impact

Between 2020-01-07 16:00 UTC and 2020-01-08 02:10 UTC bid requests that came through the Prebid Server experienced around a 75% failure rate in contacting bidding services, including Xandr’s DSP.

Timeline

2020-01-07 16:00: Incident Started: Combination of oddity with infrastructure settings and maintenance activity caused issue with availability

2020-01-07 17:24: Incident escalated to engineers

2020-01-07 19:31: Revert of release that started incident

2020-01-07 20:29: Second level of escalation begins which led to further errors causing issue

2020-01-08 02:10: Incident Resolved

2020-01-08 02:45: Incident closed and client notification sent out

Cause Analysis

The incident was created due to a bug triggered by a backend release. This bug caused connectivity issues which resulted in the failures for prebid requests.

Resolution Steps

Mitigation is occurring now and involves changing connectivity settings to greatly reduce this from happening in the future.

Next Steps

Engineering team to follow up and implement monitoring so this can be caught before application impact in the future.