Incident Summary

From approximately 19:09 UTC on Thursday, May 25 to 22:44 UTC on Thursday, May 25, 2023, users experienced a significant drop in auction times and an increase in DV360 bidder timeouts in the NYM2 datacenter.

Incident Impact

• Nature of Impact(s): Drop in DV360 demand and increase in auction times.
• Timeframe: ~3.58 Hours. 19:09 UTC on Thursday, May 25 to 22:44 UTC on Thursday, May 25, 2023.
• Scope: North America – East Coast (NYM2)
• Components:
• Ad Serving
• Bidding

Timeline (UTC)

• 2023-05-25 19:09: Incident Started. Issue was attributed to our third party partner.
• 2023-05-25 19:30: Protocol session restored.
• 2023-05-25 21:08: IM Ticket Created.
• 2023-05-25 22:41: Third party partner was engaged for further troubleshooting.
• 2023-05-25 22:44: Incident Resolved. Issue resolved by itself.
• 2023-05-26 03:00: IM Ticket Closed.

Cause Analysis

The issue was caused due to a planned maintenance activity that was carried out by our third party partner, leading to the temporary shutdown of the circuit link. This resulted in users experiencing significant timeouts when using the third party partner​​​​​​​'s DSP in the NYM2 datacenter.

Resolution Steps

The issue resolved on its own, once the maintenance activity was completed by our third party partner​​​​​​​. As a result, the timeouts gradually decreased and the affected circuit link was completely restored.

Follow Up Items

Engineering team has been performing an internal post mortem review which also involves a follow-up on the procurement of new circuits. This would ensure to avert similar occurrences in the future that would potentially result in causing a similar IM.