Between approximately 18:35-19:25 UTC and 19:50-20:10 UTC on September 8, 2020 view prediction measurements stopped delivering to the NYM2 datacenter.
The Invest DSP stopped transacting approximately 25% of auctions in the NYM2 datacenter since view prediction measurements were required for some auction targeting objects. As a result, underdelivery may have occurred.
2020-09-08 18:35: Incident began
2020-09-08 18:40: Issue flagged and investigation began
2020-09-08 19:25: Affected components within datacenter restarted
2020-09-08 19:30: Incident cleared
2020-09-08 19:50: Incident began recurring
2020-09-08 20:02: Incident flagged again
2020-09-08 20:08: Affected components within datacenter restarted
2020-09-08 20:10: Incident resolved
The root cause of the incident was determined to be insufficient capacity for the affected component which was exacerbated by inadequate communication between that component and the NYM2 datacenter.
Our engineers resolved the issue by increasing capacity available for the affected components.
Increase communication capacity between the affected components and the NYM2 datacenter by updating the mechanism in charge of this. Review capacity planning framework to better consider and adjust to traffic demands.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have identified and patched the following issue:
We are monitoring our systems closely, and will provide an update as soon as the issue has been fully resolved.