From approximately 15:10 to 19:10 UTC on July 15, 2020, a storage server in our NYM2 data center became unresponsive.
During the incident window, customers were not able to utilize the creative batch uploader service. Upon further investigation, this incident also resulted in customers being unable to utilize the batch segment service.
2020-07-15 15:10: Incident began, received alert that server was unresponsive.
2020-07-15 15:30: Engineering began troubleshooting the issue.
2020-07-15 15:45: Issue was escalated to the third party responsible for the server.
2020-07-15 19:00: Engineer on the third party team was able to restore the server and all affected services were back online.
2020-07-15 19:10: Confirmed that the issue was resolved.
The incident was caused due to a server controlled by a third party. This server was responsible for creative batch uploader as well as batch segment service which then became unavailable to our clients.
The third party responsible for this server resolved the issue. Xandr is reviewing the escalation process involving external parties for update.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have identified the following issue:
Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.