Hosted creatives uploads on UI and API are not working as expected

Incident Report for Xandr

Postmortem

Incident Summary
From approximately 14:15 UTC on September 5 to 6:55 UTC on September 6, 2019, users nearest to our Los Angeles (LAX1) and New York (NYM2) data centers were unable to upload hosted creatives.

Scope of Impact
During the incident window, customers nearest the LAX1 and NYM2 data centers were unable to upload hosted creatives.

Timeline (UTC)
2019-09-05 14:15: Incident Started: Creative API deployed.

2019-09-06 04:43: Issue escalated as incident and immediately raised to engineering.

2019-09-06 06:55: Incident Resolved: Creative API rolled back to previous version.

Cause Analysis
The Creative API release contained incorrect routing configurations.

Resolution Steps
Our engineers resolved the issue by rolling back to the application's previous version.

Next Steps
* Improve monitoring to detect future routing failures

Posted Sep 10, 2019 - 08:29 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Sep 06, 2019 - 07:05 UTC

Investigating

We are currently investigating the following issue:

Component(s): Creative uploads
Impact(s):
- Unable to upload creatives, current adserving is unaffected
Severity: Major Outage
Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Sep 06, 2019 - 06:04 UTC