Guaranteed Delivery Line Item Save Issues

Incident Report for Xandr

Postmortem

Incident Summary:

At 17:30 UTC there was a migration done to the servers in the LAX1 data center which missed a mapping for a forecasting service. This caused GDLIs to not be able to be saved when connected to the LAX1 data center.

Scope of Impact:

Customers who were trying to save changes to GDLIs in Monetize UI were unable to save it when connected to the LAX1 data center.

Timeline (UTC):

2020-06-17 17:30: Migration done to LAX1 data center
2020-06-22 03:15: IM Ticket created and escalated to engineer
2020-06-22 15:29: Incident ticket resolved

Cause Analysis:

A mapping required for a forecasting service got missed to be applied in the LAX1 data center while migrating. This resulted in the forecasting services in LAX1 to not work and hence the GDLI UI did not respond when the save button was clicked on.

Resolution Steps:

Our engineers resolved the issue by mapping the missed forecasting service in the LAX1 data center.

Next Steps:

Review necessary mapping and service definitions
Work on better error handling on faulty response
More thorough testing to be undertaken for migrations

Posted Jun 30, 2020 - 05:58 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 22, 2020 - 17:17 UTC

Investigating

We are currently investigating the following issue:

Component(s): Buy-side Console pages
Impact(s):
- Unable to save/edit some line item objects
Severity: Minor Outage
Datacenter(s): LAX1

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jun 22, 2020 - 15:47 UTC