Line Item / Insertion Order saving issue

Incident Report for Xandr

Postmortem

Incident Summary

From approximately 09:56 UTC to 15:00 UTC on Tuesday, September 14, 2021, users were intermittently unable to view order home monitoring screens.

Scope of Impact

During the incident window, customers were intermittently unable to view order home monitoring screens and therefore unable to monitor the performance of their line items, particularly GDALIs.

Timeline (UTC)

2021-09-14 09:56: Incident Started
2021-09-14 14:29: Incident escalated to engineering
2021-09-14 14:50: Engineering initiates mitigation steps
2021-09-14 15:00: Incident Resolved

Cause Analysis

The incident was caused by an increase in traffic during EU business hours.

Resolution Steps

Our engineers resolved the issue by moving the order home data pods to a different data center.

Next Steps

Review and add additional alerting to the backend systems controlling the order home monitoring screens.
Implement more monitoring to this area to help identify potential issues prior to becoming an outage.

Posted Sep 28, 2021 - 17:23 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Sep 14, 2021 - 19:00 UTC

Monitoring

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted Sep 14, 2021 - 15:32 UTC

Investigating

We are currently investigating the following issue:

Component(s): Buy-side pages, Order Home
Impact(s):
- Page load failures and errors in user interface
- Unable to save/edit objects
Severity: Major Outage
Datacenter(s): AMS1

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Sep 14, 2021 - 15:15 UTC