Unable to edit ALI
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 07:15 UTC to 09:03 UTC on Friday, December 13, users were unable to edit Line Items in the UI.

Scope of Impact

During the incident window, UI components failed to retrieving Line Item information, resulting in errors and blank editing pages.

Timeline (UTC)

  • 2019-12-13 07:15: Incident started: the last instance of the responsible service went down.
  • 2019-12-13 08:36: Incident escalated to engineering.
  • 2019-12-13 09:03: Service restarted, incident resolved.

Cause Analysis

While trying to prevent the same incidents from happening again while a permanent solution is deployed a bug was introduced in the responsible service.

Resolution Steps

Restarting the service that caused the incident.

Next Steps

  • Deploy the long term fix that removes the UI dependencies from this service.
Posted Dec 17, 2019 - 11:04 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Dec 10, 2019 - 10:14 UTC
Monitoring

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted Dec 10, 2019 - 09:10 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Buy-side Console pages
  • Impact(s):
    • Page load failures and errors in user interface
  • Severity: Minor Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Dec 10, 2019 - 08:56 UTC