LI not available
Incident Report for Xandr

Incident Summary

From approximately 08:00 UTC to 12:05 UTC on Thursday, November 14, users were unable to edit Line Items in the UI.

Scope of Impact

During the incident window, UI components failed to retrieving Line Item information, resulting in errors and blank editing pages.

Timeline (UTC)

2019-11-14 08:00: Extreme influx of database load
2019-11-14 10:55: Incident escalated to engineering
2019-11-14 12:05: Incident resolved

Cause Analysis

An unusual delay of graph data from a vendor resulted in a sudden spike of incoming load on cross-device functionality, flooding database capacity. The UI currently relies on this graph data for Line Item viewing, editing and creation, and therefore was unavailable until database load subsided.

Resolution Steps

Our engineers intervened with manual database cleanup in order to restore performance.

Next Steps

  • Increase priority of the database alert because of the UI dependencies.
  • Review data ingestion process for cross-device graph data
  • Refactor UI dependencies to ensure graceful degradation
Posted Nov 15, 2019 - 17:26 UTC


The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Nov 14, 2019 - 14:00 UTC

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted Nov 14, 2019 - 12:30 UTC

We are currently investigating the following issue in order to fix it as soon as possible:

  • Component(s): Buy-side Console pages
  • Impact(s):
    • Errors viewing/editing line items
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Nov 14, 2019 - 12:02 UTC