Unable to save campaigns/ALIs with/without updating the fields
Incident Report for Xandr

Incident Summary

From approximately 06:20 UTC to 08:30 UTC on Friday, February 14, users were unable to save Line Items and Campaigns in the UI.

Scope of Impact

During the incident window, UI components were experiencing database errors, resulting in users being unable to save Line Items and Campaigns.

Timeline (UTC)

  • 2020-02-14 06:20: Incident started: Increase in number of database issues which resulted in customers' issues.
  • 2020-02-14 06:20: Incident Ticket is created.2020-02-14 06:47: Incident escalated to engineering.
  • 2020-02-14 08:30: Database issues were caused by request patterns from a few clients and were being monitored which reduced by 8:30AM.
  • 2020-02-14 12:14: Patterns of requests from the few clients stopped which cleared database issues.

Cause Analysis

Request patterns by some clients resulted in database errors.

Resolution Steps

Monitor database traffic pattern and implement more intelligent retries of database queries.

Next Steps

  • Deploy a change to provide additional mitigation while we investigate long term solutions to fully prevent this issue.
Posted Feb 27, 2020 - 02:45 UTC


The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Feb 14, 2020 - 13:10 UTC

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted Feb 14, 2020 - 09:44 UTC

We are currently investigating the following issue:

  • Component(s): Buy-side Console pages, Console API
  • Impact(s):
    • Unable to save/edit objects
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Feb 14, 2020 - 08:35 UTC