Unable to save campaigns/ALIs with/without updating the fields

Incident Report for Xandr

Postmortem

From approximately 06:20 UTC to 08:30 UTC on Friday, February 14, users were unable to save Line Items and Campaigns in the UI.

During the incident window, UI components were experiencing database errors, resulting in users being unable to save Line Items and Campaigns.

2020-02-14 06:20: Incident started: Increase in number of database issues which resulted in customers' issues.
2020-02-14 06:20: Incident Ticket is created.2020-02-14 06:47: Incident escalated to engineering.
2020-02-14 08:30: Database issues were caused by request patterns from a few clients and were being monitored which reduced by 8:30AM.
2020-02-14 12:14: Patterns of requests from the few clients stopped which cleared database issues.

Request patterns by some clients resulted in database errors.

Monitor database traffic pattern and implement more intelligent retries of database queries.

Deploy a change to provide additional mitigation while we investigate long term solutions to fully prevent this issue.

Posted Feb 27, 2020 - 02:45 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Feb 14, 2020 - 13:10 UTC

Monitoring

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted Feb 14, 2020 - 09:44 UTC

Investigating

We are currently investigating the following issue:

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Feb 14, 2020 - 08:35 UTC