Potential Delivery Issues for Augmented Line Items Using Discovery
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 21:24 to 23:35 UTC on Wednesday, July 1st, 2020 some Augmented Line Item's using optimization may have not explored new inventory, which could have resulted in under-delivery.

Scope of Impact

During the incident window, some Augmeneted Line Item's may have not explored new inventory as a result of not receiving optimization records, potentially causing under-delivery for these objects.

Timeline (UTC)

2020-07-01 21:24: Incident Started
2020-07-01 22:41: Incident Escalated
2020-07-01 23:35: Incident Resolved: Change data capture system removed from affected optimization node.

Cause Analysis

The root cause of the incident was due to a database connection issue in the LAX data center tied to database maintenance. This connection issue resulted in a window of time during which some optimization mechanisms were not operational.

Resolution Steps

Our engineers resolved the issue by turning off the underlying system for the affected optimization mechanism.

Next Steps

  • Build fault tolerance and resilience in optimization application
  • Investigate database connection drops
Posted Aug 05, 2020 - 00:09 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jul 01, 2020 - 23:59 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Bidding
  • Impact(s):
    • Some objects may spend under budgets
  • Severity: Minor Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jul 01, 2020 - 22:48 UTC