LAX1 Timeouts
Incident Report for Xandr

Incident Summary:

From approximately 21:00 UTC on September 13 to 00:58 UTC on September 14, 2021 the LAX1 datacenter suffered from a high rate of timeout issues.

Scope of Impact:

During the incident window some customers using console bidders in LAX1 may have observed a drop in delivery.

Timeline (UTC):

2021-09-13 21:00: Incident started
2021-09-13 23:45: Escalated to engineering
2021-09-14 00:58: Incident resolved

Cause Analysis:

Engineering introduced a defect which, in addition to connection limitations, overstressed the LAX1 datacenter and caused bidder timeouts.

Resolution Steps:

A reconfiguring of the connections within the datacenter resolved the timeout issue. Engineering has since identified and fixed the root defect.

Next Steps:

Review and update internal alerting process to alert engineering of similar issues sooner.

Posted Sep 21, 2021 - 18:56 UTC


The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Ad Serving
  • Impact(s):
    • Drop in delivery on Invest
  • Severity: Partially Degraded
  • Datacenter(s): LAX1

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Sep 14, 2021 - 21:14 UTC