Unable to load some buy-side pages
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 19:24 UTC on Monday, November 25th to 20:35 UTC on Monday, November 25th, several buy-side Console and Invest pages would not load.

Scope of Impact

During the incident window, several buy-side Console and Invest pages were failing to load preventing some customers from creating, editing, or viewing buy-side objects. This includes, but was not limited to, the following pages: the buy-side grid page(s), the buy-side landing page(s), and buy-side create/edit page(s).

Timeline (UTC)

2019-11-25 19:24 UTC: Incident Started. Buy-side release pushed to production.

2019-11-25 20:04 UTC: Incident escalated to Engineering.

2019-11-25 20:35 UTC: Release rolled back. Incident Resolved.

Cause Analysis

The root cause of the incident was due to a server routing change to the server on which Buyside UI assets rely. As a result of the change the server did not receive requests for the Buyside UI assets, causing page failure.

Resolution Steps

The rollback of the changes made earlier immediately resolved the incident.

Next Steps

  • Revisit Buyside UI routing logic and minimize application changes where possible.
Posted Dec 04, 2019 - 01:32 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Nov 25, 2019 - 20:47 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Buy-side Console pages
  • Impact(s):
    • Page load failures and errors in user interface
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Nov 25, 2019 - 20:22 UTC