From approximately 19:24 UTC on Monday, November 25th to 20:35 UTC on Monday, November 25th, several buy-side Console and Invest pages would not load.
Scope of Impact
During the incident window, several buy-side Console and Invest pages were failing to load preventing some customers from creating, editing, or viewing buy-side objects. This includes, but was not limited to, the following pages: the buy-side grid page(s), the buy-side landing page(s), and buy-side create/edit page(s).
2019-11-25 19:24 UTC: Incident Started. Buy-side release pushed to production.
2019-11-25 20:04 UTC: Incident escalated to Engineering.
2019-11-25 20:35 UTC: Release rolled back. Incident Resolved.
The root cause of the incident was due to a server routing change to the server on which Buyside UI assets rely. As a result of the change the server did not receive requests for the Buyside UI assets, causing page failure.
The rollback of the changes made earlier immediately resolved the incident.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.