From approximately 08:00 UTC to 12:05 UTC on Thursday, November 14, users were unable to edit Line Items in the UI.
During the incident window, UI components failed to retrieving Line Item information, resulting in errors and blank editing pages.
2019-11-14 08:00: Extreme influx of database load
2019-11-14 10:55: Incident escalated to engineering
2019-11-14 12:05: Incident resolved
An unusual delay of graph data from a vendor resulted in a sudden spike of incoming load on cross-device functionality, flooding database capacity. The UI currently relies on this graph data for Line Item viewing, editing and creation, and therefore was unavailable until database load subsided.
Our engineers intervened with manual database cleanup in order to restore performance.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We are currently investigating the following issue in order to fix it as soon as possible:
We will provide an update as soon as more information is available. Thank you for your patience.