Incident Summary

From approximately 16:46 UTC to 17:41 UTC on Thursday, April 2nd, 2020, a MySQL database server went down, impacting accessibility to the augmented line item UI, API, and associated services.

Scope of Impact

Augmented line item, inventory list, and order home UI and API were not working properly for approximately 1 hour.

Timeline (UTC)

2020-04-02 16:46: Incident Started: MySQL database server went down

2020-04-02 17:23: Incident Escalated

2020-04-02 17:41: Incident Resolved: Traffic was pointed to back up server

Cause Analysis

The root cause of the incident was due to a failure with a MySQL database server for which there was no designated back up server.

Resolution Steps

The issue was resolved by pointing traffic to a back up server.

Next Steps

• Improve internal notifications for quicker alert response
• Designate back up servers for automatic switchover