Incident Summary

From approximately 07:43 to 09:09 UTC on Tuesday, July 28, 2020 clients were unable to access line items and insertion orders via the Order Home dashboard.

Scope of Impact

During the incident window, customers could not load line items and insertion orders via Order Home dashboard as they encountered a "Gateway Timeout" error.

Timeline (UTC)

2020-07-28 07:43: Incident started
2020-07-28 08:26: Incident escalated to on-call engineer
2020-07-28 09:09: Incident resolved

Cause Analysis

A backend process that reads line item / insertion order IDs from a queue and updates their status stopped processing, causing the queue to grow exponentially until it reached its maximum capacity. After several hours of attempting to process at maximum capacity, the resource load caused our caching mechanisms to fail and the system stopped responding to user requests.

Resolution Steps

Our engineers resolved the issue by swapping the production environment over to the backup database cluster. The primary database cluster was afterwards restarted and, after re-indexing, we were able to conclude that the issue was resolved.

Next Steps

• Set up an alert that detects if our queues have reached the maximum capacity and stayed above that for a certain time
• Status updates of the database need to be monitored