Incident Summary

From approximately 09:41 to 19:37 UTC on September 7th, 2020 data in the Seller Fill and Delivery Report was delayed by up to 24 hours.

Scope of Impact

During the incident window some clients would have seen a data discrepancy between the Monetization Dashboard and the Seller Fill and Delivery Report (that was 24 hours behind in comparison to the Dashboard).

Timeline (UTC)

2020-09-06 09:00 UTC: Incident Started: The reporting job begins to experience delays
2020-09-07 09:41 UTC: Discrepancy reported within the Monetize Dashboard
2020-09-07 11:15 UTC: Incident Escalated
2020-09-07 19:37 UTC: Incident Resolved: Node was restarted, data backfilled, failsafe implemented

Cause Analysis

A data processing node did not complete within the allotted time limit, creating a backlog. The root cause of the incident was a lack of monitoring mechanisms for report nodes that do not complete successfully within time limits, creating a backlog.

Resolution Steps

Our engineers resolved the issue by setting up the reporting job to run for 2 hours at a time (instead of one). This will prevent a bad node from blocking the following attempt from running.

Next Steps

Implement monitoring mechanisms for report nodes so that we can detect bad nodes on our end before they impact customers