Incident Summary

From 15:00 to 16:00 UTC on February 28, 2020, the age of network analytics reporting data was greater than 6 hours

Scope of Impact

Data that clients pulled during the 15:00 to 16:00 UTC hour was older than 6 hours. No data was lost

Timeline (UTC)

2020-02-28 15:00:00: Incident began, reporting data delayed longer than 6 hours

2020-02-28 15:15:00: Engineering was automatically notified and began investigating the issue

2020-02-28 16:00:00: Engineering updated the reporting hours manually so clients had access to data

2020-02-28 16:30:00: Engineering deployed fix so data was updated automatically, which resolved the issue

Cause Analysis

An internal database incident caused an internal data verification tool to become unavailable. Due to the tool being unavailable, data could not be verified and was therefore not released

Resolution Steps

The Network Analytics table was verified and updated manually until an internal fix was deployed, allowing reporting to resume as scheduled

Next Steps

• Determine the reason the delay did not alert the team earlier
• Stand up earlier notification process to avoid client facing reporting delays in the future