Incident Summary

From 2020-09-16 16:10 UTC to 2020-09-16 18:28 UTC, pacing record data was down which may have resulted in under or over delivery.

Scope of Impact

The downtime may have resulted in overspend or underspend in some buyside objects. However, because the downtime only occurred in a small portion of our LAX1 Data Center, impact was quite limited.

Timeline (UTC)

2020-09-16 16:10 UTC : Incident began and engineering was alerted
2020-09-16 16:20 UTC: Engineering began troubleshooting the incident
2020-09-16 16:35 UTC: Incident was escalated internally
2020-09-16 18:28 UTC: Issue was identified and resolved

Cause Analysis
Upon further investigation, the root cause was connection issues within our data messaging system that resulted in some bidders in LAX1 to stop ingesting pacing data updates.

Resolution Steps
After restarting and reestablishing fresh connection, the bidders resumed ingestion of pacing data and the issue was resolved.

Next Steps
Engineering is looking to add monitoring tools and processes that will allow us to identify these issues in the future