Incident Summary

From approximately 7:30AM UTC - 9:30AM UTC Console/Platform was showing the error "Error fetching" for Insertion Orders, Line Items.

Scope of Impact

During the incident window, customers would have intermittently seen errors when attempting to load order-home console pages across both console and monetize. These pages are used to view grids of advertisers, line items, and insertion orders for monitoring.

Timeline (UTC)

2019-12-14 07:30: Incident started.
2019-12-14 07:59: Issue escalated to Engineering
2019-12-14 09:27: Incident Resolved

Cause Analysis

The incident appeared intermittently and seemed to primarily affect the Amsterdam data center. Errors occurred across line item, advertiser, insertion order pages in order-home apparently caused by load issues. Some internal processes involved can be very resource intensive for members with large object counts.

Resolution Steps

Our engineering team resolved the issue by re-pointing the endpoints internally.

Next Steps

*Increasing the resources available to ensure that heavy loads are handled well.