Console/Platform: Showing Error fetching Line Items/Insertion Orders
Incident Report for Xandr

Incident Summary

From approximately 7:30AM UTC - 9:30AM UTC Console/Platform was showing the error "Error fetching" for Insertion Orders, Line Items.

Scope of Impact

During the incident window, customers would have intermittently seen errors when attempting to load order-home console pages across both console and monetize. These pages are used to view grids of advertisers, line items, and insertion orders for monitoring.

Timeline (UTC)

2019-12-14 07:30: Incident started.
2019-12-14 07:59: Issue escalated to Engineering
2019-12-14 09:27: Incident Resolved

Cause Analysis

The incident appeared intermittently and seemed to primarily affect the Amsterdam data center. Errors occurred across line item, advertiser, insertion order pages in order-home apparently caused by load issues. Some internal processes involved can be very resource intensive for members with large object counts.

Resolution Steps

Our engineering team resolved the issue by re-pointing the endpoints internally.

Next Steps

*Increasing the resources available to ensure that heavy loads are handled well.

Posted Dec 23, 2019 - 00:07 UTC


The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Console/Platform Order-Home Pages
  • Impact(s):
    • Page load failures and errors in user interface
  • Severity: Major Outage
  • Datacenter(s): Global

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Dec 16, 2019 - 09:39 UTC