Order Home not Working Properly
Incident Report for Xandr
Postmortem

Incident Summary
From approximately 07:30 UTC to 09:50 on 27 April, 2021, the Order Home Line Items View was not functioning properly. Users were able to see line items in the grid, but information displayed in the grid was inaccurate and filters returned unexpected results due to stale data, due to issues with data source in NYM Data Center used for the Order Home Line Items View.

Scope of Impact
During the incident window, users were unable to accurately rely on information displayed in the grid and filters returned unexpected results due to stale data.

Timeline (UTC)
2021-04-27 07:30: Incident Started, multiple EMEA ad server clients reported issues
2021-04-27 07:31: IM ticket raised
2021-04-27 09:37: Engineering team switched Order Home Data to rely on another data source in LAX Data Center (after testing successfully in the staging test environment)
2021-04-27 09:50: Incident Resolved
2021-04-27 10:01: Data source application in NYM data center was restarted
2021-04-27 22:58: Issue was confirmed to be resolved and IM ticket was closed

Cause Analysis
During migration of certain instances, the Order Home configuration step had some issues which resulted in Order home to look at stale data when pointed at NYM data center.

Resolution Steps
Our engineers resolved the issue by temporarily switching the data source from NYM Data Center to LAX Data Center while restarting the data source application for NYM Data Center.

Next Step(s)
• Improve and update migration steps
• Update/expand catch-point tests to try and detect intermittent failures, failures in filtering functionality

Posted May 04, 2021 - 14:54 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Apr 27, 2021 - 10:02 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Order Home UI
  • Impact(s):
    • Filtering working incorrectly, Line Items missing from the dashboard, Line Items' statuses incorrect
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Apr 27, 2021 - 09:11 UTC