Publisher Grid Objects not being Populated in Console and Monetize
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 23:38 UTC on Wednesday, November 20th to 02:07 UTC on Thursday, November 21st, data did not populate in the Publisher Grid.

Scope of Impact

During the incident window, the Publisher Grid was empty.

Timeline (UTC)

2019-11-20 23:38: Incident Started: Internal alerts triggered for no data being served in the Publisher Grid.
2019-11-21 00:10: Incident escalated to engineering
2019-11-21 02:07: Incident Resolved

Cause Analysis

This issue was caused by the removal of specific Rapid Report columns, which the Publisher Grid was depending on.

Resolution Steps

Our engineers reverted the change, which immediately resolved the incident.

Next Steps

We would like to look into how clients are notified about changes to rapid-reports or the data that feeds rapid-reports, and how we can get notification of those releases out to Xandr engineering teams.

Posted Nov 25, 2019 - 19:09 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Nov 21, 2019 - 02:06 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Sell-side Console pages
  • Impact(s):
    • Page load failures and errors in user interface
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Nov 21, 2019 - 00:28 UTC