Unable to create or access deals in Monetize UI deals dashboard (with buyers)
Incident Report for Xandr

Incident Summary:

Between June 25 (Friday) 15:00 UTC and June 28 05:00 UTC, clients experienced issues accessing the Deals dashboard in Monetize UI (stuck in "loading"), drop in volume of bid requests from some sellers, incorrect data in the deal metrics reports. This was due to a change deployed to one of the data processing jobs in New York data center.

Scope of Impact:

During the incident window clients would have experienced the following impacts:

  • Unable to access the deals (with buyers) in Monetize UI in the deals dashboard
  • Incorrect data in the Deal Metrics reports
  • Drop in the volume of bid requests from some sellers

Timeline (UTC):

2021-06-25 15:00 : Change to data processing job was deployed
2021-06-28 02:00 : IM Ticket created
2021-06-28 05:00 : Patches deployed, data synced by Engineers
2021-06-28 05:00 : Incident resolved

Cause Analysis:

The changed deployed to the data processing job was not compatible with another application downstream (which consumed the output of the job) causing these issues.

Resolution Steps:

Our engineers resolved the issues by upgrading the job to a version which was compatible with all downstream consumers, deployed a patch to fix the deals UI issue and synced data with data from LA data center which was not impacted by this issue to fix the data in deal metrics reports.

Next Steps:

Teams to validate and upgrade compatibility testing runbook, and to add more robust monitoring to detect similar problems in the future.

Posted Jul 06, 2021 - 19:48 UTC


The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 28, 2021 - 05:45 UTC

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted Jun 28, 2021 - 05:04 UTC

We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.

Posted Jun 28, 2021 - 03:51 UTC

We are currently investigating the following issue:

  • Component(s): Partner/Deal pages
  • Impact(s):
    • Monetize Deals UI keeps "loading" on deals dashboard (with buyers) when creating/ viewing deals
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jun 28, 2021 - 02:55 UTC