Between June 25 (Friday) 15:00 UTC and June 28 05:00 UTC, clients experienced issues accessing the Deals dashboard in Monetize UI (stuck in "loading"), drop in volume of bid requests from some sellers, incorrect data in the deal metrics reports. This was due to a change deployed to one of the data processing jobs in New York data center.
During the incident window clients would have experienced the following impacts:
2021-06-25 15:00 : Change to data processing job was deployed
2021-06-28 02:00 : IM Ticket created
2021-06-28 05:00 : Patches deployed, data synced by Engineers
2021-06-28 05:00 : Incident resolved
The changed deployed to the data processing job was not compatible with another application downstream (which consumed the output of the job) causing these issues.
Our engineers resolved the issues by upgrading the job to a version which was compatible with all downstream consumers, deployed a patch to fix the deals UI issue and synced data with data from LA data center which was not impacted by this issue to fix the data in deal metrics reports.
Teams to validate and upgrade compatibility testing runbook, and to add more robust monitoring to detect similar problems in the future.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.