Line item creating/editing Functionality for GDLI is having issues to load
Incident Report for Xandr

Incident Summary

From approximately 13:00 to 15:00 UTC on Monday, March 1, 2021, the line item create / edit functionality in Monetize was unavailable. The functionality stopped working when an upstream service, the Currency API, was returning duplicate data.

Scope of Impact

During the incident window, clients could not create or edit non-augmented line items in Monetize.

Timeline (UTC)

2021-03-01 13:03 UTC: First clients start reporting the issue

2021-03-01 13:52 UTC: Incident escalated to Engineering

2021-03-01 14:55 UTC: Currency API engineers request erroneous currency data to be corrected

2021-03-01 15:16 UTC: Incident resolved

Cause Analysis

The root cause of the incident was due to an error with the Currency API service dating back to February 25th. As a result, currency fx rates had to be populated for the missing days to match the reports. The plan was to read the corresponding dates from Oanda (our third-party currency fx rates provider) and populate the tables. However, there was an issue with the date returned, which inserted duplicate data for February 28th in our tables. The Currency API service's existing validation to check for unique day was set on a different table and the validation measure failed to catch the duplicate. This caused the line item create / edit form to stop functioning as expected.

Resolution Steps

Our engineers resolved the issue by correcting the erroneous currency data by deleting the duplicate data.

Next Steps

  • Update the line item create / edit form service to be more durable to upstream service outages
  • Add validation on all the relevant currency tables
  • Improve the monitoring and alerting for the currency exchange rate updating process
  • Add an automated test to monitor currency rendering on line item create / edit page
Posted Mar 31, 2021 - 08:13 UTC


The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Mar 01, 2021 - 18:34 UTC

We are currently investigating the following issue:

  • Component(s): Buy-side Console pages
  • Impact(s):
    • Page load failures and errors in user interface
    • Unable to save/edit objects
  • Severity: Minor Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Mar 01, 2021 - 14:21 UTC