Incident Summary

From approximately 13:00 to 15:00 UTC on Monday, March 1, 2021, the line item create / edit functionality in Monetize was unavailable. The functionality stopped working when an upstream service, the Currency API, was returning duplicate data.

‌

Scope of Impact

During the incident window, clients could not create or edit non-augmented line items in Monetize.

‌

Timeline (UTC)

2021-03-01 13:03 UTC: First clients start reporting the issue

2021-03-01 13:52 UTC: Incident escalated to Engineering

2021-03-01 14:55 UTC: Currency API engineers request erroneous currency data to be corrected

2021-03-01 15:16 UTC: Incident resolved

‌

Cause Analysis

The root cause of the incident was due to an error with the Currency API service dating back to February 25th. As a result, currency fx rates had to be populated for the missing days to match the reports. The plan was to read the corresponding dates from Oanda (our third-party currency fx rates provider) and populate the tables. However, there was an issue with the date returned, which inserted duplicate data for February 28th in our tables. The Currency API service's existing validation to check for unique day was set on a different table and the validation measure failed to catch the duplicate. This caused the line item create / edit form to stop functioning as expected.

‌

Resolution Steps

Our engineers resolved the issue by correcting the erroneous currency data by deleting the duplicate data.

‌

Next Steps

• Update the line item create / edit form service to be more durable to upstream service outages
• Add validation on all the relevant currency tables
• Improve the monitoring and alerting for the currency exchange rate updating process
• Add an automated test to monitor currency rendering on line item create / edit page