On September 03, at 00:04 UTC, there was a currency API response change which led to issues editing Line items with data clearing segments. The issue has since been resolved.
During the incident window, customers were unable to edit line items with data clearing segments associated to them in Invest.
2020-09-03 00:04 : Incident started
2020-09-03 00:31 : IM Ticket created
2020-09-03 01:23 : Issue identified as change in currency API response
2020-09-03 01:37 : Escalated to engineer
2020-09-03 02:10: Incident Resolved. Database update executed.
The daily rates insertion process of currency API ran at a few seconds before UTC 12th hour instead of the expected/ regular UTC 3rd hour. The change in the run time was due to a recent change in the servers where the currency API is hosted. This process picks the current date at start of its run. On September 3rd, since it started by last few seconds of the September 2nd, it picked the date as September 01. This led to duplication for September 01 rates, as the table already had data for September 01 from the run that started at early September 02.
Our engineers resolved the issue by releasing a quick fix database update to restore the run back to UTC 3rd hour. Monitoring components & validations are also being applied in the currency API to avoid duplicate insert.
Make UI applications more resilient.
Work on better error/ debug logging in UI
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.