Retroactive: From 15:00 to 16:53 UTC on March 20th, Users were Unable to Edit or Create Splits in Invest

INCIDENT SUMMARY

From approximately 15:22 UTC on Monday, Mar 20 to 16:53 UTC on Monday, Mar 20, 2023, Users received an error while creating or editing Splits in Invest, during the impact window.

INCIDENT IMPACT

• Nature of Impact: Users received an error while creating or editing Splits in Invest 

• Timeframe: ~1.51 Hours. 15:22 UTC on Monday, Mar 20 to 16:53 UTC on Monday, Mar 20, 2023.

• Scope: Global

• Components: Buy-side pages

TIMELINE (UTC)

• 2023-03-20 15:00: A scheduled MySQL mode change was deployed. 

• 2023-03-20 15:22: Incident Started. Investigation started by our Engineering Team.

• 2023-03-20 16:06: Issue escalated across other teams that were responsible.

• 2023-03-20 16:22: Triggering action identified. Initial diagnosis and hypothesis were put forward.

• 2023-03-20 16:39: Fix was deployed and tested in staging.

• 2023-03-20 16:52: Incident was mitigated and the fix was promoted to production.

• 2023-03-20 16:53: Incident Resolved.

• 2023-03-20 18:57: Ticket opened retroactively.

CAUSE ANALYSIS

The issue was caused due to a scheduled change in the MySQL mode that was deployed. This resulted in an API failure which prevented the users from creating or editing Splits in Invest, during the impact window.

RESOLUTION STEPS

A quick fix was deployed to accommodate the MySQL mode change which resolved the issue. 

FOLLOW-UP ITEMS

• Classified the team's critical applications and their dependencies to prioritize auditing of the applications. Additionally the engineering team would provide details about the expected “impact” to help application owners understand the result of any similar changes/upgrades that would occur in future.

• Ensured to align on Standard Operating Procedure for platform-wide upgrades and add test coverage to accommodate MySQL mode change to avert similar occurrences in the future that would potentially result in causing a similar IM.