Incident Summary

From approximately 20:05 UTC on Wednesday, Nov 15 to 15:14 UTC on Thursday, Nov 16, 2023, users were unable to search for deals and add them to Line item (LI) in the LI Edit workflow.

Incident Impact

• Nature of Impact(s):
• Page load failures and errors in user interface
• Unable to save/edit objects
• Incident Duration: ~19.15 Hours. 20:05 UTC on Wednesday, Nov 15 to 15:14 UTC on Thursday, Nov 16, 2023.
• Scope: Global
• Components: Buy-side pages

Timeframe (UTC)

• 2023-11-15 20:05: Incident started.
• 2023-11-16 14:47: Escalated to engineer.
• 2023-11-16 14:57: Root cause identified.
• 2023-11-16 15:14: Incident resolved.

Root Cause

The issue occurred since a version of the Buy Side UI was rolled out, which was dependent on an API change that had not been deployed to production yet. This led to page load failures and errors in the UI, preventing users from saving or editing objects in the UI, during the impact window.

Resolution

The issue was resolved by rolling out the deployment of the API change that the UI relied on. This effectively addressed the issue, enabling users to successfully search and apply deal targeting to their line items.

Immediate measures taken to mitigate the Incident

• To enhance our system's stability, our team conducted a thorough analysis to inspect relevant alerts and ensured the alerts are set with better threshold parameters and validations.
• To mitigate potential incidents and prevent unintended failures, a workaround was previously implemented that would enable users to apply Deal targeting to Line Items through Splits targeting, Deal List targeting, or the Invest Bulk Uploader tool. This would ensure to avert occurrences in the future that would potentially result in causing a similar IM