INCIDENT SUMMARY
From approximately 04:40 UTC on Thursday, May 25 to 13:40 UTC on Friday, May 26, 2023, 100% of request were timing out on the Azure US West LiveRamp Prod Cluster.
INCIDENT IMPACT
TIMELINE (UTC)
CAUSE ANALYSIS
The issue was attributed to our vendor where a maintenance activity was being carried out on their internal circuits. Due to this, the LAX1 datacenter to Azure infrastructure experienced a delay in response time which had a significant impact on latency-sensitive traffic. This resulted in all requests timing out on the Azure US West Liveramp Prod Cluster.
RESOLUTION STEPS
The issue resolved on its own once the maintenance activity was completed by our vendor’s carrier, returning the latency to normal which fully restored the network.
FOLLOW-UP ITEMS
Engineering team continues to follow-up with the vendor for a detailed RCA.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We are currently investigating the following issue::
We will provide an update as soon as more information is available. Thank you for your patience.