The connection to Azure West US was down
Incident Report for Xandr
Postmortem

Incident Summary
On Friday, June 11, 2021, Liveramp in US West Coast went down for about 8 hours due to ISP maintenance.

Scope of Impact
The identification of some parameters of the bid requests might have been failing, but they could still being transacted.

Timeline (in UTC)
2021-06-11 07:04: Incident Started.
2021-06-11 08:45: Checked that all the resources in Azure West US were unreachable.
2021-06-11 10:07: The datacenter provider started maintenance.
2021-06-11 14:49: Maintenance completed.
2020-06-14 14:25: IM Ticket closed.

Cause Analysis
There was a CoreSite maintenance of which we weren't informed. During this period, our servers weren't not able to reach Liveramp.

Resolution Steps
The coresite completed their maintenance at 14:49 UTC.

Next Steps
Make alerts escalated to several teams in order to get informed about this kind of issues as soon as possible.
Review the network setup in US West Coast and add network redundancy in order to provide service 24x7.

Posted Jun 15, 2021 - 13:14 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 11, 2021 - 17:57 UTC
Monitoring

The services came backup. We are checking with vendor to confirm the maintenance status.

Posted Jun 11, 2021 - 15:36 UTC
Update

We're still working with our hardware vendor in order to fix this. Our vendor extended the maintenance to 3PM UTC. Thank you for your patience.

Posted Jun 11, 2021 - 13:47 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Impbus API
  • Impact(s):
    • The identification of some parameters of the bid requests might be failing,but can still transacted.
  • Severity: Partially Degraded
  • Datacenter(s): LAX1

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jun 11, 2021 - 10:05 UTC