Datacenter Connectivity Issues
Incident Report for Xandr
Postmortem

Incident Summary

On November 30th, from 19:26 UTC to 20:28 UTC, log data accumulated in a single data center due to a carrier fiber outage. There was no discernible client impact.

Scope of Impact

Due to the low data log volume in the affected data center there was no discernible impact to clients.

Timeline (UTC)

2019-11-30 19:26: Incident Started: Carrier fiber cut.

2019-11-30 19:36: Incident Escalated by engineering.

2019-11-30 20:28: Carrier rerouted traffic. Incident resolved.

Cause Analysis

The incident was caused by a carrier fiber outage which took down inter data center circuits in a single data center.

Resolution Steps

The issue was resolved by the carrier by rerouting the necessary circuits.

Next Steps

  • Collaborate and work with carrier to ensure there is no single point of failure.
Posted Dec 17, 2019 - 17:12 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Nov 30, 2019 - 21:25 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Console UI, Reporting, Data delays
  • Impact(s):
    • Stale reporting data
    • Page load failures and errors in user interface
    • Unable to save/edit objects
  • Severity: Minor Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Nov 30, 2019 - 20:08 UTC