Invalid API Certificate
Incident Report for Xandr

Incident Summary

From approximately 22:40 UTC to 23:10 UTC on Wednesday, November 6, there was an invalid API certificate. An update to the certificates for load balancers caused users to experience certificate errors when attempting to use the service.

Scope of Impact

During the incident window, clients using saw certificate errors when using the service.

Timeline (UTC)

2019-11-06 22:40: Incident started, invalid API certificate uploaded
2019-11-06 22:58: Incident escalated to engineering
2019-11-06 23:10: Incident resolved, certificate change reverted

Cause Analysis

The certificates for were updated to load balancers with certificates that were signed by our private internal certificate authority.

Resolution Steps

The certificates were reverted back to the original certificate.

Next Steps

*Update System Operations procedures around release planning, testing, and validation.

Posted Nov 14, 2019 - 15:08 UTC


The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Impbus API, Console API
  • Impact(s):
    • Latency, timeouts and errors in API
  • Severity: Major Outage
  • Datacenter(s): Global

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Nov 06, 2019 - 23:22 UTC