Invalid API Certificate

Incident Report for Xandr

Postmortem

Incident Summary

From approximately 22:40 UTC to 23:10 UTC on Wednesday, November 6, there was an invalid API certificate. An update to the certificates for http://api.appnexus.com load balancers caused users to experience certificate errors when attempting to use the service.

Scope of Impact

During the incident window, clients using https://api.appnexus.com/ saw certificate errors when using the service.

Timeline (UTC)

2019-11-06 22:40: Incident started, invalid API certificate uploaded
2019-11-06 22:58: Incident escalated to engineering
2019-11-06 23:10: Incident resolved, certificate change reverted

Cause Analysis

The certificates for https://api.appnexus.com/ were updated to load balancers with certificates that were signed by our private internal certificate authority.

Resolution Steps

The certificates were reverted back to the original certificate.

Next Steps

*Update System Operations procedures around release planning, testing, and validation.

Posted Nov 14, 2019 - 15:08 UTC

Resolved

The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Impbus API, Console API
  • Impact(s):
    • Latency, timeouts and errors in API
  • Severity: Major Outage
  • Datacenter(s): Global

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Nov 06, 2019 - 23:22 UTC