Incident Summary
Between 22:30 UTC on Saturday, January 15th to 05:30 UTC on Sunday, January 16th, SSP's experienced increased latency in our New York datacenter collections.
Scope of Impact
During the incident window SSP clients may have experienced high latency traffic in the form of higher timeout rates.
Timeline (UTC)
2020-01-15 22:30: Incident Started: Mobile traffic moved into the default datacenter collection
2020-01-15 22:46: Incident Escalated
2020-01-16 01:00: New load balancers were added to the default collection.
2020-01-16 04:30: Additional load balancers were added.
2020-01-16 04:30: Incident Resolved.
Cause Analysis
The incident was caused by backend maintenance. This maintenance caused connectivity issues which then resulted in the latency increase in the New York datacenter.
Next Steps
The issue was mitigated that caused the load balancer to overload, and are continuing to develop long term solutions to continue to improve the SSP services.