Incident Summary:
At approximately 20:20 UTC on May 26, 2020, emergency server maintenance in the NYM2 datacenter caused a backup in log processing which led to delayed log level data feeds.
Customers experienced delayed delivery of standard feed and engineered features feed for hours 2020-05-26 21:00 and 22:00 UTC.
2020-05-26 21:45: First alert of delayed data
2020-05-26 22:05: Backup in other servers identified
2020-05-26 23:35: Incident ticket created
2020-05-26 23:37: Begin installation of extra servers
2020-05-27 00:15: Transfer of backlog data completed
2020-05-27 03:59: Extra servers brought online
2020-05-27 06:34: Incident resolved
The NYM2 data center had been operating with fewer machines than normal for maintenance when an additional server failed during peak hours.
Our engineers resolved the issue by adding permanent extra capacity in the NYM2 data center.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the following issue:
Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.