From approximately 09:25 UTC to 14:32 UTC on Thursday, May 28th, 2020 network maintenance in the NYM2 data center caused disruptions between services that required manual restarting to resolve. These services included Invest, Monetize, Console, and Platform.
During the incident window some clients would have experienced issues logging in and editing objects. Batch-segment upload was also affected. This happened to clients using the UI as well as the API.
2020-05-28 05:11 UTC: Planned Maintenance started
2020-05-28 09:25 UTC: Incident Reported: Console UI or API authentication alerts were reported
2020-05-28 14:32 UTC: Incident Resolved: Manual restart completed and authentication and editing available again.
The root cause of the incident was a networking loss in NYM2 that caused issues in applications that were unable to automatically recover.
Our engineers resolved the issue by shifting the network connection to our LAX1 and AMS1 data centers and manually restarted the disrupted applications.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.