Incident Summary
From approximately 2020-07-15 16:00 UTC to 2020-07-16 22:15 UTC, failure of a storage system in NYM2 datacenter caused corrupt data to enter our data streams which caused some deal related reports and UI screens fail to load properly and show incomplete data.
Scope of Impact
During the incident window, partner center screen was not loading depending on the time window selected and data for UTC hours 16 and 17 was missing from certain buyer/seller deal reports.
Timeline (UTC)
2020-07-15 16:00: Incident Started - corrupted data caused deal metrics API to error out
2020-07-16 02:10: Incident Reported - escalated to engineering team
2020-07-16 22:15: Incident Resolved: corrupted data was purged and all systems were back online
Cause Analysis
Failure of a storage system in NYM2 caused corrupted data in some deal-metrics related data streams.
Resolution Steps
Deal data was cut over to LAX1 datacenter which did not have the same fault in storage system and after purging bad data, records were copied from LAX1 to NYM2.
Next Steps
Improve detection, monitoring and alerts for prevent bad data from entering the data pipeline.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.