From approximately 16:40 UTC on Wednesday, January 29th to 17:25 UTC on Wednesday, January 29th a cookie data store release implementing a new format for frequency data resulted in the loss of frequency counts on some active cookies during the incident window.
Scope of Impact
During the incident window 25-50% of cookied users who saw impressions during the incident window lost their associated frequency counts. As a result of this incident, some clients may observe a violation of some frequency caps for affected cookied users up to one month following this incident.
2020-01-29 16:40 UTC: Incident Started: Release made to cookie data store that overwrites frequency counts
2020-01-29 17:25 UTC: Release completed.
2020-01-29 19:40 UTC: Post-release monitoring detects cookie frequency anomalies
2020-01-29 20:40 UTC: Incident escalated.
2020-01-30 15:00 UTC: Incident Resolved: All datacenters confirmed back to normal
The root cause of the incident was due to a cookie data store release meant to update the cookie frequency data format. Cookie data store instances that were not on the new release registered cookies without frequency data in the old format, overwriting the new format and effectively resetting frequency counts for these users.
Following the release engineers stopped using the new frequency count format to overwrite frequency counts.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the following issue:
Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.