Incident Summary
From approximately 16:40 UTC on Wednesday, January 29th to 17:25 UTC on Wednesday, January 29th a cookie data store release implementing a new format for frequency data resulted in the loss of frequency counts on some active cookies during the incident window.

Scope of Impact
During the incident window 25-50% of cookied users who saw impressions during the incident window lost their associated frequency counts. As a result of this incident, some clients may observe a violation of some frequency caps for affected cookied users up to one month following this incident.

Timeline (UTC)
2020-01-29 16:40 UTC: Incident Started: Release made to cookie data store that overwrites frequency counts
2020-01-29 17:25 UTC: Release completed.
2020-01-29 19:40 UTC: Post-release monitoring detects cookie frequency anomalies
2020-01-29 20:40 UTC: Incident escalated.
2020-01-30 15:00 UTC: Incident Resolved: All datacenters confirmed back to normal

Cause Analysis
The root cause of the incident was due to a cookie data store release meant to update the cookie frequency data format. Cookie data store instances that were not on the new release registered cookies without frequency data in the old format, overwriting the new format and effectively resetting frequency counts for these users.

Resolution Steps
Following the release engineers stopped using the new frequency count format to overwrite frequency counts.

Next Steps

• Ensure old and new data versions are not mixed during release(s)
• Re-release of new format that retains old format