From 5:15 UTC 04/08 to 11:26 UTC 05/08, Grapeshot requests reaching the AMS datacenter would not be authorized, as they were including a header that the parent system considered invalid. Ad serving was not impacted due to redirection to NYM datacenter.
During the incident window clients using Grapeshot data were not negatively affected by the workaround.
2021-08-04 5:15 UTC: Incident Started: Initial errors logged for AMS requests.
2021-08-04 5:15 UTC: Incident Escalated: Redirection of traffic to NYM datacenter.
2021-08-05 11:26 UTC: Incident Resolved: configuration for Authorised Requests was reverted to a previous working version.
The root cause was the system unexpectedly applying a secondary authorization layer, which was also missing alerting mechanisms for those requests throwing errors.
The system was reverted to a previous configuration which allowed requests to not error out.
More mechanisms have been put in place to alert of erroring requests; additionally, the system has been updated to handle requests without headers.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.