Incident Summary

From approximately 23:45 UTC on Tuesday, January 5, 2021 to 17:30 UTC on Thursday, January 14, 2021, some Xandr users were unable to access the Grapeshot Keywords app due to an authentication issue.

‌

Scope of Impact

During the incident window, customers trying to use the Grapeshot Keywords app were shown an error message that cookies need to be enabled in the Apps Marketplace UI. This was caused by the underlying authentication failures.

‌

Timeline (UTC)

2021-01-05 23:45: incident started after the release of an upgrade to our authentication API service

2021-01-14 11:00: issue was reported and escalated by customers

2021-01-14 12:03: issue was escalated to our engineers after initial checks

2021-01-14 13:35: customers reported that switching API authentication calls to HTTPS fixed the issue for existing users, but not new ones

2021-01-14 17:30: incident resolved - we rolled back the release that caused the problem and clients confirmed this resolved the issue for all users

‌

Cause Analysis

The incident was caused by a release to our authentication API service. This caused failures for third-party app marketplace authentication. Regular UI/API authentications were unaffected.

‌

Resolution Steps

Our engineers resolved the issue by rolling back the problematic release. Separately we are working to fix the root issue before releasing that update again.

‌

Next Steps

• Fix the library upgrade which caused the issue for the authentication API
• Improve monitoring for non-200 response codes, especially after new releases