Incident Summary
On December 4th at 19:30 UTC, users may have experienced creative previews not loading on pages of Invest and Console. This issue was fully resolved on December 6th at 01:00 UTC. Adserving was not affected.
Scope of Impact
Between 2019-12-04 19:30 UTC and 2019-12-05 23:00 UTC some members may have experienced creative previews not loading on certain pages of Console and Invest. Between 2019-12-05 20:40 UTC and 2019-12-06 01:00 UTC, users may have seen a “403 permission denied” error.
Timeline (UTC)
2019-12-04 19:30: Incident Started: API release started initial errors
2019-12-05 20:40: Access control release, starting 403 errors within Console to affect even more members
2019-12-05 22:15: Escalated to engineers
2019-12-05 22:50: API release rolled back, resolving “failed to render dynamic banner preview remotely” error
2019-12-06 01:00: Access Control release rolled back, resolve the “403 permission denied” error
Cause Analysis
The incident was caused by two separate releases. The first was an API release which caused a “failed to render dynamic banner preview remotely” error and the second was an access control release that caused the “403 permission denied” error seen in Console/Invest.
Resolution Steps
The issues were resolved by both releases being rolled back.
Next Steps
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.