Some Creatives Failing to Preview in Some UI Environments
Incident Report for Xandr

Incident Summary

On December 4th at 19:30 UTC, users may have experienced creative previews not loading on pages of Invest and Console. This issue was fully resolved on December 6th at 01:00 UTC. Adserving was not affected.

Scope of Impact

Between 2019-12-04 19:30 UTC and 2019-12-05 23:00 UTC some members may have experienced creative previews not loading on certain pages of Console and Invest. Between 2019-12-05 20:40 UTC and 2019-12-06 01:00 UTC, users may have seen a “403 permission denied” error.

Timeline (UTC)

2019-12-04 19:30: Incident Started: API release started initial errors

2019-12-05 20:40: Access control release, starting 403 errors within Console to affect even more members

2019-12-05 22:15: Escalated to engineers

2019-12-05 22:50: API release rolled back, resolving “failed to render dynamic banner preview remotely” error

2019-12-06 01:00: Access Control release rolled back, resolve the “403 permission denied” error

Cause Analysis

The incident was caused by two separate releases. The first was an API release which caused a “failed to render dynamic banner preview remotely” error and the second was an access control release that caused the “403 permission denied” error seen in Console/Invest.

Resolution Steps

The issues were resolved by both releases being rolled back.

Next Steps

  • We are working to improve data management for the creative service and implement better detection monitoring
  • Look into notification settings for access control changes
Posted Dec 13, 2019 - 16:37 UTC


The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Dec 06, 2019 - 01:00 UTC

We are currently investigating the following issue:

  • Component(s): Creative Console pages
  • Impact(s):
    • Creative preview unavailable in some Console and Invest Screens. This is not affecting adserving.
  • Severity: Minor Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Dec 05, 2019 - 22:34 UTC