Third Party Pixel Creative Page Load Errrors
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 08:29 UTC to 08:54 UTC on Friday, June 19th, 2020 two test pixels were erroneously applied at the platform level, causing disruptions on page load. The pixels were removed but a few intermittent disruptions were flagged, likely due to browser-side caching.

Scope of Impact

During the incident window publishers who allowed 3rd party data would have seen a JavaScript pop-up when creatives were served on-page.

Timeline (UTC)

2020-06-19 08:29: Incident Started: Two pixels were enabled platform-wide

2020-06-19 08:40: Pixel one was disabled platform-wide

2020-06-19 08:54: Pixel two was disabled platform-wide

2020-06-19 08:54: Incident Resolved

Cause Analysis

The root cause of the incident was due to the manual creation of third party pixels within an internal testing seat. A parameter on the pixels was erroneously set, causing the loading of the text message "secure" across creatives that served within the 25 minutes window.

Resolution Steps

The root cause of the issue was resolved by removing the erroneously set parameter on the test pixels. Due to browser-side caching, the message persisted a few hours after resolution.

Next Steps

Engineering has reassessed the option that allowed third-party pixels to apply platform-wide; it is now processing the necessary updates that will lock this feature and prevent it from being triggered by manual updates in the future.

Posted Jun 26, 2020 - 15:40 UTC

Resolved

The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Ad Serving
  • Impact(s):
    • Javascript pop-ups on creative delivery
  • Severity: Partially Degraded
  • Datacenter(s): AMS1, FRA1

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 26, 2020 - 15:37 UTC