Incident Summary

From approximately 12:35 to 13:23 UTC on Wednesday, November 25th, 2020 some impression buses failed to communicate externally through the internet in the SIN3 datacenter. As a result, outgoing bid requests and win notifications to external bidders were blocked.

Scope of Impact

During the incident window impression bus was unable to send bid requests or win notifications to external bidders, which would result in some impression counting discrepancies between Xandr and external partners as well as a drop in demand from external partners.

Timeline (UTC)

2020-11-25 12:35: Incident Started

2020-11-25 13:00: Engineering Notified

2020-11-25 13:10: Root Cause identified.

2020-11-25 13:23: Incident Resolved.

Cause Analysis

The incident was caused by the creation of a new IP subnet, which had not been added to appropriate approved internal lists thereby preventing any requests from that IP.

Resolution Steps

Our engineers resolved the issue by adding the new IP to approved internal lists and communicating the new IP to external partners to ensure it was added to appropriate allowlists.

Next Steps

• Implement more robust monitoring of the traffic from external bidders.
• Beginning on Monday, December 7th, Xandr has begun sending APAC traffic from a new, additional IP address block: 103.43.89.0/24. This will only impact customers with server instances in the APAC region. If you have any traffic filtering that is enabled based on the existing SIN3 data center IP address block, please update your filters to include the new IP address of SIN3: 103.43.89.0/24