Database lag limiting access to created objects
Monitoring -

We have mitigated the issue but are operating at a slightly reduced capacity. This might cause slower responses. However, adserving should not be affected. We apologize for the inconvenience this issue may have caused and thank you for your patience.


Oct 16, 21:37 UTC
Identified -

We have identified the following issue:

  • Component(s): Console API, Console UI
  • Impact(s):
    • Possible UI and API unable to access created objects, may result in unexpected UI and API errors
  • Severity: Minor Outage
  • Datacenter(s): Global

This is related to the issue we identified on 10/14. We have restored database capacity, but are still seeing elevated errors caused by queries timing out. Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.


Oct 16, 13:05 UTC
System Metrics Month Week Day
Ad Server Uptime
Fetching
API Uptime
Fetching
Reporting API Uptime
Fetching
Console UI Uptime
Fetching
Ad Server Data Freshness
Fetching
Ad Server Uptime
Fetching
API Uptime
Fetching
Reporting API Uptime
Fetching
Console UI Uptime
Fetching
Ad Server Data Freshness
Fetching
Ad Server Uptime
Fetching
API Uptime
Fetching
Reporting API Uptime
Fetching
Console UI Uptime
Fetching
Ad Server Data Freshness
Fetching
Ad Server Uptime
Fetching
API Uptime
Fetching
Reporting API Uptime
Fetching
Console UI Uptime
Fetching
Ad Server Data Freshness
Fetching
Ad Server Uptime
Fetching
API Uptime
Fetching
Reporting API Uptime
Fetching
Console UI Uptime
Fetching
Ad Server Data Freshness
Fetching
Analytics Report Delay
Fetching
CSA Tool Uptime
Fetching
CSA Tool API Uptime
Fetching
Past Incidents
Oct 22, 2019
Bidder Issue Causing Missing Impressions
Resolved -

We are aware of an issue that caused a small percentage of missing bid rows from 2019-10-21 16 to 21 UTC. Our team is investigating available follow-up actions, which will be detailed in the post mortem. Please see below details:

  • Component(s): Bidding, Billing reports, Analytics reports
  • Impact(s):
    • Data unavailable through affected service(s)
  • Severity: Partially Degraded
  • Datacenter(s): Global

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.


Oct 22, 14:34 UTC
Temporary bidder degradation in AMS1 and FRA1
Resolved -

The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:

  • Component(s): Bidding
  • Impact(s):
    • Increase in defaults and PSAs for Console customers
    • Drop in delivery on external supply
  • Severity: Partially Degraded
  • Datacenter(s): FRA1, AMS1

We apologize for the inconvenience this issue may have caused, and thank you for your continued support.


Oct 22, 09:39 UTC
Oct 21, 2019

No incidents reported.

Oct 20, 2019

No incidents reported.

Oct 19, 2019

No incidents reported.

Oct 18, 2019

No incidents reported.

Oct 17, 2019
QPS Limits Not Being Respected
Resolved -

This incident has been resolved and all QPS should return to expect levels within the next hour. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.


Oct 17, 21:35 UTC
Identified -

We have identified the following issue:

  • Component(s): Ad Serving
  • Impact(s):
    • QPS Limits are being exceeded for requests sent to External Bidders & Real-Time Data Providers
  • Severity: Minor Outage
  • Datacenter(s): Global

Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.


Oct 17, 18:53 UTC
Oct 15, 2019
Database lag limiting access to created objects
Resolved -

This issue has been resolved but we continue to operate at a slightly reduced capacity and expect to return to full capacity within 24 hours. We apologize for the inconvenience this issue may have caused and thank you for your patience.


Oct 15, 00:46 UTC
Monitoring -

We have patched the issue but are operating at a slightly reduced capacity. This might cause slower responses. However, adserving should not be affected. We apologize for the inconvenience this issue may have caused and thank you for your patience.


Oct 14, 02:15 UTC
Identified -

We have identified the following issue:

  • Component(s): Console API, Console UI
  • Impact(s):
    • Possible UI and API unable to access created objects, may result in unexpected UI and API errors
  • Severity: Minor Outage
  • Datacenter(s): Global

Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.


Oct 13, 22:42 UTC
Oct 14, 2019
Scheduled Database Maintenance with Downtime
Completed - The scheduled maintenance has been completed.
Oct 14, 14:40 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 14, 13:30 UTC
Scheduled - We are performing scheduled database maintenance on Monday, October 14th, 2019 from 13:30 UTC to 14:40 UTC.

Please note that during the maintenance period, Cloud Exports will not be configurable and the Log-level Data UI may return errors.

Please note that there will be no interruption to ad-serving or reporting data collection during this maintenance. If you have any questions, please contact us by creating a Xandr Support ticket.
Oct 9, 16:03 UTC
Oct 12, 2019
Scheduled Database Maintenance with Downtime
Completed - The scheduled maintenance has been completed.
Oct 12, 13:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 12, 12:00 UTC
Scheduled - We are performing scheduled database maintenance on Saturday October 12th, 2019 from 12:00 UTC to 13:00 UTC.

Please note that during the maintenance period the Console UI and Console API services may return an error. If you receive an error, please retry your request after the maintenance window has ended.

Please note that there will be no interruption to ad-serving or reporting data collection during this maintenance.
Oct 9, 16:01 UTC
Oct 11, 2019

No incidents reported.

Oct 10, 2019

No incidents reported.

Oct 9, 2019
Database lag limiting access to created objects
Postmortem - Read details
Oct 14, 18:43 UTC
Resolved -

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.


Oct 9, 16:45 UTC
Monitoring -

We have identified and patched the following issue:

  • Component(s): Console API, Console UI
  • Impact(s):
    • Possible UI and API unable to access created objects, may result in unexpected UI and API errors
  • Severity: Minor Outage
  • Datacenter(s): Global

We are monitoring our systems closely, and will provide an update as soon as the issue has been fully resolved.


Oct 9, 15:23 UTC
New Scheduled Maintenance and Downtime Notification System
Completed - The scheduled maintenance has been completed.
Oct 9, 03:43 UTC
Scheduled - As of today, Monday October 7, 2019, all scheduled maintenance and downtime notifications will be sent using our Xandr Statuspage at status.xandr.com. All contacts previously receiving email notifications for scheduled maintenance and/or downtime have been automatically enrolled in the Statuspage notification system. At status.xandr.com you can access your subscription settings (mobile, email, etc.), view past maintenances, incidents, and post mortems, and check system uptime stats in real-time.

If you have any questions, please contact us by creating a Xandr Support ticket at help.xandr.com. This message marks the completion of the migration to Statuspage.
Oct 7, 21:06 UTC
Oct 8, 2019

No incidents reported.

← Incident History Powered by Statuspage