Video Completion Rate Drop
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 18:00 UTC on July 23rd to 15:00 UTC on July 29th, video completion rates for all inventory were calculated incorrectly.

Scope of Impact

During the incident window, all clients using products relying on video completion rates were affected. Predicted video completion rates trended lower than actual rates, which may have impacted bidding behavior of Line Items and performance on tags.

Timeline (UTC)

2020-07-23 18:00: Incident Started: bad change released to video completion rate pipeline

2020-07-27 20:36: Incident Escalated

2020-07-29 15:00: Incident Resolved: corrected predicted video completion rates pushed to ad-serving platform and client notification posted

Cause Analysis

A configuration change led to video completion rates being incorrectly calculated.

Resolution Steps

Our engineers resolved the issue by correcting video completion rate tables and pushing the corrected values to our ad-serving platform. A new metric and alert was put in place to prevent a recurrence of this issue.

Next Steps

  • Continue improving monitoring for potential issues with video completion rate calculations and other video-related metrics
Posted Aug 03, 2020 - 21:54 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jul 29, 2020 - 20:47 UTC
Monitoring

We have identified and patched the following issue:

  • Component(s): Prediction Optimization
  • Impact(s):
    • Predicted Video Completion Rates were incorrectly calculated, resulting in flux in VCRs
  • Severity: Minor Outage
  • Datacenter(s): Global

We are monitoring our systems closely, and will provide an update as soon as the issue has been fully resolved.

Posted Jul 29, 2020 - 18:50 UTC