Incident Summary
On 2020-05-14 01:06 UTC, the application was overloaded in SIN3. This application is responsible for dynamically resizing images that are used in native creatives, resulting in native creatives have broken image links when rendered or taking a long time to render. The availability was restored on 2020-05-14 03:19 UTC.
Scope of Impact
Publishers with users in APAC may have seen broken image links or slow render times.
Timeline (UTC)
2020-05-14 01:06: Incident Started: Peak traffic in APAC caused hosts to be overloaded.
2020-05-14 02:44: Alert was paged by on-call engineer
2020-05-14 03:19: Incident Resolved: Reduced traffic
2020-05-01 04:13: Incident Resolved: More capacity added to the application in SIN3
Cause Analysis
We were under-provisioned in SIN3 for the application. At peak load, the hosts were not able to respond to incoming requests in a reasonable time.
Resolution Steps
The immediate resolution was the subsiding of peak traffic. Further resolution was adding more capacity for the application in SIN3.
Next Steps
The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:
From 2020-05-14 01:06 UTC to 2020-05-14 03:19 UTC, publishers may have seen blanks or broken images in native ad slots on their site. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.