Between October 14 01:20 AM to 01:55 AM UTC and October 14 2:40 AM to 02:50 AM UTC there were intermittent failures on an internal DNS server with network name resolution due to migration to new hardware. The other two DNS servers were serving traffic without any issues. 20% requests were affected in this window.
During the incident window, customers would have experienced:
-- difficulties logging into Invest
-- received "request processing error" messages while adding creatives to line items, editing objects (LIs/ IOs).
2020-10-14 01:20 : Incident started
2020-10-14 01:35 : Intermittent name resolution failures observed
2020-10-14 01:50 : Migration of the DNS server to new hardware was in in progress
2020-10-14 02:00 : Escalated to Engineer
2020-10-14 03:30 : Incident resolved.
2020-10-14 19:35 : Retroactive IM ticket created
2020-10-14 19:35 : Retroactive IM ticket resolved
One of the DNS servers experienced name resolution issues while migrating to new hardware.
Our engineers resolved the issue by removing the affected server from the cluster, restarting DNS processes, and re-adding it.
To further validate our maintenance runbooks for the DNS servers.
The following incident has been fully resolved, and we will post a post-mortem as soon as we have completed one:
We apologize for the inconvenience this issue may have caused, and thank you for your continued support.