NETWORK INCIDENTS

Incident file

Breakdown
AWS · DNS automation

Two copies of AWS's DNS automation raced, and DynamoDB's regional endpoint was left with an empty DNS record

Happened ·Published

AWS's N. Virginia (us-east-1) Region was disrupted from 11:48 PM PDT on 19 October 2025 to 2:20 PM PDT the next day, in three distinct periods of impact. It began when a race condition in DynamoDB's DNS automation left the service's regional endpoint with an empty DNS record, so systems connecting through it got DNS failures. DNS was restored within three hours, but the knock-on effects on EC2 instance launches and Network Load Balancer lasted into the afternoon.

Incident replay · Watch it rebuilt in the lab

A lab model of the failure pattern, not AWS's systems. Watch on YouTube · Run this lab yourself (setup)

What users saw

  • Increased DynamoDB API error rates. Customers and other AWS services that depend on DynamoDB couldn't make new connections to it, from 11:48 PM to 2:40 AM PDT.
  • DynamoDB global tables kept working through replicas in other Regions, with prolonged replication lag to and from us-east-1.
  • From 11:48 PM to 1:50 PM PDT, EC2 had increased API errors and latencies, and new instance launches failed with "request limit exceeded" or "insufficient capacity" errors. Instances launched before the event began stayed healthy throughout.
  • Some Network Load Balancers had increased connection errors, from 5:30 AM to 2:09 PM PDT.
  • Signing in to the AWS Management Console with an IAM user had increased authentication failures until 1:25 AM PDT. Lambda, ECS, EKS, Fargate, Amazon Connect, STS and Redshift were also affected, some into the afternoon; the last Redshift clusters were restored at 4:05 AM PDT on 21 October. AWS points to its event history for the full list of affected services.
  • It reached beyond us-east-1. From 11:47 PM to 1:20 AM, Redshift customers in all AWS Regions couldn't run queries with IAM user credentials. Customers signing in to the console with their root credentials, or with identity federation configured to use signin.aws.amazon.com, got errors in Regions outside us-east-1.

What broke

  1. DNS PLANNERPlansA component called the DNS Planner watches the load balancers' health and capacity, and periodically creates a new DNS plan for each DynamoDB endpoint: a set of load balancers and weights.
  2. 3 ENACTORSThree appliersDNS Enactors apply the plans in Route 53, running independently in three Availability Zones. Before applying a plan, an Enactor checks once that it is newer than the plan already applied.
  3. 11:48 PM PDTThe raceOne Enactor was unusually delayed, retrying several endpoints. Meanwhile another applied a much newer plan, then started a clean-up of plans significantly older than it. At the same time, the delayed Enactor applied its much older plan to the regional endpoint, overwriting the newer one: its "newer" check was stale by then.
  4. EMPTY RECORDThe live plan is deletedThe clean-up then deleted that older plan, now the live one, and all IP addresses for the regional endpoint were removed. The system was left unable to apply new plans, and needed manual operator intervention.
  5. RESULTNothing could find DynamoDBEvery system connecting to DynamoDB in us-east-1 through the public endpoint got DNS failures, including AWS's own internal services that rely on it.

Why it lasted into the afternoon

DNS was restored by 2:25 AM PDT, but EC2's DropletWorkflow Manager (DWFM), which holds a lease on each physical server behind EC2 and depends on DynamoDB, had been seeing those leases slowly time out. When it tried to re-establish them, there were so many servers that the work couldn't finish before the leases timed out, and DWFM entered what AWS calls congestive collapse.

From 4:14 AM engineers throttled incoming work and began selective restarts of DWFM hosts, and leases were back by 5:28 AM. Network Manager then had a backlog of network configuration to push, and from 6:21 AM propagation slowed, so newly launched instances could start without network connectivity. Propagation was back to normal by 10:36 AM; engineers began relaxing request throttles at 11:23 AM, and EC2 was fully recovered at 1:50 PM.

That delay meant Network Load Balancer health checks sometimes failed on healthy nodes, so results alternated and nodes were removed from DNS and returned again. The extra load degraded the health check subsystem and triggered automatic AZ DNS failover, taking capacity out of service for multi-AZ load balancers; applications saw connection errors when the remaining capacity was too small. Engineers disabled automatic health check failover at 9:36 AM, which resolved the errors, and re-enabled it at 2:09 PM once EC2 had recovered.

Timeline (PDT)

  1. DynamoDB's regional endpoint is left with an empty DNS record; DNS failures and API errors begin.
  2. Engineers identify DynamoDB's DNS state as the source of the outage.
  3. Temporary mitigations let some internal services connect to DynamoDB, and repaired key internal tooling that unblocked further recovery.
  4. All DNS information restored; customers can connect again as cached records expire, by 2:40 AM.
  5. With EC2's DWFM stuck in congestive collapse, engineers throttle incoming work and begin selective restarts of DWFM hosts.
  6. DWFM leases re-established; new EC2 launches start to succeed, many still throttled.
  7. Some Network Load Balancers begin to see increased connection errors.
  8. Network Manager's propagation of network configuration to new instances slows.
  9. Monitoring detects the Network Load Balancer health check failures.
  10. Automatic health check failover for NLB disabled; healthy nodes back in service, resolving the connection errors.
  11. Network configuration propagation back to normal; new EC2 launches working normally, though some new instances still had connectivity issues until 1:50 PM.
  12. All EC2 APIs and new instance launches operating normally.
  13. Automatic DNS health check failover for NLB re-enabled; the NLB impact window ends.
  14. The event ends, by AWS's account; some Redshift clusters stay impaired until 4:05 AM PDT on 21 October.

The networking lesson

  • Fix the automation before turning it back on. AWS disabled the DNS Planner and Enactor automation worldwide, and says it will fix the race condition and add protections against applying incorrect DNS plans before re-enabling it.
  • Limit how fast automation can remove capacity. For Network Load Balancer, AWS is adding a velocity control that limits how much capacity one load balancer can remove when health check failures cause failover.
  • Test the recovery path too. For EC2, AWS is building a test suite that exercises the DWFM recovery workflow, and will rate-limit incoming work in its data propagation systems based on how much is already waiting.
  • An empty answer is still an answer. In general, DNS will serve a name with no addresses without complaint. The check has to live in whatever writes the records: it can refuse to publish an empty set for a live service. our analysis
  • A clean-up job needs its own "is this still current?" check. The apply relied on a check made long before the write, and the clean-up deleted by age a plan that had by then become the live one. Re-check right before the change that can't be undone. our analysis
  • Recovery can be its own outage. When a dependency comes back, everything that waited on it retries at once. Plan for the backlog, not just the fix. our analysis

Source: Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region, Amazon Web Services (official post-event summary). Facts, times and causes are as the source states them; anything marked our analysis is ours.

More incident breakdowns

  • Cloudflare, 22 Jan 2026Removing a prefix-list left Cloudflare's Miami export policies too permissive, and IPv6 routes leaked to some of its peers and providers
  • Cloudflare, 18 Nov 2025A database permissions change doubled a config file, and Cloudflare's proxy fell over