Cross-Region Disaster Recovery and Failover on AWS

01 / Recovery goals

Recover the service people use.

Disaster recovery must restore a usable business system within an agreed time. That includes data, applications, dependencies and the path users take to reach them.

RTO · Recovery time objective: The maximum acceptable time to restore service after disruption. Detection, decisions, database takeover, capacity, validation and routing all use this budget.
RPO · Recovery point objective: The maximum acceptable data-loss window, measured in time. The actual recovery point depends on the data available when recovery happens.

Agree these objectives with the business before choosing the design. Validate the complete recovery path, including the database, application dependencies and critical user operations.

One Availability Zone fails

Use resources across AZs within a Region. Check remaining application capacity, database availability and persistent data. An AZ failure does not automatically require rebuilding the entire EKS cluster.

An entire Region is unavailable

The recovery environment must be elsewhere. This example uses separate EKS clusters and VPCs in two Regions. Regional recovery complements the AZ-level resilience designed inside each Region.

02 / Strategy & trade-offs

Pay for readiness where it matters.

Backup and restore is a valid DR strategy. Warm standby prepares more of the system in advance to reduce the work left after an outage.

Lower ongoing cost

Backup and restore

Keep recoverable backups, code, images and access available in another Region. After failure, create infrastructure, restore data, deploy the application and validate it before directing users there.

There is less running infrastructure to pay for, but more recovery work and greater dependence on provisioning services during the incident.

The main example

Warm standby

Keep a smaller application environment running in the recovery Region and continuously replicate database changes. On takeover, make the database writable, add capacity and verify the service.

Readiness costs more. A very small standby saves money but depends more on scaling during recovery. A shorter RTO may require more capacity already running.

Replication does not replace backups. An accidental deletion or bad update can reach the replica too. Keep retained recovery points and test restoring them. The achievable RPO from a backup can differ from the replica’s recovery point.

03 / Example architecture

A smaller environment, ready to take over.

Each Region has its own VPC, ALB and EKS application environment. The primary database replicates asynchronously to an RDS PostgreSQL read replica in the recovery Region. The standby application is deployed and running at reduced capacity; write paths and scheduled writers stay inactive until takeover.

Illustrative warm standby architecture. Users normally reach the primary ALB, EKS application and writable PostgreSQL database. A separate Region contains its own ALB, smaller EKS environment and an asynchronous read replica. After fencing old writers, promoting the replica, scaling and validating the application, DNS directs new traffic to recovery. Separate backup copies are retained in the recovery Region. — Conceptual AWS design. Line styles distinguish normal traffic, takeover traffic, replication and backup copies. The view focuses on regional responsibilities and the recovery sequence. Open the full-size diagram

Keep both recovery paths available

Promote the replica for regional takeover when its data is suitable. For corruption or deletion, restore a known-good backup instead. A snapshot restore creates a separate DB instance and needs its own connection and configuration checks.

Make backup access independent

Copy backups to the recovery Region before an outage. Preserve retention and restrict deletion. Verify that recovery identities can read and decrypt them without relying on the failed Region’s keys or services.

04 / Component responsibilities

Terraform prepares infrastructure. Recovery coordinates the system.

Terraform: Build and maintain the regional networks, EKS resources, load balancers and access configuration in advance. Reuse modules with explicit Region-specific provider settings and inputs. Separate state per environment can help contain changes; it is a design choice, not a Terraform requirement.
Database service: Replicate changes, create and restore backups, and promote the recovery replica. Terraform can declare resources and settings; RDS performs these data operations.
Delivery process: Keep application versions and configuration ready in recovery. Update database connections when needed and control background workers during takeover.
Recovery procedure: Coordinate the decision, writer isolation, database takeover, capacity, user-flow checks and routing. Assign an owner to each step and define when to pause.

Terraform state is not a business-data backup. It records managed infrastructure and its relationship to configuration. One terraform apply does not complete a disaster recovery operation. Copying state and changing only the Region is not a safe way to clone an environment.

Prepare for losing access to the primary Region.

Code and state: configuration, modules, provider versions, remote state, locking and a usable runner must remain accessible. Avoid two active writers to the same state during recovery.
Images and configuration: required container images and application settings must be available to the recovery cluster. Test access without the primary registry or deployment runner.
Backups and access: retain backup copies, recovery credentials, certificates and destination-Region KMS access. Test the permissions and decryption.
Dependencies and capacity: account for identity services, queues, object data and external systems. Include their recovery requirements in the runbook and check quotas, node capacity and database limits.

After emergency changes outside Terraform, reconcile configuration, state and actual resources. Review the next plan before applying it so normal automation does not undo the recovery.

05 / Failover sequence

Establish one writer. Validate. Then move traffic.

The recovery decision path

1Confirm impact
2Database takeover & capacity
3Validate the application
4Switch traffic
5Keep verifying

Confirm the failure and make the decision.

Check the affected services and Region. Compare the likely outage with the recovery objectives. The incident owner authorises takeover and records the decision; a failed health endpoint alone is not enough.
Establish one writer and prepare recovery capacity.

Treat an unreachable primary as potentially still running. Use a pre-tested way to isolate its write paths, including workers and scheduled jobs. Keep it fenced when it returns. If exclusive write ownership cannot be established, pause and make that risk explicit before proceeding.

Check replica lag and the latest available data. Asynchronous replication can leave recent transactions behind. If the source or current lag cannot be checked, report the uncertainty; do not declare zero data loss.

Promote the recovery RDS PostgreSQL read replica and wait for it to become available for writes. Promotion stops its old replication relationship and involves a reboot. Point the application at the recovery database endpoint, updating configuration if it still refers to the former primary.

Scale EKS nodes and application replicas, check database capacity and confirm dependencies are ready. Keep general user traffic away until checks pass.
Verify the service before opening the public route.

Use a controlled recovery endpoint to test critical user operations, application dependencies and database reads and writes. Confirm that writes persist in the promoted database. Check workers can run without creating duplicate work.
Switch the entry point after validation.

Only after the readiness checks pass, change the Route 53 records or enable the recovery target in the routing procedure. Clients then connect to the recovery ALB. Allow for cached DNS answers and existing connections during the transition.
Keep verifying the recovered service.

Observe errors, latency, database connections and load under real traffic. Confirm worker ownership, data consistency and backup health. Record the recovery time and recovery point, including uncertainty or data loss that still needs investigation.

DNS is only the entry-point switch. Route 53 does not promote RDS, stop old connections or disable background jobs. DNS caches and existing connections can delay the move, so old writers must remain fenced throughout the transition. This design uses a readiness gate rather than routing to an unprepared standby.

06 / Rehearsal & failback

Test the whole journey, including the return.

Rehearse with the primary Region unavailable to the recovery process. Time the journey from disruption to usable service, including decisions and validation. Compare the latest recovered transaction with the pre-failure data to establish the recovery point. Record results against the agreed RTO and RPO.

Validate readiness under load

Exercise critical user paths, read and write data, and test the standby at the expected traffic level. Include the time needed to add capacity. Restore backups separately and test code, state, image and credential access.

Keep the recovery Region healthy

After takeover, check backups, monitoring, job ownership and data consistency. Update the recovery procedure with failed steps and measured timings before relying on it again.

Failback is another controlled migration.

Keep the recovered Region authoritative until the original Region has current data and a verified environment. Rebuild or resynchronise the original database using a suitable replication or restore process. RDS read-replica promotion does not reverse replication automatically.

Plan a controlled handover: stop or fence the current writers, complete the agreed final data sync, validate the destination and then move traffic. Confirm a single writer again and monitor the return. Changing DNS back as soon as a Region recovers risks serving stale data or creating conflicting writes.

07 / Official references

Sources behind the design.

AWS and HashiCorp guidance for the service choices and recovery sequence, checked on 14 October 2026.

AWS · Disaster recovery strategies
Backup and restore, warm standby, replication and cost trade-offs.
AWS · Plan for disaster recovery
Recovery objectives, testing and recovery procedures.
AWS · Promote an RDS read replica
Write takeover, asynchronous replication and promotion behaviour.
AWS · Restore an RDS snapshot
A separate recovery path that creates a DB instance from a backup.
AWS · Route 53 failover routing
DNS routing between primary and recovery endpoints.
HashiCorp · Provider configuration
Explicit provider settings for regional resources and reusable modules.
HashiCorp · The purpose of Terraform state
How state maps configuration to managed infrastructure.

Back to technical deep dives