What happens when the cloud itself goes down? That question sounds like a riddle, but it has real answers. The cloud runs on physical buildings, fiber cables, power grids, and software written by humans. Every one of those can fail, and each has failed in public, at scale, with thousands of businesses watching their dashboards turn red.
The most useful cloud disaster recovery examples are not the marketing case studies. They are the post-incident reports, the outages that cascaded across services, and the companies that survived because they had rehearsed the bad day in advance. This article walks through several of those events, explains what each one teaches about RTO (recovery time objective) and RPO (recovery point objective), and shows how to turn those lessons into a plan your team can actually run.
Why real incidents beat hypothetical scenarios
Disaster recovery planning has a credibility problem. A plan that lives in a shared drive and says “fail over to the secondary region” looks fine until someone asks who has the credentials, whether the database replication is actually running, and how long DNS takes to propagate.
Real outages answer those questions brutally. They also reveal a pattern: most cloud failures are not caused by a meteor strike on a data center. They are caused by configuration changes, capacity limits, operator error, and dependencies nobody mapped. A recovery plan built around the wrong threat model is theater.
One practical starting point is knowing exactly what you own before you plan for its loss. A structured inventory of systems, data flows, and dependencies is the foundation of every plan that works under pressure, and the checklist in this guide to what to inventory before a cloud migration applies just as well to recovery planning.
Example 1: The S3 outage that took down half the internet
In February 2017, an operator debugging the billing system at Amazon Web Services typed a command that removed more servers than intended. The result was a multi-hour outage of S3 in the us-east-1 region, and because so many other services depended on S3, the failure cascaded into status dashboards, launch tools, and a long list of consumer apps.
The lesson is not “S3 is unreliable.” The lesson is dependency depth. Teams had built architectures that assumed object storage was always available, including the very tools they needed to communicate during an incident. Recovery plans that depend on the broken system are not recovery plans. (9481.org)
Practical fixes that came out of this era:
- Keep incident communication tools independent of the primary cloud provider.
- Cache or replicate critical static assets so a storage outage degrades the site rather than deleting it.
- Test what happens when your monitoring and your production share a failure domain.
Example 2: A single region, a single point of failure
In June 2019, a configuration change during scheduled maintenance triggered an outage in Google Cloud’s us-central1 region. Services well beyond Google’s own portfolio felt it, including parts of popular consumer apps. The technical trigger was mundane. The blast radius was not.
This is the classic single-region risk. Many teams run multi-zone, which protects against the failure of one data center, and then assume they are resilient. Zones share a region. A regional control-plane problem can take out every zone at once.
The distinction matters when you write your recovery objectives:
| Failure scope | Typical protection | What it does not cover |
|---|---|---|
| Single server | Auto-replacement, health checks | Anything above the instance |
| Single availability zone | Multi-zone deployment | Regional control-plane failure |
| Single region | Multi-region failover | Data consistency and DNS lag |
| Single provider | Multi-cloud or hybrid | Operational complexity and cost |
The last row is where most arguments start. Splitting workloads across accounts or providers adds real resilience, but it also adds identity management, networking, and billing complexity. The trade-offs are worth understanding before committing, and this explanation of when to split workloads across cloud accounts walks through the decision honestly.
Example 3: When the provider’s own capacity runs out
In November 2020, a front-end fleet for Amazon Kinesis in us-east-1 exceeded its thread limits, and the resulting failure rippled into services including Cognito and CloudWatch. Months later, a capacity issue in the same region affected a different set of dependent services.
The pattern here is shared fate through managed services. When you use a managed queue, identity provider, or monitoring service, you inherit its availability. That is usually a good trade. It becomes a bad trade when your recovery runbook assumes the managed service will be there to help you recover.
Three habits that reduce this risk:
- Document every managed service your recovery path depends on, including the ones you do not think of as infrastructure.
- Keep a break-glass procedure that does not require the identity provider that might be down.
- Store runbooks somewhere readable without your production stack, such as a printed copy or an offline wiki.
Example 4: The fire that erased a data center
In March 2021, a fire destroyed a data center at OVHcloud’s Strasbourg site and damaged neighboring buildings. Some customers who had no off-site copies lost data permanently. The event became a reference point for everyone who had quietly treated “the cloud” as someone else’s redundancy problem. (A clearer way to track cloud data transfer costs)
This is the clearest possible argument for the 3-2-1 rule: three copies of data, on two different media, with one copy off site. In cloud terms, “off site” means a different provider, a different region with independent infrastructure, or an on-premises copy. A snapshot in the same account is not a backup. A replication target in the same region is not a backup.
The harder question is whether your backups actually restore. A backup that has never been tested is a hope, not a control. Building a scheduled recovery drill for a managed database turns that hope into evidence, and the same discipline applies to object storage, configuration, and identity.
Example 5: Weather, geography, and the limits of “the cloud is everywhere”
Hurricane Sandy in 2012 flooded Lower Manhattan and knocked out data centers and power in a dense cluster of facilities. Firms that had assumed geographic diversity discovered that their “diverse” sites were a few blocks apart, sharing the same flood plain and the same power substation.
Geography is a real dependency, and cloud regions inherit it. Two regions in the same metro area may share power infrastructure, network routes, and even the same weather event. When you evaluate a region for disaster recovery, ask where the fiber enters, where the power comes from, and how far apart the sites truly are. Reading a cloud region announcement carefully means looking past the headline location to the infrastructure claims underneath.
What the examples have in common
Strip away the specifics and the same four failures repeat:
- Configuration and change management. Most incidents start with a well-intentioned change.
- Hidden dependencies. The recovery tool depends on the thing that broke.
- Capacity assumptions. Limits exist, and they get hit during traffic spikes and failovers.
- Untested plans. The runbook was written once and never executed.
Against those, the countermeasures are unglamorous but effective. Define RTO and RPO per system, not per company. Write runbooks that a tired engineer can follow at 3 a.m. Rehearse failover on a schedule. Keep an offline copy of the plan. Review dependencies quarterly, because they drift.
How to build your own example list
The best disaster recovery examples are the ones from your own environment. A useful exercise is to write down every incident in the past two years, including the small ones, and ask what would have happened if it had lasted eight hours instead of forty minutes.
Then map each system to a recovery tier:
| Tier | Example systems | Target RTO | Target RPO |
|---|---|---|---|
| Critical | Payments, authentication | Under 1 hour | Near zero |
| Important | Internal APIs, reporting | Under 4 hours | Under 1 hour |
| Standard | Marketing sites, dev tools | Under 24 hours | Under 24 hours |
| Deferrable | Archives, sandboxes | Days | Days |
The tiers force a conversation about money. Near-zero RPO costs real currency in replication, engineering time, and complexity. Most organizations discover that only a handful of systems genuinely need the top tier, and that spreading the top tier everywhere is how recovery budgets get cut entirely.
A mature program also tracks recovery confidence rather than recovery intent. If the last successful failover test was eighteen months ago, the plan is a document, not a capability. Architecture decisions and recovery testing belong together, which is why teams that treat cloud infrastructure and architecture as a single discipline tend to recover faster than teams that treat DR as a separate project. (What is disaster recovery? (AWS overview))
Conclusion
The cloud does not remove disaster recovery. It relocates it. The failures that matter most are rarely exotic: a typed command, a configuration push, a capacity ceiling, a fire in a building, a storm that ignores your architecture diagram. What separates organizations that recover in an hour from those that lose data permanently is rarely heroism. It is a short list of tested runbooks, honest recovery objectives, and the discipline to keep backups somewhere the primary failure cannot reach.
Start with one system. Write the runbook. Run the drill. Then do it again next quarter.
Frequently asked questions
What is the difference between RTO and RPO?
RTO is the maximum acceptable time to restore a service after an outage. RPO is the maximum acceptable amount of data loss, measured in time. A system with a one-hour RTO and a fifteen-minute RPO should be back online within an hour and lose no more than fifteen minutes of transactions. Both should be set per system, based on business impact, not on what sounds impressive.
Can a cloud provider’s outage be covered by their SLA?
Service level agreements typically provide service credits, not compensation for lost revenue. They also usually require you to request credits and to have followed the provider’s guidance. Treat an SLA as a refund policy for availability shortfalls, not as a disaster recovery plan. Your own architecture and backups remain your responsibility.
How often should disaster recovery drills run?
At least annually for every critical system, and quarterly for the top tier. Drills should include everyone who would actually respond, beyond the platform team, and should test the parts most likely to fail: credentials, DNS changes, and data restoration. A drill that only confirms a backup exists has tested almost nothing.
Is multi-cloud necessary for disaster recovery?
No. Multi-cloud adds genuine resilience against provider-wide failures, but it also multiplies operational complexity, skills requirements, and cost. Many organizations get sufficient protection from multi-region deployments within one provider, combined with tested backups stored outside that provider. Choose based on your actual risk tolerance and your team’s capacity to operate what you build.
What is the 3-2-1 backup rule in cloud terms?
Three copies of data, on two different types of storage, with one copy off site. In practice that might mean live data in a primary region, snapshots in a second region, and an immutable copy in separate storage or on premises. The rule exists to survive the loss of an entire facility, which is exactly what happened in Strasbourg in 2021.
What should a recovery runbook contain?
A runbook should name the trigger conditions, the decision maker, the exact commands or console steps, the verification checks, and the rollback path. It should also state what not to do. Store it somewhere accessible without the production environment, keep the credentials it needs in a break-glass vault, and revise it after every drill and every real incident.
