What happens if the backup you have trusted for years turns out to be unreadable on the one day you actually need it? That question makes most IT teams uncomfortable, and for good reason. Cloud backup feels solved. You point an agent at a bucket, a schedule runs nightly, a green checkmark appears in a dashboard, and everyone moves on. The problems hide underneath that green checkmark. (How to read a cloud region announcement)
This article walks through the failures that show up again and again across organizations of every size, from ransomware reaching backup repositories to egress bills that dwarf the original storage cost. It also covers what actually works in practice, including the configuration habits that separate a backup that restores from a backup that merely exists. (Backup)
Why “the backup ran successfully” tells you almost nothing
A successful backup job confirms one thing: data moved from point A to point B. It says nothing about whether that data can be reassembled into a working system. Practitioners learn this the hard way. A job can report success while silently skipping locked files, while writing a corrupted archive, or while capturing a database mid-transaction in a state that will never mount cleanly.
The fix is unglamorous. Treat restore testing as a separate, scheduled workload with its own owner, its own success criteria, and its own place in the on-call rotation. A backup that has never been restored is a hypothesis, not a safety net. Many teams discover during their first serious test that the runbook is out of date, the decryption keys live in a spreadsheet owned by someone who left, and the restore takes four times longer than the recovery time objective allows.
Problem one: ransomware now targets the backups themselves

The oldest assumption in data protection is that attackers go after production and ignore the copies. That assumption is dead. Modern intrusion campaigns deliberately hunt for backup infrastructure, credential stores, and management consoles, because destroying the copies is what converts an outage into a ransom negotiation.
The practical countermeasures are well documented. Official guidance on ransomware response recommends keeping offline, encrypted backups and, critically, testing restores rather than assuming them. The CISA #StopRansomware guide is worth reading in full because it treats backup integrity as a security control, not an IT chore.
What works in practice:
- Keep at least one copy that is physically or logically unreachable from production credentials.
- Use separate identities for backup administration, with no path from a compromised production account to backup deletion rights.
- Turn on object storage versioning and immutability features so that even a privileged attacker cannot quietly overwrite history.
- Alert on deletion events in backup storage. Nobody should be deleting backup objects at 3 a.m. without a change ticket.
The most common mistake here is treating the backup console as just another internal tool. If a stolen laptop with cached credentials can reach the console, the backups are effectively online and writable.
Problem two: the 3-2-1 rule gets quoted but not implemented
3-2-1 means three copies of the data, on two different media types, with one copy off-site. It is the baseline everyone cites and a surprising number of teams quietly fail. The pattern is familiar: production storage, snapshots on the same array, and a replication target in the same cloud account. That looks like three copies. It is really one failure domain wearing three hats.
Modern practice extends the rule to 3-2-1-1-0: one copy offline or immutable, and zero errors verified on restore testing. The extra digits matter more than the original three, because they force the team to prove the copy is usable.
A quick self-audit that takes ten minutes:
| Question | Weak answer | Strong answer |
|---|---|---|
| Where do copies live? | Same account, same region | Separate account, separate region, one immutable |
| Who can delete them? | Anyone with admin | A break-glass role with alerting |
| When was the last restore? | “Last year sometime” | Documented, within the test cycle |
| Are keys stored separately? | In the backup tool | In a managed key service with escrow |
If two or more rows land in the weak column, the backup strategy is decorative. For teams mapping dependencies before changing anything, a structured inventory of what must be protected is the sensible starting point, and this guide on what to inventory before a cloud migration covers the same groundwork.
Problem three: restore is the weak link, and everyone knows it
Ask a room of engineers how confident they are in their restores and watch the body language. Restore testing is the task that gets deferred when the sprint is full, which is always. Vendor-sponsored surveys across recent years have repeatedly reported that a large share of organizations either fail restore tests or skip them entirely. The exact percentages swing wildly between vendors and methodologies, so treat any single figure as indicative rather than precise. The direction of the finding, however, is consistent and matches what most practitioners see.
What actually works is making restore testing cheap enough to do often. That means:
- Restoring to an isolated network or sandbox account, not over production.
- Automating a small canary restore nightly and a full restore quarterly.
- Recording the wall-clock time for each restore and comparing it to the stated recovery objectives.
- Keeping a written runbook that a tired on-call engineer can follow at 2 a.m. without tribal knowledge.
The single most useful metric is time to first byte of recovered data, not the size of the backup repository. A team that can restore a critical service in forty minutes beats a team with a petabyte of unverified archives.
Problem four: egress and retrieval costs ambush the budget
Storage is cheap. Getting data back out is where the invoice gets interesting. Cloud providers charge for data leaving their network, and a full restore of a large repository can move terabytes across that boundary in a single afternoon. A disaster recovery event can therefore arrive with a bill attached.
This is not a reason to avoid cloud backup. It is a reason to model retrieval cost before the incident, not after. Practical steps include knowing the current egress pricing structure for your provider, keeping a rough cost estimate per terabyte restored in the runbook, and checking whether the provider offers any free transfer allowance that changes the math for smaller restores. Tracking how transfer costs accumulate across services is its own discipline, and this explanation of how to track cloud data transfer costs is a useful companion.
The common mistake is budgeting for storage growth and forgetting retrieval entirely. Finance approves the monthly storage line and never sees the scenario where a regional failover triples it.
Problem five: account lockout and provider-side deletion
Cloud data lives behind an account. Lose control of that account, and the data may as well not exist. A lapsed payment method, a suspended subscription, a lost administrator identity, or a botched tenancy migration can all put backups out of reach. Terms of service govern how long data survives after an account closes, and those windows are shorter than most people assume.
Mitigations are administrative rather than technical:
- Keep at least two human administrators on the billing and root accounts, with hardware-backed second factors.
- Store recovery codes somewhere that is not the same system being protected.
- Read the retention terms for the specific plan, because they differ between tiers and providers.
- Test account recovery procedures the same way you test data recovery.
This is also where splitting workloads across accounts pays off. A dedicated backup account with its own billing relationship and its own administrators is harder to take down by accident. The reasoning behind that separation is covered in this piece on when to split workloads across cloud accounts.
Problem six: encryption keys that nobody can find
Client-side encryption is the right default. It also creates a new single point of failure: the key. Teams encrypt backups, rotate credentials, and then discover during an incident that the key material lived in the same account that was compromised or deleted.
Key management is a lifecycle problem, not a one-time configuration. The NIST Special Publication 800-57 series on key management is the reference standard for how keys should be generated, stored, rotated, archived, and destroyed, and it is worth skimming even if you never implement it formally. (NIST Cybersecurity Framework)
In practice, the workable pattern looks like this:
- Generate keys in a managed key service, not in application config.
- Keep an offline escrow copy of the recovery key in a physically secure location.
- Document which key encrypts which dataset, with dates.
- Test decryption as part of the restore drill, not as a separate exercise.
A backup encrypted with a lost key is not a backup. It is random data occupying storage you are still paying for.
Problem seven: compliance and residency requirements arrive late
Regulatory obligations shape where backups may live and what must be provable about them. The General Data Protection Regulation, in force since 2018, constrains transfers of personal data outside the European Economic Area and requires that recovery capability be demonstrable rather than asserted. Sector rules add their own retention and location constraints on top.
The recurring failure is architectural: a backup design is built first and mapped to compliance obligations afterward, at which point the storage region, the subprocessor list, and the retention schedule are all wrong. Getting the obligations onto the whiteboard before the bucket is created saves months.
Problem eight: configuration drift quietly disables protection
Immutability, versioning, and lifecycle rules are powerful and easy to leave switched off. A new bucket created by a script that predates the policy, a lifecycle rule that expires objects after thirty days because someone copied a template, a retention lock applied to the wrong prefix: each of these turns protection into theater.
The countermeasure is treating backup configuration as code. Version it, review changes, and run a periodic check that compares live configuration against the intended baseline. Drift detection catches the bucket that was created last Tuesday without immutability long before an incident does.
Configuration is also where cost and reliability meet. A cost review that engineers actually read tends to surface orphaned snapshots, duplicate replication, and retention settings that keep data far longer than policy requires. The approach described in this guide to making cloud cost reviews useful to engineers is a good model, and the broader material on cloud infrastructure and architecture at 9481.org covers how these controls fit together. (9481.org)
A practical checklist that survives contact with reality
Strip away the vendor language and the durable practices are few:
- Restore something small every night automatically.
- Restore something large every quarter, on the record, with timings.
- Keep one copy immutable and one identity that can delete it, with alerts on use.
- Know the retrieval cost per terabyte before you need it.
- Escrow keys offline and test decryption.
- Review backup configuration against a baseline monthly.
- Write the runbook for a tired stranger, not for yourself.
None of this is exotic. All of it gets skipped when the team is busy, which is the actual root cause behind most cloud backup failures. The technology rarely fails on its own. Attention does.
Conclusion
Cloud backup problems are rarely about the storage service. They are about assumptions: that a green checkmark means recoverable, that three copies in one account count as three copies, that retrieval is free, that the key is somewhere safe. Each assumption is cheap to test and expensive to discover during an outage. The teams that recover calmly are not the ones with the most sophisticated tooling. They are the ones who restored last month, wrote down how long it took, and fixed what broke while it was still a drill.
Frequently asked questions
How often should cloud backups be tested with a real restore?
Small canary restores can run nightly because automation makes them nearly free. Full restores of critical systems should happen at least quarterly, with the elapsed time recorded and compared to recovery objectives. The right cadence is the one the team will actually sustain. A monthly test that happens beats an annual test that gets postponed.
Is cloud backup enough on its own, or do I still need local copies?
Most organizations benefit from both. Cloud copies protect against site loss and local hardware failure, while a local copy shortens recovery time for large datasets that would be slow and expensive to pull back over the network. The deciding factors are recovery time objectives and retrieval cost, not ideology.
What is the biggest cause of failed restores?
In practice, three causes dominate: missing or mismanaged encryption keys, backups that were never validated as complete and consistent, and runbooks that assume knowledge the on-call engineer does not have. All three are process failures rather than storage failures, which is why they recur even after tooling upgrades.
Does immutability really stop ransomware from destroying backups?
It removes the most common path. Immutable or object-locked storage prevents overwrite and deletion for a defined retention period, even by privileged accounts. Attackers can still cause damage elsewhere, and misconfigured retention periods still expire, so immutability belongs alongside separate administrative identities and deletion alerting rather than replacing them.
How do I estimate what a full restore will cost?
Multiply the volume of data to be retrieved by the provider’s current egress rate, then add any retrieval fees specific to the storage class in use. Do this before an incident and record the figure in the runbook. Pricing changes over time, so revisit the estimate whenever the provider updates its documentation.
Can small teams manage this without dedicated backup staff?
Yes, if the scope stays honest. Automate the nightly canary restore, put the quarterly full restore on the calendar as a fixed commitment, store keys with an offline escrow copy, and keep the runbook short enough to read under pressure. Small teams fail at backup for the same reason large ones do: the work is invisible until it is urgent.







