Why does a storage system that promises eleven nines of durability still manage to ruin your Tuesday afternoon? Ask anyone who has run a data pipeline against an object store and they will tell you: the failures are rarely dramatic. They are quiet, slow, and expensive. A listing loop that never finishes. A bucket bill that triples overnight. An upload that hangs at 98 percent and never completes.
Object storage is the backbone of modern cloud architecture, and for good reason. It scales almost without limit, it is cheap per gigabyte, and it speaks a simple HTTP API. But that simplicity hides a set of sharp edges. This article walks through the problems practitioners actually hit, why they happen, and what genuinely works when you run into them. (Cloud)
The Consistency Trap: When Reads Don’t Match Writes
The oldest and most misunderstood problem in object storage is eventual consistency. In early S3-style systems, a newly written object was immediately readable, but overwrites and deletes could take time to propagate. A process that deleted a file and then listed the bucket might still see the file. A reader might fetch a stale version minutes after the writer thought the update was done.
This behavior caused years of subtle bugs. Teams built “write, then read” verification loops that passed locally and failed in production. Caches were layered on top of an already inconsistent layer, multiplying the confusion.
The situation changed significantly in December 2020, when Amazon announced strong read-after-write consistency for all S3 read, write, and list operations, meaning a completed write is immediately visible to any subsequent request. That solved the problem for S3 itself, but it did not solve it everywhere. Many on-premises object stores, S3-compatible appliances, and multi-region replication setups still offer only eventual consistency. The practical lesson is to never assume strong consistency unless the vendor explicitly documents it, and to design idempotent writes so a retry does not create a duplicate object.
Uploads That Stall: The Expect Header and Proxy Problem
Large uploads fail in ways that are maddening to debug because the client reports almost nothing useful. One of the most common culprits is the `Expect: 100-continue` header. Well-behaved HTTP clients send this header before a large body, asking the server to confirm it is willing to accept the data. If the server or an intermediary proxy mishandles the handshake, the upload stalls until a timeout fires.
The behavior is defined in the HTTP/1.1 specification, and the details of how servers should respond to the Expect header are worth reading if you are debugging a stalled transfer. In practice, the fix is often to disable the header on the client, or to configure the load balancer to pass it through correctly. Neither is obvious from the error message, which usually just says “connection reset” or “timeout.”
Other upload failures come from body size limits on reverse proxies, TLS inspection appliances that buffer and re-chunk traffic, and clients that retry without exponential backoff. If uploads succeed from one network and fail from another, suspect the network path before you suspect the storage service.
Orphaned Multipart Uploads and the Bill Nobody Notices
Multipart upload is how large files get to object storage reliably. The client splits the object into parts, uploads them in parallel, and then asks the service to assemble them. It is efficient and resilient. It is also a classic source of silent cost.
When a multipart upload is started but never completed, the uploaded parts sit in the bucket. They are billable. They are invisible in a normal object listing. And they stay there until someone aborts them. Teams discover this months later when a cost review shows a bucket “containing” far more data than the objects suggest.
The standard remedy is a lifecycle rule that aborts incomplete multipart uploads after a set number of days. Setting that rule up takes a few minutes and is one of the highest-return housekeeping tasks in cloud storage. The same lifecycle mechanism handles transition to colder storage tiers and expiration of old objects, so it pays to treat lifecycle policy as a first-class part of bucket design rather than an afterthought.
Listing Large Buckets: Pagination and Incomplete Inventories
A naive listing loop is a rite of passage. Someone writes code that calls the list operation and assumes it returns everything. It works fine in a test bucket with fifty objects. In production, with forty million objects, it returns the first page and stops.
Object storage list operations are paginated, and the modern API uses continuation tokens rather than simple offsets. A correct implementation loops until the response no longer includes a continuation token, and it must handle the fact that the listing is a snapshot in time. Objects added or deleted during a long listing may or may not appear. For inventory work, this matters enormously: a backup job that lists a bucket while writes are ongoing can silently miss files.
The practical pattern is to use a managed inventory feature where one exists, or to reconcile listings against a manifest. If you must list, sort keys by a stable prefix and accept that the result is approximate unless you control writes during the scan.
Hot Partitions and Throttling Under Load
Object stores distribute load across partitions based on key prefixes. If every request hits the same prefix, or if a workload suddenly writes thousands of objects per second to one narrow key range, the store throttles those requests. Clients see errors that translate roughly to “slow down,” and retries pile on top of an already saturated partition.
This is a design problem more than an operational one. Sequential key names, such as timestamps or incrementing IDs at the start of the key, concentrate traffic. Randomizing the leading portion of the key spreads it. The trade-off is that random prefixes make listing less intuitive, so many teams use a hybrid: a date prefix for human navigation, with a hash segment inserted to distribute load.
Retry behavior deserves equal attention. A client that retries immediately and aggressively turns a brief throttle into an outage. Exponential backoff with jitter is not optional in a serious object storage client.
Small Objects: The Hidden Cost of Granularity
Object storage charges per request and stores metadata per object. A workload that writes millions of tiny files pays far more in requests and metadata overhead than in actual data. Latency per object also adds up: fetching ten thousand small objects takes far longer than fetching one large one, even at identical total bytes. (Blog)
The usual fixes are to batch small records into larger objects, to keep frequently queried small items in a database and only archive them to object storage, or to use a log-structured format that appends records into a few large files. Each has trade-offs around update granularity and read amplification, but the underlying principle holds: object storage is optimized for large, immutable blobs, and fighting that design is expensive.
Retrieval Costs and Archive Latency
Archival storage tiers are seductive. The per-gigabyte price is a fraction of standard storage. What the pricing page makes less obvious is that retrieval is a separate transaction with its own cost and its own waiting time. Restoring an object from a deep archive tier can take hours, and the retrieval fee can exceed the storage savings if the data is accessed more than expected.
The mistake teams make is archiving data they still occasionally need without modeling the access pattern. A useful discipline is to tag objects with an expected access frequency and to review those tags quarterly. If a dataset is restored more than a couple of times a year, it probably belongs in a warmer tier.
Durability Is Not Backup
Eleven nines of durability is an impressive number, and it is also widely misunderstood. Durability describes the probability that stored data is not lost due to hardware failure. It says nothing about whether someone deletes the object by accident, whether ransomware encrypts it, or whether a misconfigured lifecycle rule expires it prematurely.
Protection against those events requires versioning and, for regulatory or ransomware scenarios, object lock with a write-once retention period. Versioning keeps prior copies when an object is overwritten or deleted, turning a catastrophic mistake into a recoverable one. Object lock goes further by preventing deletion entirely for a defined window. (Working with object metadata)
The common failure here is enabling versioning but never testing a restore, or enabling it without a lifecycle policy to expire noncurrent versions, which quietly inflates storage costs. Both are worth a quarterly drill. Teams that practice restoring from version history discover their gaps before an incident does.
Compatibility Gaps in S3-Compatible Stores
The S3 API has become a de facto standard, and a long list of vendors advertise compatibility. Compatibility, however, is rarely complete. Versioning semantics, object lock support, multipart edge cases, and the exact behavior of list operations often differ between implementations.
The practical consequence is that code validated against one provider can fail against another in ways that are hard to diagnose. Before migrating, test the specific features you depend on rather than the basic put-and-get path. A short compatibility checklist covering versioning, lifecycle rules, multipart behavior, and access control saves weeks of debugging later.
Key Naming Limits and Character Handling
Object keys are not file paths, and treating them as such causes bugs. Keys are limited to 1,024 bytes of UTF-8, and unusual characters, leading slashes, and repeated separators can behave differently across tools. Some clients normalize paths, some do not. A key that works in one SDK may throw an error in another.
The safe practice is to restrict keys to a conservative character set, avoid characters that need URL encoding, and never assume that two keys differing only in case are distinct. If your application needs to store user-supplied filenames, hash or sanitize them rather than passing them through.
Practical Habits That Prevent Most of These Problems
Most object storage incidents trace back to a handful of design shortcuts. A few habits catch them early:
- Write down your consistency assumptions and verify them against the provider’s documentation before launch.
- Set lifecycle rules for incomplete multipart uploads and noncurrent versions on day one, not month six.
- Use exponential backoff with jitter in every client, and test what happens when the store throttles.
- Model storage, request, and retrieval costs together, not separately.
- Test a restore from version history at least once a quarter.
For teams tracking how these patterns evolve across providers and platforms, the news coverage and the cloud section are useful places to follow changes in the space. (News)
Conclusion
Object storage problems are rarely about the storage itself. They are about assumptions: that reads are instantly consistent, that a listing returns everything, that durability equals backup, that a compatible API behaves identically. Each assumption is reasonable in isolation and wrong in some specific context.
The good news is that none of these problems is exotic or unsolvable. They are well documented, they have standard remedies, and most of them cost nothing to prevent if you address them during design rather than after an incident. Treat consistency, lifecycle policy, retry behavior, and restore testing as part of the architecture, and object storage becomes what it promises to be: boring, cheap, and reliable.
Frequently Asked Questions
Is object storage eventually consistent?
It depends on the provider. Amazon S3 moved to strong read-after-write consistency for all operations in December 2020, so a completed write is immediately visible. Many on-premises systems, appliances, and cross-region replication setups still offer only eventual consistency. Always check the specific provider’s documentation rather than assuming.
Why do my large uploads keep timing out?
A frequent cause is the `Expect: 100-continue` header being mishandled by a proxy or load balancer, which stalls the transfer until a timeout fires. Other causes include body size limits on reverse proxies, TLS inspection appliances, and clients that retry without backoff. Test the same upload from a different network to isolate whether the problem is the path or the service.
Why is my storage bill higher than the data I actually store?
Two common culprits are orphaned multipart upload parts, which are billable but invisible in normal listings, and retained noncurrent object versions after versioning is enabled. Both are fixed with lifecycle rules: one to abort incomplete multipart uploads, another to expire old versions after a set period.
What causes throttling errors when writing many objects at once?
Object stores distribute load by key prefix, so writing thousands of objects per second to a narrow key range concentrates traffic on one partition and triggers throttling. Spreading keys with a hash segment and using exponential backoff with jitter in the client resolves most cases.
Is high durability the same as having a backup?
No. Durability measures the likelihood that data survives hardware failure. It does not protect against accidental deletion, ransomware, or a misconfigured lifecycle rule. Versioning and object lock with a retention period are what protect against those events, and a restore should be tested regularly.
Why does my code work with one S3-compatible provider but fail with another?
S3 compatibility is usually partial. Versioning semantics, object lock, multipart edge cases, and listing behavior often differ between implementations. Before committing to a migration, test the specific features your application depends on rather than only basic put and get.
