Why S3-Routed Reads Make Latency Worse as Your Cluster Grows

A query that hits local NVMe waits microseconds. A query that hits S3 waits for network latency and the overhead of the HTTP request-response cycle before the first byte even starts moving. That gap exists in every cluster that routes reads through object storage, before anyone adds a single additional node. It is not a tuning artifact or a sign of a misconfigured workload. Object storage is a remote HTTP service with a cost attached to every single request, and a block device simply is not.

The numbers make the gap concrete. NVMe runs roughly two to three orders of magnitude lower in latency than S3 Standard, and one to two orders of magnitude lower than S3 Express One Zone, while also delivering several times the throughput. That speed costs more per terabyte to store, so the advantage of local storage is real, but it is not free. None of this makes S3 a poor choice for every job. When data is cold and a workload is dominated by long sequential scans, S3's aggregate bandwidth can hold its own, and that's the honest shape of the tradeoff rather than a case against object storage as a tier. The question a cluster architect has to answer early is which of those two worlds a given workload actually lives in, because the rest of this piece is about what happens when that question gets answered wrong at scale.

HTTP/1.1 and per-request overhead worsening S3 latency beyond the storage benchmark

The raw storage gap described above is the floor, not the ceiling. S3 still depends on HTTP/1.1, and that protocol brings head-of-line blocking and TCP overhead that push per-request latency higher than the storage medium alone would predict. HTTP/1.1 pipelining delivers responses in the same order requests were made. Sending a 1 GB GET and a 1 MB GET back to back forces the small object to sit and wait for the large one to finish, even though it is a fraction of the size. That queuing behavior has nothing to do with disk speed. It is a property of the protocol carrying the request.

A second source of variability occurs in the client itself: connection pooling, retry logic, and socket management inside the SDK all add jitter that a local block device call never has to deal with. That jitter appears as noise around the mean rather than a clean, predictable tax, which is what makes p99 latency hard to reason about: the average request might look fine while the worst requests stretch far past it.

Part proliferation makes the problem worse in a way that is specific to how these systems manage data over time. When a time-to-live policy moves data out to S3 faster than background merges can consolidate it into larger files, the result is a sprawl of small objects instead of a handful of large ones. Every one of those small objects needs its own GET request. A point query that has to scatter across many small files ends up paying the HTTP overhead over and over, once per file, instead of once per query. The protocol tax that looked small on paper multiplies by however many fragments the query has to touch.

Parallel fan-out across compute nodes amplifying a per-request penalty cluster-wide

At this point the per-request penalty stops acting as a fixed cost, because parallel fan-out turns it into a multiplier across the cluster. Every additional compute node in a shared-storage cluster adds more concurrent S3 GET requests against an object store with bounded throughput, so latency grows along with the cluster instead of shrinking as more machines get added. That runs against the usual intuition that more compute means more capacity and faster answers. Why would adding nodes make things worse instead of better?

The answer sits in how distributed queries resolve. In any system where a query has to wait for the slowest of many parallel sub-requests, adding more parallel requests raises the odds that at least one of them turns into a straggler. Tail latency at the query level gets set by the single worst GET, not by the average GET. This is the "tail at scale" dynamic, and it holds regardless of what storage sits underneath. Brokers dispatch sub-queries to servers in parallel, each server fans out its own requests to S3, and the broker has to wait for every response before it can return a result. One slow GET anywhere in that tree delays the whole query.

LinkedIn's research on its Pinot system documented this exact fail-slow straggler problem, and it occurred even on clusters running local storage. It is a property of scatter-gather query execution generally. The LinkedIn result matters here because it shows the fan-out tail-latency problem is real and severe even on local-storage clusters; on S3-routed clusters the baseline per-request latency is already higher, so the same fan-out dynamic starts from a worse floor.

S3 also enforces per-prefix request-rate limits, specifically 3,500 PUT, COPY, POST, or DELETE requests per second and 5,500 GET or HEAD requests per second per partitioned prefix, with per-connection throughput capped at 5 Gbps. A cluster running five nodes might stay comfortably under those limits. The same cluster at twenty nodes can routinely cross them, and when it does, the throttling that follows looks, from the query layer's point of view, like random latency spikes with no obvious cause. Nobody changed the query. Nobody changed the data. The cluster just got bigger, and bigger meant more concurrent requests competing for the same bounded resource.

Put the pieces together: adding compute nodes to an S3-routed cluster does not make any individual request faster. It increases the total number of concurrent requests in flight, which increases contention on the object store, which raises the probability that any given query gets caught behind a slow tail request. Scaling out, in this architecture, scales the problem right alongside the compute.

Zero-copy replication, shared metadata, and operational risk as clusters grow

Latency is not the only thing that gets worse as these clusters grow. The shared-storage design carries operational fragility that compounds right alongside rising fan-out costs, and it appears in places that have nothing to do with query speed.

Zero-copy replication, where replicas share the same S3 objects instead of each keeping its own copy, has been disabled by default since version 22.8 because of known data integrity and stability problems, including data loss and deadlocks. That caveat means a feature that sounds like it should save both storage cost and replication time is not ready for production use in most workloads. The reason traces back to how metadata works in this design: each replica changes its own metadata independently, so there is no way to commit a change atomically across every replica at once. Merges and mutations end up contending for exclusive locks in the coordination layer, and that contention gets worse, not better, as more nodes join the cluster.

Metadata also stays tied to compute rather than living alongside the data it describes. If a node loses its local metadata, the S3 objects that node was managing turn into orphans that nothing else in the cluster can read. That is a disaster-recovery exposure, and it grows in proportion to how many nodes are each holding their own independent slice of metadata state. A cluster with fifty has fifty.

High server counts also create contention on the replication log, and durability in this architecture depends on object storage plus a coordination service plus local disk working together. That is a wider failure surface than the "stateless compute" framing around these architectures usually suggests. The practical result for anyone operating one of these clusters is that the usual monitoring signals stop meaning what they used to mean. Disk latency might actually be an API round trip to S3. A spike in network traffic might actually be S3 throttling wearing a different disguise. Replication lag might track WAN conditions that have nothing to do with the database itself. None of these signals point directly at their own cause, and all of them arrive late.

Which workloads feel the compounding fan-out penalty most severely

Not every workload feels this the same way. The penalty concentrates hardest on real-time customer-facing analytics, observability pipelines, and anything running against a tight p99 latency target.

Consider a dashboard that customers expect to refresh every second, built on hundreds of millions of rows ingested daily. The latency budget for that kind of dashboard only exists as long as the cache keeps holding. The moment a cluster takes a cold-read breach, the budget disappears, and that is not an abstract engineering concern. Users feel it directly, in the form of a dashboard that stalls.

Observability pipelines sit in a particularly exposed spot because they combine two properties that do not play well together. They take in high write volume, often from thousands of monitoring agents streaming small events continuously, and they get queried on recent data that has not had time to warm the cache. The workload most likely to miss the cache is also the workload least able to tolerate the delay.

Langfuse's own scaling documentation shows a concrete response to exactly this kind of contention. Its architecture routes UI and public-API read queries to a dedicated read-only endpoint, keeping dashboard traffic separate from the inserts and background merges happening on the primary write path. That is a real mitigation, built by an engineering team dealing with a real problem, and the fact that it was necessary says something on its own: contention between reads and writes on shared compute is a problem serious enough to build infrastructure around. It does not, however, remove the underlying cache-miss latency floor. It routes around the contention rather than eliminating the floor itself.

Character.AI's observability infrastructure, built to serve logging for thousands of GPUs, offers the other side of that same coin. By keeping hot data local, the team reported bringing log query times down from a range of minutes to a range of seconds, alongside a meaningful cost reduction. That before-and-after gap shows what a cache miss actually costs in a live production system, and it shows what closing that gap is worth.

Large sequential scans remain the exception where S3-routed reads hold up well: cold batch analytics over historical data, where S3's aggregate bandwidth can match or beat a single NVMe node, is the one case where the tradeoff tilts in S3's favor. That is a different workload shape than a latency-sensitive query running against recent, hot data: one is a workload S3 handles well, the other is one it handles poorly.

AI agents multiply S3 fan-out costs through database probing

The workloads described above are mostly built around human query patterns: a person looks at a dashboard, runs a query, waits for an answer.

An agent working through a task rarely jumps straight to one clean final query. It runs schema introspection first, samples filters, checks cardinality, and only then constructs the query it actually wants to run. Every one of those probing steps is its own request. If a probe misses the cache, it pays the full S3 latency penalty on its own, and that cost gets paid once for every step in the agent's chain.

A human analyst might issue a handful of queries across an entire session. Agents issue many, often from several agent instances running in parallel, and that request pattern happens to be exactly the one that stresses a bounded-throughput object store the hardest. A multi-step agentic chain can push cache-miss costs up by an order of magnitude compared to a single human query, because each intermediate probing step is its own cold read if the data it touches has not been accessed recently.

Running that back through the fail-slow straggler dynamic from LinkedIn's Pinot research compounds the amplification further still. Agents issue many short queries in rapid succession rather than a few long ones, and that pattern raises the odds that any given query in the chain lands on a slow S3 response. Every additional probe is another roll of the dice against the object store's tail latency. A cache layer and a read path isolated from the main write workload stop being nice-to-haves under this kind of load. Without them, a burst of agent queries composed of many small, high-latency round trips against cold data pushes both the latency and the bill for serving it into territory that stops being sustainable.

What flattens the latency curve as clusters scale

The rising latency curve described through this piece is a structural consequence of routing reads through object storage in the first place, not something a configuration change fixes, and flattening it means keeping hot data in memory and off the S3 request path entirely.

The fix is a cache layer sitting between compute and object storage, holding hot data in memory or on local NVMe, so that most query-time reads never generate a GET request at all, no matter how many nodes the cluster is running. That single design choice breaks the multiplication described in the fan-out section: if a read never touches S3, it cannot contribute to per-prefix throttling, and it cannot become the slow straggler that an entire scatter-gather query waits on.

This does not mean object storage was the wrong choice for durability or elasticity. S3 as the durable system of record is not the problem. Routing read-time query execution through S3 at query time is the problem, and the two roles, durable storage and query-time read path, should be kept separate. Read and write path isolation, the approach Langfuse documents, reduces contention between ingestion and dashboard traffic, but it is a necessary operational practice sitting alongside a cache-first read architecture, not a replacement for one.

Workload isolation matters just as much for clusters serving multiple tenants or multiple agent workloads at once. LinkedIn's QWI system, built for Pinot, isolates workloads from each other to prevent noisy-neighbor interference in a shared-storage environment, and the published results show predictable tail latencies with under 1 percent overhead at petabyte scale. For agentic workloads specifically, the same principle applies: isolated read-only compute running over shared data, with full query logging and sandboxed access, keeps an agent's relentless probing from saturating the cache or the object store for every other user on the cluster.

The economics follow the same logic as the latency. A cache-mesh design that keeps hot data off S3 removes the per-GET cost that otherwise compounds as the cluster grows, and at scale, the cost of S3 request volume becomes significant on its own, separate from the cost of storage itself. A cache-first architecture addresses both halves of that bill at once, because both latency and cost trace back to the same root: too many requests reaching an object store that was never built to serve them one at a time, at query speed, under a cluster's full concurrent load.