ReplicatedMergeTree Quorum Write Configuration and Its Latency Cost
Understanding when quorum writes protect data versus when they just slow you down.
ReplicatedMergeTree quorum write configuration is a tradeoff between durability and latency, but the setting only pays off when it matches the replica topology and the way applications read. The key is to understand what happens between an INSERT and its acknowledgment, then apply quorum selectively to spend that waiting time where it protects data.
How ReplicatedMergeTree handles writes by default, and what that costs on a node failure
Default replication acknowledges a write before the other replicas have it. When a replica receives an INSERT, it writes the part to its own disk, registers a GET_PART log entry in Keeper, and acknowledges the client. Its peers pull the part asynchronously in the background. That keeps the client from waiting for replication, but success at this point guarantees the data is available only on the receiving replica.
The failure window lies between acknowledgment and the peers receiving the part. If the acknowledging node dies during that window, the rows are gone from the surviving cluster. Replication trouble does not necessarily produce an immediate query failure, so lag can accumulate behind apparently successful requests. A failover then exposes stale data that routine queries had given no reason to suspect. The lever that changes this acknowledgment contract is insert_quorum.
What the insert_quorum setting actually does, step by step through the coordination layer
insert_quorum makes the client wait until the set number of replicas have confirmed the write. Its default integer value is 0, which disables quorum writes. When it is set to 2, at least two replicas must write the data successfully before the client gets a response.
Keeper tracks which replicas have confirmed and tells the acknowledging node when enough have done so to meet that threshold. The path therefore includes the local disk write, replication to participating peers, their confirmations, and coordination through Keeper. Keeper sync adds measurable latency on top of the disk-write and replication work. The quorum consistency contract covers previous INSERT queries on participating replicas and linearizes the INSERT sequence.
Concurrency is a separate part of this mechanism. insert_quorum_parallel defaults to 1, allowing multiple quorum inserts to proceed concurrently. Setting it to 0 serializes writes, which enables select_sequential_consistency but reduces throughput. Waiting for enough copies and ordering writes are related costs, but they are not the same operation. That distinction matters when a reader expects to see data immediately after writing it.
Where the latency comes from: the per-merge timeline that makes quorum expensive
GitHub issue #92976 documents a three-node SCADA IoT cluster where sustained ingestion outpaced merge throughput. In less than three hours, write lag had grown substantially and tens of millions of rows were in backlog. The reported root cause was replication serialization blocking merge completion. Each merge had to wait for peer replication to acknowledge it before it could finish, and when ingest rates were high, the queue never drained.
A merge reaches the point where it needs peer acknowledgment, waits for that replication, and only then completes. Meanwhile, sustained ingestion continues to add work faster than the blocked pipeline can clear it. The SCADA case gives no time for each step on its own, but it shows why the order of those steps matters.
For quorum inserts, the added latency follows the replication lag of the slowest replica needed to confirm the write. In a healthy cluster, that is typically tens of milliseconds. Under load or with a lagging replica, it can stretch to seconds.
The read-your-writes trap: how mixing consistency settings produces silent data loss
The cost now turns from latency to correctness. A consistency setting on reads cannot make a non-quorum write visible on a replica that has not received it. The read-your-writes contract requires insert_quorum ≥ 2 and insert_quorum_parallel = 0 on the write side, paired with select_sequential_consistency = 1 on the read side. Applying only part of that contract can leave a pipeline reporting success while silently omitting data.
PeerDB's production bug showed this failure in its CDC pipeline using NORMALIZE. Each batch inserted raw rows and immediately read them back with INSERT…SELECT. On a replicated cluster with insert_quorum=0, the read could land on a replica that had not yet received the just-written parts. Those rows were never normalized, without a query error announcing their omission. The raw write succeeding was not enough to make the downstream transformation complete.
GitHub issue #97557 describes another visibility problem after mixing sequential consistency and quorum settings on a Distributed table over ReplicatedReplacingMergeTree. Rows inserted later became invisible to reads using select_sequential_consistency = 1, even after OPTIMIZE FINAL, and the issue was unresolved as of its filing date. These failures are harder to detect than rejected queries because a successful result can still have a plausible row count. The missing data may come to light only when a downstream consumer finds a gap.
Choosing a quorum value: what replica topology makes each setting viable or dangerous
A quorum value is constrained by replica count and the application's durability requirements. With two replicas per shard, the cluster can survive a single node failure, but it cannot meet insert_quorum = 2 while one replica is down. Inserts then wait until the replica recovers or the quorum is lowered. Three replicas per shard can lose one replica and still satisfy a quorum of two. That makes three the practical minimum topology for quorum writes in production when writes must continue through one failure.
insert_quorum = auto uses a majority of total configured replicas, calculated as ⌊number_of_replicas/2⌋ + 1. This makes the threshold more resilient to topology changes than a hard integer, but it does not make an unhealthy replica available. A quorum above the number of currently healthy replicas blocks rather than returning a clean error, potentially stalling the ingestion pipeline. That tradeoff is worth making for financial records, audit logs, billing events, and multi-replica durability required for compliance. For high-throughput, loss-tolerant telemetry and clickstream workloads, the overhead is usually not justified.
Reducing the throughput penalty: replica role separation and write traffic routing
Heavy insert workloads can degrade query-serving replicas because replication lag makes those replicas wait for consistency before returning results. Separating replica roles is the main architectural mitigation. Specific replicas handle inserts, while others serve queries, so receiving write traffic does not become every replica's job.
Routing writes to a dedicated write replica keeps query-serving replicas out of the direct INSERT acknowledgment path when they receive replication asynchronously. The write replica still participates in quorum coordination through Keeper, and the configured quorum still needs enough confirming replicas. Role separation therefore isolates work rather than removing the acknowledgment requirement. It also narrows the impact of a SCADA-class backlog, since ingestion outpacing merges on the write replica does not immediately degrade end-user query latency. This works only if the application or load balancer routes traffic carefully, through client-side routing or a proxying layer.
Where zero-copy replication was supposed to help with S3-backed clusters, and why it still can't be relied on
S3-backed storage does not automatically make self-managed ReplicatedMergeTree replication lighter. In the standard model, each replica writes its own full copy of each part to S3. A cluster with N replicas therefore produces N full copies, with redundant transfers consuming network and disk bandwidth. Quorum waiting adds to the work of writing separate copies on each replica.
The consequences extend beyond the time spent waiting for acknowledgments. Together, nodes with high PUT/s can go over S3's per-prefix PUT rate limit and get 503 SlowDown errors. This pressure worsens as replica count grows, and changing quorum cannot prevent it. The write path therefore carries API-rate, cost, and latency constraints. Confirmations arrive within a replication model that still writes separate copies.
Zero-copy replication was intended to remove that duplication, but it remains experimental and cannot be relied on for production use. SharedMergeTree avoids duplicating data on each replica through shared storage. The open-source equivalent remains unavailable for production use, so it is not a dependable escape from the self-managed S3 tradeoff.
Why agentic pipelines are the most vulnerable consumers of a misconfigured quorum
PeerDB's NORMALIZE bug is the canonical example of the write-then-read pattern that also exposes agentic pipelines. An agent working with data that was just committed can insert raw rows and read them back right away with INSERT…SELECT. With insert_quorum=0 and sequential consistency unenforced, that read can reach a replica missing the new parts. The agent then reasons over an incomplete result from a request that succeeded. Unlike a human-facing query, there is no reader with an intuition that the result looks thin.
What the agent produces downstream depends on which rows it received. Which replica a Distributed table routed the read to can change those rows, yet that routing is not visible in any log unless full query logging is enabled. A pipeline can therefore produce different outputs without an obvious explanation in its normal record of success. Full query logging is required for agentic workloads to diagnose this class of failure after the fact. Without a way to reproduce an output or understand how it was produced, there is no sound basis for improving it or trusting it.
Configuration decisions that are actually worth the latency they cost
Quorum writes earn their latency when the workload needs linearized consistency and the topology can sustain the required confirmations. On clusters where the number of replicas may change, use insert_quorum = auto instead of a hard integer so the threshold follows the configured majority. This avoids a fixed quorum becoming mismatched to the topology, though a shortage of healthy replicas can still block writes. For read-your-writes guarantees, pair insert_quorum_parallel = 0 with select_sequential_consistency = 1 and a quorum that supplies the required confirmations.
Apply quorum selectively by table or pipeline. For financial and audit data, waiting for more than one copy is worth the cost, but for loss-tolerant telemetry and clickstream it usually is not. For agentic pipelines that write and read in the same session, treat insert_quorum = auto and insert_quorum_parallel = 0 as required defaults, with the matched read setting. PeerDB's July 2026 default flip is the clearest industry signal that this is now the expected baseline.
The remaining decision is operational, because a suitable configuration can still meet a workload that outruns it. Monitor replication queue depth as a leading indicator of whether quorum serialization is accumulating lag. A growing queue shows that work is coming in faster than the replication path can clear it, before requests that succeed give a false sense that all is well. That is the signal to watch before the delay becomes a 35-minute backlog.
