Lambda and Kappa Architecture Trade-Offs for Observability Pipelines

Observability pipelines force a choice that most data architecture debates leave abstract: Lambda or Kappa, and what each one actually costs once alerts start firing. This piece works through that choice using the exact demands observability puts on a pipeline.

Why observability pipelines are a stress test for data architecture choices

Most pipelines can afford to be slow in one place if they're fast in another. Observability pipelines don't get that luxury. They need three things at once, and none of them are negotiable.

First, high-cardinality ingestion. Logs, traces, metrics, session data, all arriving continuously at machine scale, often with hundreds of attributes per event. Second, low-latency querying. An alert that fires four minutes late might as well not have fired at all, because the incident it was meant to catch is already spreading. Third, reliable reprocessing. Schema changes happen. Late events appear in the pipeline. Backfills are routine.

Those three together produce a workload that punishes architectural shortcuts fast. A system that unifies logs, traces, metrics, and session replay through OpenTelemetry into a single columnar store, scaling from one node to multi-petabyte, gives a useful sense of what "observability at scale" really means in practice. It's infrastructure that has to hold up while the thing it's monitoring is on fire, not a side project bolted onto a data warehouse. It's infrastructure that has to hold up while the thing it's monitoring is on fire.

The decision here is about picking which cost a team is willing to carry. It's about picking which cost a team is willing to carry: two codebases that slowly drift apart, a replay bill that grows every quarter, or query latency that quietly undercuts the alerts the whole system exists to produce.

What Lambda architecture costs an observability team at 2 AM

Lambda's structure is well known by now. A batch layer handles correctness over the full historical dataset. A speed layer handles freshness, giving a fast but possibly incomplete view of recent data. A serving layer merges the two so a query gets a single answer. Simple enough on a whiteboard.

The trouble is what that structure guarantees over time, not what it looks like on day one. Running two codepaths for the same transformation logic guarantees divergence over time. It's a setup where divergence is basically built in. Batch jobs and streaming jobs get touched by different engineers, on different schedules, for different reasons, and the business rules inside them drift apart even when nobody intends it. Late events and schema changes have to get handled twice, in two systems that don't share the same assumptions about ordering, completeness, or null handling.

At 2 AM, an alert fires, and the batch layer says one value for a metric while the speed layer says another. That's what the architecture was always going to produce once enough time passed. It's what the architecture was always going to produce once enough time passed. And for an observability platform, "which number is correct?" is a question that should never come up while an incident is live. Worse, figuring out which layer produced which view becomes step one of incident response, before anyone's even looked at what's actually broken.

A governance cost sits in here too. Lineage now crosses two systems with different semantics, which makes auditing painful in a domain where the pipeline itself is often the thing being audited. Observability data exists partly to answer "what happened and when." A pipeline that can't answer that about itself is working against its own purpose.

Kappa's promise and the storage cost that complicated it

Jay Kreps laid out the alternative in a 2014 O'Reilly piece, "Questioning the Lambda Architecture." His point wasn't that streaming is faster than batch, though it often is. His point was structural: an immutable, ordered log combined with stream processing can support both live computation and full reprocessing. The second codepath Lambda requires just isn't necessary.

That gives Kappa three real advantages for observability work. One codebase means one place where transformation bugs live and one place to fix them. Reprocessing works by replay: fix a bad parsing rule in a metric extraction step, replay the affected window, and the historical view and the live view come out consistent, because they were computed by the same logic. And the log itself becomes the contract. Producers write events once, and every consumer derives its view from that same ordered history, so there's no argument later about whose number is right.

But a Kappa design only works if the replay it promises is actually affordable, and that's where the traditional version of this architecture runs into a wall. Long retention on a standard Kafka cluster means storage that's mostly cold sitting on infrastructure built for hot, low-latency writes. Replicated broker volumes grow right alongside the log, so storage cost scales with how much history gets kept, even when almost none of it gets read. When a backfill does happen, it competes with live traffic for the same broker resources. And because partitions are tied to specific stateful brokers, scaling compute up or down means moving a lot of data around just to rebalance.

A Kappa pipeline that can't retain enough history without starving live traffic is a slogan with a bill attached. It's a slogan with a bill attached.

How diskless Kafka changes the replay cost calculation

Kafka-compatible brokers stop being the thing that owns persistent local partitions, and object storage takes over as the durable source of truth. That one change moves the economics of long retention somewhere else entirely.

Storage cost now scales with object storage pricing rather than replicated broker disk. AWS S3 Standard runs $0.023 per gigabyte per month in us-east-1, which is a different cost category altogether from provisioning and replicating broker-attached storage for the same data. Compute and storage also scale independently under this model, so a replay job doesn't have to fight live ingestion for the same node's resources anymore. And because retention is no longer tied to expensive broker disk, keeping a long window around is just how the system runs, with no special-case decision needing sign-off.

For an observability team, this makes the replay story Kappa always promised something you can actually operate. A month of trace retention kept around for SLO reprocessing stops being a line item someone has to defend in a budget review.

None of that erases judgment from the equation, though. Governance still matters. Historical corrections still need a process. Workload isolation between live traffic and backfill jobs is still something to think through. Query latency over cold history is still a separate problem from storage cost. Diskless Kafka removes one major objection to Kappa. It doesn't remove all of them, and treating it like it does would be its own mistake.

Where S3-routed query paths break observability latency requirements

Cheap storage and fast queries are not the same problem, and observability pipelines expose the gap between them quickly.

Look at the latency tiers involved. NVMe-backed storage, the kind used in latency-sensitive tiers, operates at sub-millisecond to low-millisecond latency. Standard object storage like S3 runs 100 to 200 milliseconds, which is fine for a lot of applications. Archive tiers like Glacier are designed for infrequent retrieval, not interactive queries, and are useful for compliance retention and not much else.

Now think about what an observability query actually asks: show the p99 latency for a service over the last 15 minutes, or find every trace containing a specific error code. Queries like that touch a lot of data across many small pieces, and if every one of those pieces requires a round trip through S3's 100 to 200 millisecond floor, that floor becomes a structural tax on every query, not an occasional inconvenience.

A pipeline can get ingestion completely right under Kappa and still fail the observability use case, if the query layer routes every read through S3. Correct ingestion and fast querying are separate engineering problems, and solving one doesn't solve the other.

The sparse index approach used in columnar systems built for this workload shows the contrast well. Take a table with 8.87 million rows. Instead of indexing every row, the system stores the first value from every 8,192nd row, producing an index with just 1,083 entries, around 97 KB in size. A binary search over that small index tells the query engine which chunks of data are worth reading and which can be skipped entirely, often skipping the overwhelming majority of the table. That trick only pays off, though, if the data it's pointing at sits in memory or on fast local disk. If every one of those chunks still has to be fetched from S3, the index tells you what to read, but reading it still costs 100 to 200 milliseconds each time.

The properties observability workloads need from a pipeline architecture

Strip away the architecture labels for a second; four requirements actually matter.

Consistent transformation logic comes first. Alerts and dashboards need to agree on what a metric means, full stop. Any setup that computes the same calculation two different ways under two different conditions is going to fail this, sooner or later.

Affordable, scoped replay comes second. Schema changes and parsing fixes aren't rare events, they're Tuesday. The architecture needs to support replaying a specific window, a day or a week, without a cost that makes engineers quietly avoid doing it.

Hot-data query performance comes third. Alert latency isn't a nice-to-have feature; it's the entire point. The query path for recent data can't be allowed to depend on cold storage retrieval time.

High-cardinality ingestion without merge pressure comes fourth. Observability data is wide, often with hundreds of attributes per event, and the storage engine handling it has to absorb that without insert latency backing up into everything downstream. The failure mode to watch for in columnar systems is that small, frequent batches force the storage engine into constant background merging, which eats resources that should be going toward serving queries.

How Lambda maps against observability requirements

Against requirement one, consistency, Lambda comes up short in a fairly fundamental way. Two codepaths can't guarantee identical semantics once they're both being modified independently over months and years. For observability, where the same event needs to produce the same metric whether it lands on time or three minutes late, that's disqualifying for most teams, most of the time.

Against requirement two, replay, Lambda is actually strong. The batch layer exists specifically to recompute history, and that's a first-class, well-understood operation in this architecture. If scoped, reliable replay is the top priority, Lambda has a real answer here.

Against requirement three, hot-data query performance, Lambda doesn't really have a position one way or the other. It depends entirely on how the speed layer gets built, not on anything inherent to the Lambda pattern. Teams under-invest in the speed layer precisely because they assume the batch layer will eventually correct whatever it gets wrong.

Against requirement four, ingestion and merge pressure, Lambda actually doubles the work. Both layers need to handle ingestion independently. Two places where a merge storm can happen means two separate batching configurations that need tuning and monitoring.

How Kappa maps against observability requirements

Consistency is where Kappa earns its keep. One codebase, one set of transformation rules, the same result regardless of when an event shows up. For SLO calculations and metric definitions, where the whole point is that everyone's looking at the same number, Kappa meets this requirement structurally, not by convention or discipline.

Replay is where the answer gets conditional. Kappa can meet this requirement, but the cost depends heavily on how the storage layer underneath it is built. Diskless Kafka makes long-window replay affordable in a way traditional Kafka never quite managed. Teams still running Kappa on traditional Kafka with months of retention are going to feel that broker storage bill directly, no matter how clean the single-codebase story is.

Query performance is neutral territory for Kappa, the same way it was for Lambda. Kappa delivers events to whatever sits downstream of it, but it doesn't determine how that downstream layer serves queries. A Kappa pipeline feeding a query engine with hot data cached on fast local storage behaves nothing like a Kappa pipeline feeding a query engine that routes every read through S3. The streaming layer isn't the thing deciding which of those two happens.

Ingestion and merge pressure is a clear win for Kappa: one path to tune instead of two. Guidance on keeping batch sizes aligned with a storage engine's internal block structure only needs to get applied once here, not duplicated across a batch layer and a speed layer that might drift toward different settings over time.

The modern hybrid pattern: streaming pipeline plus lakehouse for historical queries

Looking at how these tradeoffs actually shake out, a pattern that isn't quite pure Kappa and isn't Lambda either emerges. A streaming backbone, built diskless so long retention doesn't punish the storage budget, handles ingestion and keeps one consistent set of transformation rules. Sitting downstream of it, a fast query layer with hot data on local or low-latency storage handles the alerting and dashboard traffic where object storage retrieval latency is simply not acceptable. Further downstream still, a lakehouse layer holds the long tail of historical data for the queries that can tolerate slower response times, like quarterly trend analysis or compliance lookback.

This isn't a compromise so much as an acknowledgment that the four requirements don't all get satisfied by the same component. Consistency comes from the single streaming codepath. Affordable replay comes from diskless log storage. Query speed comes from a serving layer built for it. Historical depth comes from a lakehouse designed for exactly that job and nothing else.

Which raises the real question a team ought to be asking, not "Lambda or Kappa" as a label, but where in this chain do the actual dollars and milliseconds go, and which of those costs is the team actually equipped to absorb.