Iceberg V2 Row-Level Delete Files and Their Scan Cost in OLAP Engines

Iceberg V2 row-level delete files make mutations cheaper by leaving more of the work to readers that run analytical queries. That trade-off becomes dangerous when delete artifacts accumulate faster than maintenance removes them. Understanding where the cost lands, from planning through row filtering, makes it possible to manage that debt deliberately rather than discover it through production query latency.

Why Iceberg V2 moves delete cost from writers to readers

Object stores do not support in-place edits, and Parquet files are immutable. Deleting a row therefore requires either rewriting its data file or recording an instruction to ignore it. Copy-on-write takes the first route, keeping reads clean but rewriting an entire file even when only one row changes. Merge-on-read takes the second, keeping writes cheap while asking every future reader to reconcile the instructions.

Think of an errata pamphlet tucked into a book. Publishing a correction is easy, but every reader must consult it. Iceberg V2 delete files serve that role, and the deferred work stays manageable only while the pamphlets remain bounded. Copy-on-write therefore fits tables that rarely change but are read often, while merge-on-read fits continuous mutations that need a fast write path.

What position delete files and equality delete files actually write, and how readers consume them

Position delete files store (file_path, row_position) pairs that give the exact address of each deleted row. The writer first locates the row in the target file. That lookup buys a cheaper read-time check, because the reader knows its current file and row position. Entries must be sorted by path and position, and a manifest entry can identify a single referenced data file when all its deletes target that file.

Equality delete files record values. A deletion can say to remove rows where user_id = 42, without finding their positions first. The reader then evaluates rows in applicable data files against those values. Flink and Spark CDC sinks default to this approach because their writers cannot know row positions, so avoiding the writer lookup creates work for subsequent scans.

Position deletes apply to data files with lower or equal sequence numbers, while equality deletes apply only to files with strictly lower sequence numbers. That distinction protects newly written rows from earlier equality deletes. It also means reading a historical snapshot requires resolving the delete state belonging to that snapshot.

How delete file accumulation compounds with commit frequency

V2 puts no limit on the delete files associated with a data file, and it does not require writers to consolidate them. When commits happen often, hot files can end up with hundreds of delete artifacts. More delete files mean more work for each scan, even when few rows have been deleted.

Small batches make this dynamic particularly costly. Many small delete files need more bookkeeping than one larger file for the same rows, and object store requests are priced by the number of operations. Each subsequent scan needs the applicable delete objects, so request volume multiplies with query traffic. A streaming pipeline can thus keep its write path responsive while steadily increasing both read latency and request costs.

Why equality deletes create a qualitatively worse read problem than position deletes at scale

Position deletes have a bounded scope, though not a bounded count. They affect the specific data files they reference. A query that avoids those files avoids their delete reconciliation too. Equality deletes spread work across their partition scope because the reader must find which files have matching rows.

This makes partition size part of the read penalty. An applicable equality delete may require evaluation even when it removes nothing from a particular file. As delete artifacts accumulate, streaming CDC writes can leave repeated partition-wide reconciliation for later queries. The difficulty is not just more bookkeeping, but less certainty about where the work can stop.

Siemens encountered a compatibility consequence of this distinction. Its CDC data written as equality deletes could not be read by Snowflake external tables until a periodic Spark compaction job cleared those files. That added latency, compute cost, and architectural complexity. Equality deletes also complicate changelog semantics, because identifying the rows actually removed requires scanning for matches, weakening row-lineage guarantees for audit pipelines, regulatory reporting, and historical queries.

Where OLAP engines pay the scan cost: planning, I/O, and merge overhead

The cost starts to add up before the engine reads data rows. During scan planning, it opens manifests, finds the delete files that apply, and builds a merge plan for each data file. Position-delete references help link instructions to files, while equality deletes need the reader to work out their partition scope. As the number of small artifacts grows, traversing this metadata becomes substantial work of its own.

Object retrieval adds another layer of delay. Each delete file is a separate object store GET, so opening a data file can entail fetching many related objects before returning rows. The engine must then reconcile sorted position-delete entries with the row stream or evaluate equality values against rows in scope.

What happens to predicate pushdown and file pruning? Equality reconciliation limits how far the engine can avoid work, because it must establish whether rows match the delete values. Statistics-based pruning cannot be assumed to remove that obligation. Faster planning alone therefore cannot repair a scan still burdened by object retrieval and row-level reconciliation.

How GDPR erasure and regulatory delete patterns interact with the accumulation problem

A logical delete is not a physical erasure. In merge-on-read mode, the underlying Parquet file still contains the row, while an Iceberg-aware reader masks it. A direct Parquet reader bypasses that reconciliation entirely. An ETL job, lifecycle tool, or ad-hoc query opening the object directly can therefore see data that the logical table considers deleted.

Equality deletes can retain sensitive values themselves. If the delete key is an email address rather than a surrogate key, the erasure instruction contains the personal data it targets. For regulated deletion workflows, committing a delete file cannot be treated as the completed storage cleanup.

Compaction rewrites live data, snapshot expiration removes old snapshot references, and orphan file removal follows expiration. Compaction must also precede storage-class transitions. Scheduling lag stretches the interval between masking a row and physically removing it, making maintenance part of the deletion workflow.

Why compaction is load-bearing maintenance and what happens when it falls behind

Compaction incorporates accumulated deletes into rewritten data files containing only live rows. Eliminating the old merge cost from later reads of those files is the primary way to retire V2 delete debt. It also explains why leaving compaction behind can make latency worsen without a failed write or a conspicuous error.

The remedy has a cost of its own. Rewriting large files to remove a few rows consumes compute and creates write amplification, so overly aggressive maintenance can cost more than the read work it saves. Yet delaying it lets every intervening query pay reconciliation again. The balance depends on commit frequency, query latency requirements, and the permitted physical-erasure window.

Sequencing makes this harder than a periodic rewrite alone. A regulated table still needs expiration and cleanup after compaction, and those steps must finish within its deletion window. Treating maintenance as a background job leaves these dependencies easy to miss. Missed maintenance dependencies cause silent, cumulative increases in query latency and storage bills, even when jobs succeed.

What Iceberg V3 deletion vectors change about the accumulation problem

Iceberg V3 deletion vectors change the bookkeeping structure. A vector is a Roaring Bitmap stored in a Puffin sidecar file and associated with a specific data file. The Iceberg specification permits at most one deletion vector per data file in a snapshot, with new deletes merged into that structure. Readers therefore no longer have to open more and more separate position-delete artifacts.

This bounds the per-file retrieval fan-out from delete history. The reader uses the data file and its paired vector. It still applies deletions at read time, however. V3 limits the accumulation of delete artifacts without making merge-on-read free, and it does not by itself eliminate partition-wide equality-delete evaluation.

Apache Spark supports deletion vectors, while Trino has experimental read and write support as of mid-2026, without DV-aware compaction or full V3 production readiness. Engines that do not implement the V3 specification cannot safely read such tables. The format's stronger bound is useful only when the read path supports it.

How schema design choices determine whether delete files accumulate quickly or slowly

Narrower partitions on a high-churn dimension put fewer data files within each delete's scope. More partitions are not automatically better, though. Over-partitioning creates small files, so the useful target is the natural scope of mutations.

Typed, filterable identifiers let engines use column statistics during equality-delete evaluation, whereas identifiers embedded in JSON blobs or maps can require full row decoding. The reader gains better information for limiting work than an opaque nested value provides, while equality deletes still require matching values to rows.

Separating append-only data from historically mutable data prevents broad partition scopes from drawing unaffected files into delete evaluation. Sequence numbers already protect later data from earlier deletes, while partition boundaries make that separation explicit. Sorting on the primary mutation key also concentrates affected rows into fewer files, reducing delete fan-out without prescribing one schema for every workload.

Why agentic query workloads are disproportionately exposed to delete file accumulation

AI agents can issue many narrow, repeated, parallel queries during iterative analysis. Every applicable probe can repeat the same equality-delete evaluation across its partition scope. The deferred work from a write is therefore multiplied by a much higher query cadence than a human analyst typically produces.

Historical probes add snapshot resolution to that cost. A query using FOR SYSTEM_VERSION AS OF with a snapshot ID needs the delete files active at that snapshot. A long uncompacted history makes this resolution more involved than reading only the current state.

An agent opening Parquet through S3 without Iceberg metadata cannot resolve delete state and may see masked rows. The operational response is governed access through a structured interface that preserves logical table state, enforces read-only access on isolated compute, and logs queries. Tracking the cost of each query then shows where repeated reconciliation adds work, even amid the queries agents run.

A framework for auditing and managing delete file accumulation before it surfaces in production latency

The first operating step is to inspect delete state before tuning queries. Metadata exposes per-data-file delete counts, sequence numbers, and file sizes. Teams should look for disproportionate counts on hot files, equality deletes spanning broad partitions, and maintenance outside its expected window.

The next choice belongs at the table level. Rarely changed tables serving latency-sensitive dashboards may warrant copy-on-write despite higher write cost. Where continuous mutation warrants merge-on-read, the delete mechanism must match the engines that will consume it. Equality-delete incompatibility in a reader such as the Snowflake external-table path is a write-side constraint.

Compaction then needs an explicit service-level agreement tied to commit cadence, read requirements, and regulatory deletion windows. For regulated tables, that window must cover compaction, snapshot expiration, and orphan cleanup in sequence. Agent-facing tables also need query isolation, full logging, and cost visibility. Understanding whether the expense lies in planning, retrieval, or reconciliation is what makes these choices deliberate, while instrumented delete state and explicit maintenance commitments keep production latency from becoming the first warning.