Z-Order and Hilbert Curve Clustering in Iceberg and Delta Lake for OLAP Scan Pruning

Z-order and Hilbert clustering help analytical queries skip files by keeping related rows close across several columns. Hilbert preserves that proximity more faithfully, but geometry alone does not decide the better choice. Stable, selective predicates can favor Z-order, while varied queries favor Hilbert. In Iceberg and Delta Lake, the maintenance model is just as important as the curve.

Why multi-dimensional file pruning is harder than it looks

The minimum and maximum values in a columnar file tell a query engine whether the file could hold matching rows. When similar values are spread across many files, those ranges overlap so much that the engine can rule out few files. Sorting by one column tightens its ranges. A filter on that column can then skip files cleanly, but filters on other columns get no such guarantee.

Hierarchical sorting extends the order across several columns, but it gives them a fixed priority. A query that filters on the leading column benefits, while one filtering only on a secondary or tertiary column can still face a full scan. The leading sort has scattered those other values. No linear order can perfectly preserve proximity along every axis, so multi-dimensional clustering seeks a compromise that tightens file statistics across several columns at once.

How space-filling curves map multiple columns onto one dimension

Both techniques use the values in a row’s clustering-column fields as coordinates in multi-dimensional space. Each curve defines a traversal through that space and assigns the row a position along it. Rows are then sorted by that position. Conceptually, the traversal visits small regions of a grid in sequence, turning several coordinates into one sortable value without simply making one column dominant.

That order changes which rows end up in the same files. Delta Lake Z-ordering internals explains how data-skipping information is gathered automatically at write time and how column minimum and maximum values are then used to rule out files at query time. Both curves aim to make those ranges tighter. The engine can reject more files because fewer have ranges that overlap the predicate.

The clustering key sets a firm boundary on that benefit. For a column outside the key, both techniques leave skipping no better than it is with unordered data. This is an important limit on “multi-dimensional” clustering.

Where Z-order and Hilbert diverge geometrically, and why it matters for pruning

Z-order makes large jumps across the original space, while positions next to each other along a Hilbert curve stay next to each other in that space. Those jumps have a file-level consequence. Rows close in the original space can end up far apart in Z-order, and a file that spans a jump must report wider ranges that make it harder to rule out.

Hilbert avoids those discontinuous jumps and preserves locality more faithfully. Even so, nearby points can end up apart in its one-dimensional order. As Z-order’s interleaved bit pattern spreads across more dimensions, its locality weakens, with each added column beyond the third or fourth diluting the pruning benefit.

Iceberg PR #16827 reports a benchmark on TPC-H SF10 lineitem data in which Hilbert reduced files scanned across all six queries compared with Z-order. But most of the differences in files scanned are within one standard deviation. The benchmark used a laptop and small files, and measured pruning at planning time, with execution latency outside its scope. The PR author states that scaling the dataset and file size together preserves the pruning ratio, making the Hilbert-to-Z-order ratio more robust than absolute seconds per operation. This provides directional evidence.

How Iceberg and Delta Lake expose these curves — and what that changes operationally

Delta Lake’s Liquid Clustering embeds Hilbert in an incremental maintenance system. Files first receive a Hilbert coordinate at OPTIMIZE time. The distinction is operationally important. OPTIMIZE processes files marked as unclustered, including newly written files and small or deletion-vector files, while leaving already-clustered files untouched through its ZCube tracking model.

Changing Liquid Clustering keys takes a metadata-only ALTER TABLE update that completes immediately. The next OPTIMIZE works incrementally under the new keys. The metadata change is immediate, but it does not mean every existing file has instantly been rearranged.

Apache Iceberg instead applies curve clustering through explicit compaction with rewrite_data_files, without write-time coordinate assignment or a per-file clustered/unclustered flag. PR #16827, opened June 15, 2026, proposes Hilbert through CALL system.rewrite_data_files(table => 'db.tbl', strategy => 'sort', sort_order => 'hilbert(c1, c2)'). Its initial scope is Spark 4.1, with backports to 4.0 and 3.5 to follow. The proposal remains pending merge review.

That shared procedure does not supply Delta’s maintenance tracking. Once files are in a healthy size band, Iceberg’s size-based planner can miss newly ingested files that overlap in curve space. A successful rewritten=0 result can therefore coexist with degrading clustering. Engine support adds another boundary. Trino’s OPTIMIZE gained sort-based compaction through sorted_by, merged October 2026, but not Z-order or Hilbert compaction, which remain Spark-specific capabilities in this context.

Write amplification — the cost each approach pays for its pruning gains

Z-order’s read savings require substantial rewrite work because its compaction is not incremental. Each run re-sorts the entire affected dataset, so a small addition can trigger a rewrite many times larger than the new data. The relevant cost is not just ingestion. It also covers the work done over and over to restore the layout that helps pruning work well.

Liquid Clustering’s ZCube model instead keeps amplification close to one-to-one for the clustering rewrite. New data is written at ingest, then OPTIMIZE rewrites only the unclustered portion. This benefit depends on the maintenance architecture. Iceberg’s proposed Hilbert support uses the same batch rewrite framework as its Z-order support.

Fabric Runtime 1.3 has a flagged hazard in which each auto-compaction trigger with liquid clustering rewrites all data in Z-Cubes smaller than 100 GB. That produces severe amplification. The scheduling guidance for CPU-intensive Z-order work is therefore weekly off-peak compaction rather than hourly runs, while Spark-based Iceberg rewrites also require dedicated cluster provisioning.

What the benchmark evidence actually shows about which curve wins which query

Kushal Kumar Vishwakarma’s InfoQ benchmark, reviewed by Renato Losio and published October 12, 2026, compared Partitioning, ZOrder, and Liquid Clustering on Delta Lake. It used tens of millions of records to test four types of workload. No approach won every query class. The results show that what works best for stable, selective predicates can differ from what works best for broader combinations of columns.

The point lookup exposes a weakness in a familiar proxy. Liquid Clustering was much faster even though it scanned more data, because locality within files cut seek overhead. File pruning still helps, but the amount scanned tells only part of how much work it takes to retrieve matching rows.

The benchmark also exposes a time-range failure mode. Putting a date column second or third in a ZOrder expression scatters sequential date values across files, making time-range queries materially slower than a simple date-partitioned table. Key placement can therefore work against the query’s shape. This is not evidence that clustering always improves on a simpler layout.

The benchmark’s boundaries deserve equal attention. Its dataset was synthetic, it did not measure write amplification, and it did not test schema drift. The Iceberg PR adds a narrower, cleaner planning-time signal because file-selection counts are deterministic and independent of execution variance. Together, the evidence supports a workload-dependent comparison.

The case for Z-order — when stable, selective predicates make it the right choice

Z-order has a strong case when nearly every query filters on the same two or three high-cardinality columns and that pattern stays fixed. In that workload class, its precise pruning can outperform Hilbert-based clustering by a significant margin. Selectivity gives the layout a clear target.

When the query pattern is known and a weekly compaction window is acceptable, Z-order needs no extra infrastructure to track clustered files or assign coordinates at write time. A scheduled OPTIMIZE provides the maintenance step. Its rewrite cost is easier to accommodate when ingest is infrequent and the layout does not need constant revision.

The InfoQ author frames the diagnosis through a practical question. Do the top queries consistently filter on the same two or three columns? If so, ZOrder on those columns, with the most selective listed first, is the proposed choice. This makes Z-order a deliberate match for a stable workload, rather than merely an older alternative to Hilbert.

Choosing between the two curves — a workload-driven decision framework

Z-order suits small sets of predicates that stay the same, while Hilbert’s broader locality suits combinations that vary or change over time. Column count changes that balance. Because Z-order’s locality degrades above three to four columns, Hilbert is the stronger default for clustering across four or more columns.

The next constraint is the maintenance budget. High-frequency ingestion favors Liquid Clustering when compaction costs must scale with new data alone. Low-frequency ingestion can tolerate Z-order’s batch amplification more readily. When keys are likely to change, Liquid Clustering’s immediate metadata update also helps, while changing a Z-order key means rewriting the whole table.

Format and engine then determine which choice is actually available. Delta offers Hilbert with incremental ZCube tracking through Liquid Clustering, while Iceberg on Spark offers Z-order and has Hilbert pending through the rewrite procedure. These are not interchangeable packages. Choosing Hilbert in Iceberg would change the geometric layout without importing Delta’s incremental maintenance model.

Emerging agentic workloads put pressure on the stable-predicate assumption. Agents probe diverse columns in unpredictable combinations, which favors Hilbert for tables shared with exploratory queries and human analysts. Z-order can still suit tables set aside for a narrow purpose. Teams serving agents are also advised to log every query in full for audit and use isolated read-only compute paths so exploratory probes do not slow production work.

What Iceberg’s Hilbert addition changes, and what it doesn’t

Iceberg’s proposed Hilbert addition would bring its geometry much closer to Delta Lake’s by using a familiar Spark procedure to give it stronger multi-dimensional locality. It would not close the architectural gap. The size-based planner would remain, and compaction would remain a batch operation with an O(table size), rather than O(new data), cost model.

Without a clustered/unclustered file flag, new files can quietly weaken the layout, and rewritten=0 does not establish that clustering is fresh. Initial support is scoped to Spark 4.1. Other engines’ graceful fallback does not mean they gain curve compaction from the proposal.

Could tighter Hilbert grouping reduce later upsert amplification by making MERGE or copy-on-write touch fewer files? The benefit is data-dependent and requires merge keys to align with clustering columns. It is not yet a general cost saving to assume. For Iceberg teams, the guidance is to track the pending proposal rather than deploy it immediately, while using production-ready Z-order where supported and where the workload justifies it.