Parquet Row Group and Page Index Statistics Used by Columnar OLAP Engines
Sorted data and page indexes unlock far faster scans than statistics alone.
Parquet scan performance depends on both columnar storage and a layered pruning hierarchy. Row-group statistics rule out broad ranges of data, while page indexes narrow the scan within the row groups that are left. Which layers a columnar analytical query engine uses explains why the same files can cost so much more or less to scan, and why tuning starts when the files are written.
What Parquet files actually contain and why the footer comes first
A Parquet file has row groups, each row group has column chunks, and each chunk has pages. The file opens with a PAR1 magic marker, followed by row-group data, footer metadata, the footer length, and another PAR1 marker. Readers begin at the other end. They locate the footer, parse its serialized metadata, and learn the schema and the locations of row groups and column chunks before decompressing a data page. This footer-first access pattern is part of the format’s design.
Row groups split the table horizontally into self-contained units that can be read in parallel or skipped. A file of one million rows might have four row groups, with one chunk for each column in each group. Each chunk therefore contains one column’s values for that group. Queries referencing only a few columns can leave the other chunks unread.
Pages are the smallest units of encoding and compression within a chunk. A chunk contains data pages and, where applicable, a dictionary page. For a column such as country, dictionary encoding stores unique values once and compact references in the data pages. The footer describes these physical units and records the fields that make row-group pruning possible, including per-column minimums, maximums, and null counts.
How row-group statistics enable the first and coarsest layer of data skipping
Those minimums and maximums form zone maps, layer one of the pruning hierarchy within a Parquet file. For WHERE amount > 500, an engine can inspect each row group’s max(amount) and reject a group whose maximum is below 500. No data page needs to be decompressed. Null counts support a related decision for IS NULL and IS NOT NULL predicates, allowing groups that cannot match to be excluded in the same pass.
How much can a pair of bounds reveal? That depends on the order of the values. If amount is sorted across files, each file and row group can cover a narrow, non-overlapping range. If values are spread at random, every group may cover nearly the whole domain, so none can be ruled out. Random high-cardinality identifiers expose this weakness especially clearly, though bloom filters offer a complementary test for equality predicates.
Group size also changes the cost of pruning. The recommended range is 128 MB–512 MB. Smaller groups mean more metadata overhead and less data eliminated per skip decision, while larger groups reduce parallelism and make pruning coarser. Some writers omit the footer’s min/max fields, leaving this layer unavailable regardless of how carefully the data is arranged.
The gap row-group statistics leave and why bloom filters close only part of it
When UUIDs or user IDs are scattered across every row group, the value in WHERE user_id = 'abc' may lie within every group’s bounds even when it is missing from most groups. Min/max statistics cannot distinguish those cases. The engine needs a membership test.
Bloom filters answer whether a value is definitely absent from a column chunk. A definitive negative allows the engine to skip the row group, while a possible match leaves it in consideration. Parquet uses the Split Block Bloom Filter, or SBBF, designed for cache performance and SIMD support. These filters operate at column-chunk granularity and help equality predicates, but they cannot prune ranges or inequalities such as >, <, or BETWEEN.
The commonly omitted bloom-filter-ndv setting supplies the expected number of distinct values. Without it, inferred bitset sizing can produce a false-positive rate approaching certainty, making almost every group appear relevant. Even a well-sized filter leaves a structural gap. With only zone maps and bloom filters, one matching page can keep an entire row group in the scan, including its many nonmatching pages.
How the page index pushes min/max skipping below the row-group boundary
The page index uses the zone-map idea for each page within a row group that is still in the scan. Its ColumnIndex stores per-page minimums and maximums so the reader can identify pages that could match. Its OffsetIndex maps each page to its row range within the group. This second structure connects pruning across columns. When the engine rejects pages in the filter column, it can identify the corresponding rows to skip in columns the query returns but does not filter.
Both structures sit near the footer, separate from row-group metadata. Readers that do not need selective scans can avoid deserializing them, so full scans need not incur index I/O. Before this structure, gathering page bounds required reading page headers sequentially. That meant touching much of the column data simply to decide what could be left unread.
The benefit still depends on local order. Sorted or clustered pages can have narrow bounds, so selective equality tests and narrow ranges can rule out more pages. This works for ID lookups on ID-sorted tables, short time windows on timestamp-sorted events, and queries for a single tenant on tenant-sorted data. When pages are ordered at random, each may cover almost the whole row-group range. Full scans and broad aggregations likewise have little to discard, so finer metadata alone does not make them selective.
The correctness hazard inside nested columns that makes page-index skipping dangerous to implement
Skipping pages requires more than finding bounds that do not match. The reader must also keep rows aligned across columns, which is straightforward for flat columns because the relationship between rows and encoded values is known. Nested columns complicate that relationship. Arrays, maps, and structs use Dremel repetition and definition levels, and the number of encoded values per row can vary.
To skip whole rows, the reader must follow that level structure and find where each row starts and ends. A mistake need not cause a crash. Values can slide between rows, so the result looks normal but has the wrong data. The hazard is therefore not merely slower execution or a failed query. It is a loss of correctness that can remain silent.
This distinction helps explain why readers have been slow to add support and differ in what they support, though page indexes are still worth using. Writing an index and using it safely are separate tasks, so an engine may write indexes before its reader supports page skipping. The practical guidance is to prioritize writer-side generation so files contain the index. Reader-side enablement, especially for nested data, remains an engine-specific capability and configuration to verify.
How major engines differ in which pruning layers they actually use
Almost all engines that read Parquet use row-group zone maps for predicate pushdown, but they differ more in what they support below that level. A study published on arXiv found that Arrow and DuckDB used Parquet zone maps at the row-group level but did not use PageIndex. The same study found that InfluxDB and Spark had PageIndex and bloom filters turned on, using more storage space to make selection faster. These are findings about the implementations evaluated. They are not a substitute for checking the engine version processing a workload.
Amazon Athena added page-index support for Iceberg tables on November 21, 2025. Its behavior also depends on the storage setting. Iceberg tables in S3 general-purpose buckets require explicit enablement through a Glue table property, while S3 Tables use the indexes automatically when they are present.
Apache DataFusion uses bloom filters when available, with particular value for equality predicates. This adds another distinction to the comparison. An engine may use membership information yet differ from another engine in how it uses page indexes. Consequently, a well-written Parquet file establishes what is possible, while the reader determines which of those possibilities becomes reduced scan work.
How table-format metadata layers multiply the effect of file-level statistics
Iceberg and similar table formats extend pruning above the individual file. Partition filters can eliminate manifest entries without inspecting data files, while per-file column bounds in manifests can reject files before their Parquet footers are opened. The engine then uses row-group pruning on the files that are left, followed by page-index pruning where it is supported. Each stage narrows the next stage’s input. The footer is therefore the beginning of a file scan, but not necessarily the beginning of the pruning decision.
Iceberg data skipping documentation shows how these layers work together with a query that filters on a date partition and customer ID. Across hundreds of thousands of potential file reads, the query can be resolved with only a handful of row-group reads, a 99.999% reduction. That example depends on the layers working together. It is not a performance promise for any table using the format.
A missing layer shifts work downstream. Even with tight row-group bounds, the engine may still have to open every file’s footer if partition pruning cannot rule out files. Even when partitioning is correct, unsorted data in the files can leave every row group in the files that remain in the scan. Iceberg’s manifest bounds are themselves derived from Parquet row-group statistics at write time. Statistics correctness therefore supports decisions above the file as well as inside it.
Sort order and clustering strategies that determine how much of the pruning hierarchy a table can use
For bounds-based pruning, data order determines how much the metadata can exclude. Sorting on the most selective filter column gives that column tight row-group and page ranges, but does not provide the same benefit to other filter columns. Which column deserves that position? Column selection should start with production query logs, ranking columns by their frequency in WHERE, JOIN, and GROUP BY clauses. Clustering without evidence about actual queries risks optimizing for columns the workload rarely uses.
When queries filter on different columns, multidimensional clustering offers a compromise. Z-order uses bit-interleaved space-filling curves in Iceberg’s rewrite_data_files to cluster several columns together. It gives several columns fairly tight bounds, while the bounds for any one column are less tight. That flexibility has a write cost. Compaction that re-sorts data consumes compute, so its cadence depends on query selectivity and how quickly new data disrupts the existing order.
Smaller pages allow tighter bounds and more precise skipping, but they add to footer size and metadata overhead, so page sizing needs to be tuned for each table. Event and log tables are strong candidates for this work. Sorting by timestamp, or by tenant plus timestamp, fits queries for narrow time windows or one tenant’s data. Partitioning then limits which files are touched, while sorting controls how much is read inside those files.
What to verify and configure to make statistics-based pruning work end to end
These pruning structures are optional, and their absence does not prevent correct query results. A table can cost more to scan without anyone noticing. To check the writer side, verify that it emits footer min/max statistics and explicitly enable page-index writing where it is not the default. For columns intended to use bloom filters, set bloom-filter-ndv to the expected distinct-value count.
The next check is the file itself. A tool that inspects Parquet metadata should show statistics fields with values in them instead of null fields, along with both ColumnIndex and OffsetIndex structures. Writer configuration alone is not evidence that the intended metadata reached the files. On the reader side, confirm support for page-index skipping in the engine that actually executes the query. For Athena’s Iceberg tables in S3 general-purpose buckets, set the Glue table property use_iceberg_parquet_column_index = true. For bloom-filter-capable engines including Trino, Spark SQL, Presto, and DataFusion, verify that bloom filter pruning is enabled in the query planner configuration.
Selective queries provide the final operational test. Compare bytes scanned with bytes in the table, using filters on the sort key that return a small share of rows. If the scan remains broad, investigate the writer-to-file-to-engine chain for a missing layer. This cost compounds in agentic and high-concurrency workloads because every non-pruning query repeats avoidable reading and decompression. Memory or NVMe caches for hot row groups bear that cost in a different way from readers that keep fetching from object storage. How much data did the query actually avoid reading?
