Iceberg's Puffin Sidecars: Portable Stats for the Open Lakehouse

The cardinality problem nobody quotes in the lakehouse pitch decks

Every open-lakehouse pitch deck of the last four years walks a customer through the same arithmetic. Object storage at $0.023/GB-month. Compute decoupled from storage. Bring your own engine. Multi-engine reads. No vendor lock-in. The math is compelling, which is why Iceberg adoption has been one of the fastest-moving things in data infrastructure since the Databricks–Tabular acquisition closed in 2024.

What the pitch deck never mentions is the moment a Trino engineer points the new lakehouse at a 2 TB fact table and runs the first analytical join. The planner has no statistics. It doesn't know the NDV of the join key. It defaults to broadcasting the wrong side, shuffles four times more data than it needs to, and the query takes eight minutes when the same query on a warehouse would have taken forty seconds. The engineer's first impression of the open lakehouse is that it is slow.

The reason isn't the engine — Trino's planner is excellent. The reason is that nothing wrote any stats into the table. Iceberg's manifest layer carries min/max/null-count per column per data file, which is enough for partition pruning but nowhere near enough for join planning. The optimizer needs NDVs, histograms, sometimes Bloom filters, and those have to live somewhere portable — somewhere the engine that wrote them and the engine that reads them next don't have to agree on a Java class name to interpret. That "somewhere" is Puffin, and it is the part of the open lakehouse architecture almost nobody outside the Iceberg PMC is paying close enough attention to.

What Puffin actually is

Strip away the marketing and Puffin is delightfully simple. It is a single binary sidecar file with the structure Magic Blob₁ Blob₂ ... Blobₙ Footer — a magic-number header, a sequence of opaque blob payloads, and a footer carrying a UTF-8 JSON index describing what each blob is, which snapshot it belongs to, which columns it covers, and how to decompress it (per the Apache Iceberg Puffin spec).

The crucial design choice is that Iceberg does not interpret the blob payloads. The footer says "blob 3 is type apache-datasketches-theta-v1, applies to column l_orderkey on snapshot 4982371, is 16 KB long, starts at byte offset 8192." Iceberg knows where the blob is and what it claims to be. It does not crack the bytes open — that job belongs to whatever library knows how to read that blob type.

That separation is the same trick Parquet's footer uses for column chunk encodings, and it's the reason it scales. New stat types can be added to Puffin without changing the Iceberg core. An engine that doesn't recognize a blob type silently skips it. An engine that does, decodes it and uses it. There is no version-coupling between Iceberg and the things stored inside its sidecars — the only design that survives in a multi-engine world.

The four blob types that matter most

The spec officially defines two blob types today: apache-datasketches-theta-v1 for NDV sketches, and deletion-vector-v1 for Roaring-bitmap deletion vectors (added with Iceberg v3). Two more are under active discussion. Together they cover the four use cases that decide whether a planner produces a good plan:

1. NDV sketches (Theta). Approximate distinct-value count, stored as a serialized compact Apache DataSketches Theta sketch. A 4096-entry sketch carries ±1.56% relative standard error at 68% confidence — accurate enough to drive every join-order and broadcast-vs-shuffle decision a modern optimizer makes. The blob metadata can also carry a pre-computed ndv property so readers don't have to deserialize the sketch just to read the cardinality.
2. Bloom filters. Not yet a formal blob type, but the spec discussion on bloom-filter properties has been live since early 2025. Practical use: shuffle the build-side bloom to probe-side workers and skip entire row groups guaranteed not to match.
3. Histograms. Equi-depth histograms or KLL quantile sketches give the planner distribution information rather than just NDV. The "update supported blob types" thread proposes both a ColumnStatistics blob and a KLL sketch type. Hive 4 already writes histogram-enriched Puffin per snapshot.
4. Expression statistics / partial-aggregation reuse. The newest, least-settled category. Cache pre-computed sub-aggregates so a later query asking a superset question composes the answer from the sidecar rather than rescanning. Same idea as warehouse materialized views, expressed as a portable blob.

The combination matters. NDVs decide join order. Bloom filters cut probe-side work. Histograms turn predicate-selectivity guesses into accurate row counts. Expression stats short-circuit the scan. A planner with all four matches warehouse plan quality. A planner with none of them is the eight-minute Trino above.

Why vendor lock-in dies here

The concrete walk-through. A team operates an Iceberg fact table on S3 with a REST catalog. Monday night a Spark job lands fresh data and runs the Iceberg compute_table_stats procedure (or a Trino job runs ANALYZE, which writes NDV statistics to the Iceberg Puffin file). The sketch lands as a blob in a sidecar. The snapshot commits.

Tuesday morning, a different team queries the same table from Snowflake's externally-managed-Iceberg path — which became GA in October 2025. Snowflake's planner reads the manifest, sees the Puffin file reference, decodes the NDV blob, uses it. Snowflake didn't have to run its own ANALYZE. The Tuesday query plans correctly on the first execution because the stats Trino wrote on Monday night are still in the sidecar, in a format Snowflake understands.

That single property — stats surviving the engine that produced them — is what makes the open lakehouse architecturally credible. Without it, every engine pays the "first-touch tax": scan-and-rebuild of statistics the moment a new engine encounters the table. With it, the stats are infrastructure, not per-engine state.

The lakehouse-with-Puffin number is not equal to the warehouse number — the warehouse still wins on result caching, micro-partition skipping, and warm compute pools. But the gap is single-digit-multiplier, not order-of-magnitude. The catastrophic bar is the bottom one, and that bar exists in production today on every Iceberg deployment whose operators haven't wired up Puffin writes. The cure is not a new engine. The cure is calling ANALYZE.

The production gotchas

Honest accounting: Puffin in 2026 is not a finished story, and four things bite operators in practice.

• Writer support is opt-in and patchy. Trino writes Puffin NDVs through ANALYZE. Spark writes them through the compute_table_stats procedure. Most other engines don't write Puffin at all yet — writing is still not widely adopted and is not fully standardized across the ecosystem. If nobody in your stack writes Puffin, nobody benefits from reading it.
• Reader support is also uneven. Even engines that ship "Iceberg support" may not consult the Puffin sidecar during planning. Snowflake's external-Iceberg path reads Puffin; Spark's NDV consumption is still tracked as an open issue. Check the version matrix before you assume a stats blob is being used.
• Cardinality estimates rot. A Theta sketch written against snapshot 4982371 describes that snapshot's data. Append 30% more data over a week without re-running ANALYZE and the sketch is stale — still readable, still parsing, just wrong. Iceberg's snapshot model lets readers detect this, but it's a discipline question more than a format question. Schedule the stats refresh the same way you schedule compaction.
• Sketches are approximate by construction. A Theta sketch carries ±1.56% relative standard error at 68% confidence, ±3.1% at 95% for the default 4K-entry sketch. That is fine for join planning and group-by sizing. It is not fine for a billion-row anti-join where false positives in a Bloom filter inflate the probe side by 1%. Know the error budget of the structure you're trusting.

None of these are fatal. They are the normal cost of a format that is still maturing in public. They are the reason the post is titled "the most strategically important subsystem nobody is talking about" rather than "the finished thing you already use."

What this means for table-format wars

The 2020-2024 lakehouse pitch was about storage economics — Parquet on object storage versus columnar storage in a warehouse. That war is over; storage costs converged. The 2025-2026 pitch is about planner quality — whether the open ecosystem can carry warehouse-grade statistics across the engines that share a table. That war is being fought right now, in the Puffin spec discussions and the writer/reader matrices.

Delta Lake is making the same move from the other side. Its deletion vectors are now stored using the same Puffin deletion-vector-v1 blob layout Iceberg uses, and there is active work to push more Delta stats into Puffin-shaped sidecars. Hudi has its own indexing story tuned for high-frequency upserts. Each format is climbing the same hill — portable, engine-neutral statistics — by slightly different routes. The portability winner will be whoever ends up with the largest readers-and-writers matrix, not whoever has the prettiest spec.

The bet worth making is that Iceberg + Puffin gets there first, not on technical superiority but on the fact that more engines have agreed to read its sidecars than read anything else's. Snowflake reads Iceberg Puffin. Trino writes it. Spark is closing the gap. The newer entrants — DuckDB-Iceberg, the streaming engines, the GPU-accelerated layer — all default to Iceberg compatibility. The layer doing the load-bearing work in that win is the unflashy one with the puffin on its logo.

If you operate an Iceberg table in production and you are not writing Puffin sidecars, you are running a slow warehouse and calling it a lakehouse. The fix is two lines of SQL and one cron job. The strategic implication is much larger than that suggests.