r/ApacheIceberg • • 2d ago

Inside Apache Iceberg V3: Deletion Vectors and Puffin Files

Post image
15 Upvotes

r/ApacheIceberg • • 3d ago

Apache Iceberg Beginners Course: an Ethereum lakehouse with PyIceberg in Rosetta DBT Studio

Thumbnail
youtu.be
0 Upvotes

Covers the full Iceberg stack locally: SQLite catalog, Parquet/CSV/JSON imports, ACID snapshots, deletes and upserts, schema and partition evolution, sort order, time travel, tags and branches (write-audit-publish), metadata tables, and maintenance. Built inside Rosetta DBT Studio. All code on GitHub.


r/ApacheIceberg • • 4d ago

Merge on read and maintenance on GPUs

Thumbnail github.com
2 Upvotes

r/ApacheIceberg • • 4d ago

Streamhouse and Lakehouse with Apache Iceberg

Thumbnail
lakeops.dev
1 Upvotes

r/ApacheIceberg • • 18d ago

Latest status on GPU engines for Iceberg?

5 Upvotes

r/ApacheIceberg • • 26d ago

Apache Iceberg TCO: The Real Cost of Lakehouse Table Maintenance

Thumbnail
lakeops.dev
6 Upvotes

r/ApacheIceberg • • 28d ago

StreamFusion (OSS flink accelerator on datafusion) now supports iceberg, kafka, paimon, and fluss sources and sinks

Thumbnail
github.com
3 Upvotes

r/ApacheIceberg • • 28d ago

Real-Time Lakehouse Guide: Apache Iceberg in Production

Thumbnail
lakeops.dev
5 Upvotes

r/ApacheIceberg • • Oct 14 '26

WriteSerializable as an isolation level?

3 Upvotes

I was reading the deltalake docs for databricks. They seem to have come up with this term "WriteSerializable" as an isolation level.

That name really bothers me. It is based primarily on snapshot isolation; and the behavior of this is by and large the same as snapshot.

I would never compose the term "WriteSerializable" with the help of the term "serializable". The behavior of WriteSerializable is not anywhere close to the familiar ANSI serializable isolation level. Sure, the mechanism can guarantee serialization of the writes, but it is really not that impressive from an isolation perspective. The clients using this mechanism will constantly experience unexpected behaviors; just like the one in their example (below)

TLDR: they shouldn't name WriteSerializable with the help of "Serializable" (any more than they should rename "Snapshot" to be "SnapshotSerializable"). It diminishes and tarnishes the meaning of a truly serializable transaction.

I have NOT found the term "WriteSerializable" used in the apache iceberg community yet. Is this terminology going to be eventually introduced, as Databricks exercises more influence? Where iceberg is concerned, I only see the more familiar terms : Serializable (the default) and Snapshot.


r/ApacheIceberg • • Oct 04 '26

Cross-Catalog Sync: Iceberg on Polaris, Glue, and Unity

Thumbnail
lakeops.dev
2 Upvotes

r/ApacheIceberg • • Sep 29 '26

The Growing Iceberg Ecosystem Being Built on iceberg-rust

Post image
5 Upvotes

r/ApacheIceberg • • Sep 21 '26

Iceberg on a single node is coming together

Post image
4 Upvotes

r/ApacheIceberg • • Sep 20 '26

The Story of iceberg-rust: The Official Rust Implementation of Apache Iceberg

Post image
4 Upvotes

r/ApacheIceberg • • Sep 18 '26

Apache Iceberg Virtual Meetup: GSoC Projects and Commutative Compaction · Zoom · Luma

Thumbnail
luma.com
2 Upvotes

Tomorrow at 9:00 AM PST (global)


r/ApacheIceberg • • Sep 17 '26

How AI Agents Query Apache Iceberg Data with MCP

Thumbnail
lakeops.dev
1 Upvotes

r/ApacheIceberg • • Sep 15 '26

GitHub - sanderdw/iceberg-data-platform: Educational Open Source Apache Iceberg Dataplatform based on Polaris, RustFS, FastAPI and Marimo workspace

Thumbnail
github.com
5 Upvotes

I build a open source dataplatform for educational purposes, check also my reddit post here: https://medium.com/@sanderdw/e89c5d0bed98


r/ApacheIceberg • • Sep 14 '26

Unity Catalog: Pros and Cons

Post image
2 Upvotes

Apache Iceberg won the open table format war when Databricks acquired Tabular, followed by its subsequent adoption across the industry. Then, the catalog war began.

In a lakehouse, storing data in object storage and using the Apache Iceberg format is only part of the story. You also need a catalog that helps lakehouse query engines like Spark, Flink, or RisingWave discover tables, manage metadata, enforce access control, and work with governed data across different systems.

That is where Unity Catalog comes in.


r/ApacheIceberg • • Sep 11 '26

The Future of Iceberg Isn't One Engine. It's an open Control Plane with many engines.

Thumbnail
lakeops.dev
2 Upvotes

r/ApacheIceberg • • Sep 09 '26

Turns out "Iceberg is open" doesn't mean every engine can actually read your table

2 Upvotes

Read something this week that put a name to a problem I've half run into before but never really understood the mechanics of. Sharing because I think a lot of people assume Iceberg interop is more solved than it is.

Everyone knows the pitch: Iceberg is an open spec, so any Iceberg compatible engine can read any Iceberg table. Mostly true, until you start doing row level deletes, and then it falls apart in a way that's honestly kind of sneaky because nothing looks wrong until a query actually fails.

Quick walkthrough of the scenario in the post. You've got a customers table, three rows, one Parquet file, tracked by whatever catalog you're using. At this stage every engine reads it fine because there's nothing to interpret, it's just a metadata pointer to a file.

Then a row gets deleted. Parquet files are immutable so the writer has two options: copy on write (rewrite the file without that row) or merge on read (leave the file alone and write a separate delete file that readers apply at scan time). The writer in this example goes merge on read and emits an equality delete file, which basically just says "for this data file, treat any row where customer_id = 102 as removed." Under the hood Iceberg uses field IDs and sequence numbers to make sure an old delete doesn't accidentally nuke a newer row with a reused key, but the equality matching is the part that matters for compat.

Spark reads the new snapshot, understands equality delete semantics, does what's effectively a left anti join between the data file and the delete file, and returns the correct two rows. Fine.

Snowflake hits the exact same catalog, same metadata file, same Parquet file. It can resolve the table, read the schema, open the data file. But if that access path doesn't implement equality delete reads, the scan planner just throws an unsupported feature error the moment it hits delete-0002.parquet. Query fails. Same snapshot, same files, two completely different results depending purely on what the reader implements.

The bit that actually reframed how I think about this: the catalog isn't a translation layer. It's job is basically just "here's where the current metadata lives," commit coordination, namespace and access management. It's not opening delete files and rewriting them into a format each engine understands. A REST catalog like Polaris doesn't change this, it still just points you at metadata, it doesn't apply deletes for you.

The post also gets into position deletes vs deletion vectors vs copy on write, with a rough cost tradeoff table (equality delete is cheap to write and requires equality delete support to read, position delete requires resolving key to physical position and is heavier on write, deletion vectors need Iceberg v3 support specifically, copy on write is the most expensive to write but has basically universal read compatibility since there's no outstanding delete file involved).

The framework that's actually useful operationally: your safe feature set is the intersection of every required engine's capabilities, not the union. If Spark supports equality and position deletes but Snowflake only does position deletes, you write position deletes, because "at least one engine supports it" doesn't help you when you have three engines that all need to read the same table.

There's a decent pre production checklist too, don't just run a SELECT COUNT after your first write, actually insert some rows, update one, delete one, commit, then read the same snapshot from every engine you care about and diff both counts and values.

Full post if you want the details: https://olake.io/blog/iceberg-interoperability-myth-row-level-deletes/

Disclosure since it's relevant, I work on OLake, it gets a brief mention near the end, but the actual content here is engine agnostic and applies no matter what's writing your tables.

Has anyone actually hit this for real, table looks completely fine, one engine just refuses to read the current snapshot because of the delete encoding?


r/ApacheIceberg • • Sep 08 '26

Apache Iceberg Table Cleanup: A Production Guide

Thumbnail
lakeops.dev
1 Upvotes

A practitioner's guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle.


r/ApacheIceberg • • Sep 07 '26

Best way to go about benchmarking Indexing?

3 Upvotes

Hey folks,

I'm a security professional was looking to make Iceberg a bit more performant for my own needs - SOC operations (Faster needle searches [pruning]).

I've built an index and proxy that people can point their catalog configuration at, so there's minor changes to their stack.

I figured in for a penny in for a pound,
I've done some quick search and ran clickbench and "httplogs"

(Numbers so far: httplogs)

StockUsing Kahshe Proxy
opens 991 filesopens 2 files
execution time: 6.6–11.5sexecution time: 0.34–0.44 s
1.3gb read2.6mb read

Its all looks good on paper, I think?
but I was wondering if there were more credible/industry standard methods that data engineers use/care about to benchmark these types of technologies?

Repo if interested: https://github.com/Kahshe-io/kahshe


r/ApacheIceberg • • Sep 06 '26

Apache Iceberg Compaction Best Practices

Thumbnail
itnext.io
2 Upvotes

r/ApacheIceberg • • Sep 06 '26

Exploring Apache Iceberg in Rosetta DBT Studio (v1.6.7)

Thumbnail
youtube.com
1 Upvotes

Discover the new Apache Iceberg integration in Rosetta DBT Studio v1.6.7! In this video, we explore how to seamlessly connect, manage, and query Iceberg DataLakes directly from your desktop.

Whether you're using Polaris, Nessie, Lakekeeper, Hive, or SQLite, Rosetta brings true ACID transactions, time travel, and schema evolution to your analytics workflow.


r/ApacheIceberg • • Sep 01 '26

BigQuery's Iceberg REST Catalog (BigLake): how table discovery actually changed

Thumbnail
reddit.com
4 Upvotes

r/ApacheIceberg • • Aug 31 '26

Apache Iceberg Performance Optimization: Queries to Tables

Thumbnail
lakeops.dev
3 Upvotes