r/ApacheIceberg • • Dec 15 '25

What React and Apache Iceberg Have in Common: Scaling Iceberg with Virtual Metadata

0 Upvotes

I want to start this post by talking about React. I know what you’re thinking: “React? I thought this was a blog post about Iceberg!” It is, so please just give me a few minutes to explain before you click away.

Note: We've posted this entire blog to the subreddit so you don't have to go to our website to read it, but if you're prefer to read it there, you can access it via this link.

React’s Lesson in Declarative Design

React launched in 2013 and one of the main developers, Pete Hunt, wrote a thoughtful and concise answer to a simple question: “Why did we build react?”. The post is short, so I encourage you to go read it right now, but I’ll summarize what I think are the the most important points:

React’s declarative model makes building applications where data changes over time easy. Without React, developers had to write the declarative code to render the application’s initial state, but they also had to write all of the imperative code for every possible state transition that the application would ever make. With React, developers only have to write one declarative function to render the application given any input state, and React takes care of all the imperative state transitions automatically.

React is backend-agnostic. Most popularly, it’s used to render dynamic web applications with HTML, but it can also be used to drive UI rendering outside the context of a browser entirely. For example, react-native allows developers to write native iOS and Android applications as easily as they write web applications.

The Similarity Between the DOM and Iceberg Metadata  

So what does any of this have to do with data lakes and Iceberg? Well, it turns out that engineers building data lakes in 2025 have a lot in common with frontend engineers in 2013. At its most basic level, Iceberg is “just” a formal specification for representing tables as a tree of metadata files.

Similarly, the browser Document Object Model (DOM) is “just” a specification for representing web pages as a tree of objects.

While this may sound trite (“Ok, they’re both trees, big deal”) the similarities are more than superficial. For example, the biggest problem for engineers interacting with either abstraction is the same: building the initial tree is easy, keeping the tree updated in real-time (efficiently and without introducing any bugs) is hard.

For example, consider the performance difference between inserting 10,000 new items into the DOM iteratively vs. inserting them all at once as a batch. The batch approach is almost 20x faster because it performs significantly less DOM tree mutations and re-rendering.

In this particular scenario, writing the batching code is an easy fix. But it’s not always easy to remember, and as applications grow larger and more complex, performance problems like this can be hard to spot just by looking at the code.

React's Virtual DOM: A Layer of Indirection for Efficiency and Correctness

React solves this problem more generally by introducing a layer of indirection between the programmer and the actual DOM. This layer of indirection is called the “virtual DOM”. When a programmer creates a component in React, all they have to do is write a declarative render() function that accepts data and generates the desired tree of objects.

However, the render function does not interact with the browser DOM directly. Instead, React takes the output of the render function (a tree of objects) and diffs it with the previous output. It then uses this diff to generate the minimal set of DOM manipulations to transition the DOM from the old state to the new desired state. A programmer could write this code themselves, but React automates it even for large and complex applications.

This layer of indirections also introduces many opportunities for optimization. For example, React can delay updating the DOM for a short period of time so that it can accumulate additional virtual DOM manipulations before updating the actual DOM all at once (automatic batching).

Managing Iceberg Metadata: A Tedious, Error-Prone Chore

Let’s transition back to Iceberg now. Let’s walk through all of the steps required to add a new Parquet file (that’s already been generated) to an existing Iceberg table:

  1. Locate and read the current metadata.json file for the table.

  2. Validate compatibility with the Iceberg table’s schema.

  3. Compute the partition values for the new file.

  4. Create the DataFile metadata object.

  5. Read the old manifest file(s).

  6. Create a new manifest file listing the new data file(s).

  7. Generate a new version of the metadata.json file.

  8. Optionally (but must be done at some point):

    1. Expire old snapshots (metadata cleanup).

    2. Rewrite manifests for optimization.

    3. Reorder or compact files if needed for read performance.

That’s a lot of steps, and it all has to be done 100% correctly or the table will become corrupted. Worse, all of these steps have to be done single-threaded for the most part. 

This complexity is the reason that there is very little Iceberg adoption outside of the Java ecosystem. It’s almost impossible to do any of this correctly without access to the canonical Java libraries. That’s also the reason why Spark has historically been the only game in town for building real time data lakes.

How We Built WarpStream Tableflow

WarpStream’s Tableflow implementation has a lot in common with React. At the most basic level, the goal of our Tableflow product is to continuously modify the DOM Iceberg metadata tree efficiently and correctly. There are two ways to do this:

  1. Manipulate the metadata tree directly in object storage. This is what Spark and everybody else does.

  2. Create a “virtual” version of the metadata tree, manipulate that, and then reflect those changes back into object storage asynchronously, akin to what React does.

We went with the latter option for a number of reasons, foremost of which is performance. Normally, Iceberg metadata operations must be executed single-threaded, but our virtual metadata system can be updated millions of times per second. This allows us to reduce ingestion latency dramatically and scale seamlessly from from 1MiB/s to 10+GiB/s of ingestion with minimal orchestration.

But what exactly is the “virtual metadata tree”? In WarpStream’s case, it’s just a database. The exact same database that powers metadata for WarpStream’s Kafka clusters! This database isn’t just fast, it also provides extremely strong guarantees in terms of consistency and isolation (all transactions are fully serializable) which makes it much easier to implement data lake features and ensure that they’re correct.

Tableflow in Action: Exactly-Once Ingestion at Scale

So what does this all look like in practice?

Let’s track the flow of data starting with data stored in a Kafka topic (WarpStream or otherwise):

  1. The WarpStream Agent issues fetch request(s) against the Kafka cluster to fetch records for a specific topic.

  2. The Agent deserializes the records, optionally applies any user-specified transformation functions, makes sure the records match the table schema, creates a Parquet file, and then flushes that file to the object store.

  3. The Agent commits the existence of a new Parquet file to the WarpStream control plane. This operation also atomically updates the set of consumed offsets tracked by the control plane which provides the system with exactly-once ingestion guarantees (practically for free!).

  4. At this point the records are “durable” in WarpStream Tableflow (the source Kafka cluster could go offline or the Kafka topic could be deleted and we wouldn’t lose any records), but not yet queriable by external query engine. The reason for this is that even though the records have been written to a Parquet file in object storage, we still need to update the Iceberg metadata in the object store to reflect the existence of these new files.

  5. Finally, the WarpStream control plane takes a new snapshot of its internal table state and generates a new set of Iceberg metadata files in the object store. Now the newly-ingested data is queryable by external query engine [1].

That’s just the ingestion path. WarpStream Tableflow provides a ton of other table management services like:

  • Data expiration

  • Orphan file cleanup

  • Background compaction

  • Custom partitioning

  • Sorting, etc

But for brevity, we won’t go into the implementation details of those features in this post.

It’s easy to see why this approach is much more efficient than the alternative despite introducing an additional layer of indirection: we can perform concurrency control using a low-latency transactional database (millions of operations/s), which reduces the window for conflicts when compared to a single-writer model on top of object storage alone. For table operations which don’t conflict, we can freely execute them concurrently and only abort those with true conflicts. The most common operation in Tableflow, the append of new records, is one of those operations that is extremely unlikely to have a true conflict due to how our append jobs are scheduled within the cluster.

In summary, unlike traditional data lake implementations that perform all metadata mutation operations directly against the object store (single-threaded), our implementation trivially parallelizes itself and scales to millions of metadata operations per second. In addition, we don’t need to worry about the number of partitions or files that participate in any individual operation.

Earlier in the post, I alluded to the fact that this approach makes developing new features easier, as well as guaranteeing that they’re correct. Take another look at step 3 above: WarpStream Tableflow guarantees exactly once ingestion into the data lake table and I almost never remember to brag about it because it just falls so naturally out of the design we barely thought about it. When you have a fast database that provides fully serializable transactions, strong guarantees and correctness (almost) just happen [2].

Multiple Table Formats (for Free!)

We’ve spent a lot of time talking about performance and correctness, but the original React creators had more than that in mind when they came up with the idea of the virtual DOM: multiple backends. While it was originally designed to power web applications, today React has a variety of backends like React Native that enable the same code and libraries to power UIs outside of the browser like native Android and iOS apps.

Virtual metadata provides the same benefit for Tableflow. Today, we only support the Iceberg table format, but in the near future we’ll add full support for Delta Lake as well. Let’s take another look at the Tableflow architecture diagram:

The only step that needs to change to support Delta Lake is step 4. There is (effectively) a single transformation function that takes Tableflow’s virtual metadata snapshot as an input and outputs Iceberg metadata files. All we need to do is write another function that takes Tableflow’s virtual metadata snapshot as input and outputs Delta Lake metadata files, and we’re done.

Every other feature of Tableflow (compaction, orphan file cleanup, stateless transformations, partitioning, sorting, etc) remains completely unchanged. If Tableflow operated on the metadata in the object store directly, every single one of those features would have to be rewritten to accommodate Delta Lake as well.

Footnotes

‍[1] For security reasons, the WarpStream control plane doesn’t actually have access to the customer’s object storage bucket, so instead it writes the new metadata to a WarpStream-owned bucket and sends the Agents a pre-signed URL they use to copy the files into the customer’s bucket.

[2] I’m exaggerating a little bit, our engineers work very hard to guarantee the correctness of our products.


r/ApacheIceberg • • Sep 01 '23

Building a Data Lakehouse using Apache Iceberg and MinIO

Thumbnail
blog.min.io
0 Upvotes

r/ApacheIceberg • • Jan 18 '26

11 Iceberg Performance Optimizations You Should Know

Thumbnail overcast.blog
0 Upvotes

File sizing, compaction, snapshot expiration, orphan cleanup, manifest rewrites, partitioning, and delete-file management all matter on their own. But in practice, what really determines long-term performance is whether these operations stay aligned with how tables are actually used.

Once you have many tables, multiple engines, and continuous ingestion, manual maintenance inevitably falls behind reality.

These techniques and tools can help solve this problem.


r/ApacheIceberg • • Sep 09 '26

Turns out "Iceberg is open" doesn't mean every engine can actually read your table

2 Upvotes

Read something this week that put a name to a problem I've half run into before but never really understood the mechanics of. Sharing because I think a lot of people assume Iceberg interop is more solved than it is.

Everyone knows the pitch: Iceberg is an open spec, so any Iceberg compatible engine can read any Iceberg table. Mostly true, until you start doing row level deletes, and then it falls apart in a way that's honestly kind of sneaky because nothing looks wrong until a query actually fails.

Quick walkthrough of the scenario in the post. You've got a customers table, three rows, one Parquet file, tracked by whatever catalog you're using. At this stage every engine reads it fine because there's nothing to interpret, it's just a metadata pointer to a file.

Then a row gets deleted. Parquet files are immutable so the writer has two options: copy on write (rewrite the file without that row) or merge on read (leave the file alone and write a separate delete file that readers apply at scan time). The writer in this example goes merge on read and emits an equality delete file, which basically just says "for this data file, treat any row where customer_id = 102 as removed." Under the hood Iceberg uses field IDs and sequence numbers to make sure an old delete doesn't accidentally nuke a newer row with a reused key, but the equality matching is the part that matters for compat.

Spark reads the new snapshot, understands equality delete semantics, does what's effectively a left anti join between the data file and the delete file, and returns the correct two rows. Fine.

Snowflake hits the exact same catalog, same metadata file, same Parquet file. It can resolve the table, read the schema, open the data file. But if that access path doesn't implement equality delete reads, the scan planner just throws an unsupported feature error the moment it hits delete-0002.parquet. Query fails. Same snapshot, same files, two completely different results depending purely on what the reader implements.

The bit that actually reframed how I think about this: the catalog isn't a translation layer. It's job is basically just "here's where the current metadata lives," commit coordination, namespace and access management. It's not opening delete files and rewriting them into a format each engine understands. A REST catalog like Polaris doesn't change this, it still just points you at metadata, it doesn't apply deletes for you.

The post also gets into position deletes vs deletion vectors vs copy on write, with a rough cost tradeoff table (equality delete is cheap to write and requires equality delete support to read, position delete requires resolving key to physical position and is heavier on write, deletion vectors need Iceberg v3 support specifically, copy on write is the most expensive to write but has basically universal read compatibility since there's no outstanding delete file involved).

The framework that's actually useful operationally: your safe feature set is the intersection of every required engine's capabilities, not the union. If Spark supports equality and position deletes but Snowflake only does position deletes, you write position deletes, because "at least one engine supports it" doesn't help you when you have three engines that all need to read the same table.

There's a decent pre production checklist too, don't just run a SELECT COUNT after your first write, actually insert some rows, update one, delete one, commit, then read the same snapshot from every engine you care about and diff both counts and values.

Full post if you want the details: https://olake.io/blog/iceberg-interoperability-myth-row-level-deletes/

Disclosure since it's relevant, I work on OLake, it gets a brief mention near the end, but the actual content here is engine agnostic and applies no matter what's writing your tables.

Has anyone actually hit this for real, table looks completely fine, one engine just refuses to read the current snapshot because of the delete encoding?


r/ApacheIceberg • • Aug 30 '26

Data Lakehouse with Apache Iceberg: A Guide

Thumbnail
lakeops.dev
1 Upvotes

r/ApacheIceberg • • Sep 08 '26

Apache Iceberg Table Cleanup: A Production Guide

Thumbnail
lakeops.dev
1 Upvotes

A practitioner's guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle.


r/ApacheIceberg • • 3d ago

Apache Iceberg Beginners Course: an Ethereum lakehouse with PyIceberg in Rosetta DBT Studio

Thumbnail
youtu.be
0 Upvotes

Covers the full Iceberg stack locally: SQLite catalog, Parquet/CSV/JSON imports, ACID snapshots, deletes and upserts, schema and partition evolution, sort order, time travel, tags and branches (write-audit-publish), metadata tables, and maintenance. Built inside Rosetta DBT Studio. All code on GitHub.


r/ApacheIceberg • • Aug 07 '24

Can't create iceberg tables in Databricks

Post image
0 Upvotes

I am using databricks runtime is DBR 14.3 LTS Spark 3.5.0. Scala 2.12. And using iceberg iceberg_spark_runtime 3_5_2 12 16 0.jar. Is it correct version I am using, because when I installed those jar in Databricks it is not recognizing iceberg command in notebook and not allowing me to create iceberg tables. I can create regular tables but not iceberg tables.

Resources used: https://www.dremio.com/blog/getting-started-with-apache-iceberg-in-databricks/

I have also tried multiple ways but no use.


r/ApacheIceberg • • Dec 10 '25

👋 Welcome to r/granicaai — Our First Post

Thumbnail reddit.com
0 Upvotes

r/ApacheIceberg • • Jun 07 '26

Preparing Your Iceberg Lake for AI Agent Queries

Thumbnail levelup.gitconnected.com
0 Upvotes

r/ApacheIceberg • • May 10 '26

Autonomous Iceberg Data Lake Management

Thumbnail
youtube.com
0 Upvotes

Hi, sharing this video - it's a commercial product but has a free tier, it automatically manages your lakehouse ops with Iceberg.

Meanwhile, here are a few useful links:

* New website: https://lakeops.dev/
* Platform: https://lakeops.dev/platform
* Solutions: https://lakeops.dev/solutions
(you can go into use-cases pages like managed Iceberg, cost reduction, Lake obesrvabilty, AI readiness etc)
* Docs: https://lakeops.dev/docs
* Video overview: https://www.youtube.com/watch?v=irRsF9VYP20

Interesting concept.

,


r/ApacheIceberg • • 28d ago

Real-Time Lakehouse Guide: Apache Iceberg in Production

Thumbnail
lakeops.dev
5 Upvotes

r/ApacheIceberg • • 18d ago

Latest status on GPU engines for Iceberg?

5 Upvotes

r/ApacheIceberg • • Aug 24 '26

Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files

Thumbnail
itnext.io
6 Upvotes

r/ApacheIceberg • • 4d ago

Streamhouse and Lakehouse with Apache Iceberg

Thumbnail
lakeops.dev
1 Upvotes

r/ApacheIceberg • • Aug 02 '25

Iceberg, The Right Idea - The Wrong Spec - Part 2 of 2: The Spec

Thumbnail database-doctor.com
0 Upvotes

(not an endorsement, but for discussion)


r/ApacheIceberg • • Nov 14 '25

Iceberg-Inspired Safe Concurrent Data Operations for Python

0 Upvotes

As head of data engineering, for years I am working with Iceberg in a large bank, but integrating for non-critical projects meant dealing with Java dependencies and complex infrastructure that I couldn't handle. I wanted something that would work in pure Python without all the overhead, please take a look at it, you may find it useful:

links:

install

pip install datashard

Contribute

I am also looker for a maintainer, so don't be shy to DM me.


r/ApacheIceberg • • May 11 '26

Managed Iceberg Lakehouse: A Practical Guide

Thumbnail itnext.io
1 Upvotes

r/ApacheIceberg • • Jun 05 '26

Apache Iceberg 1.11.0 — What's New?

Thumbnail
lakeops.dev
1 Upvotes

r/ApacheIceberg • • Jun 29 '26

A skill for optimizing Iceberg with Claude

Post image
2 Upvotes

In the last year I’ve been working with a very data-intensive organizations on their Apache Iceberg implementation.

Iceberg is genuinely easy to get started with, but I’ve seen many teams hit a wall in production. Latency creeps up and costs grow.

Most of the issues trace back to the same thing: Iceberg has a lot of knobs, and the defaults were not optimized for your workload.
Partition specs, compaction strategies, delete modes, snapshot retention - teams accept the defaults and the problems compound silently.

That’s why I built the Iceberg Optimizer Skill — a Claude Code skill that codifies this knowledge and guides teams through the decisions that actually matter.

It runs a structured diagnostic:
• Profile the table’s physical state
• Reconstruct write patterns and access behavior from metadata and logs
• Interview for the intent that can’t be inferred (latency SLAs, compliance, cost priority)
• Simulate suggested optimization scenarios across latency, cost and storage tradeoffs
• Generate engine-specific plans with exact parameters and schedules

The skill includes a benchmark of 20+ real-world failure scenarios — streaming death spirals, CDC delete accumulation, partition misalignment, GDPR ordering mistakes, bloom filter misuse and more.

It’s only v0.1 - support for Spark, Trino, AWS Glue, Snowflake and Flink is in, and there’s a lot more to add.

This is an evolving effort, and I’d love for the Iceberg community to contribute — whether that’s a failure pattern you’ve hit, a platform-specific edge case, or a recommendation you’d add.

If running Iceberg in production is hard, it shouldn’t stay that way.
📖 Blog post: https://www.itamarweiss.com/blog/apache-iceberg-optimization-skill/
💻 Repo: https://github.com/itamarwe/iceberg-optimizer-skill


r/ApacheIceberg • • Jul 21 '26

7 Managed Iceberg Lakehouse Solutions You Should Know

Thumbnail
levelup.gitconnected.com
2 Upvotes

r/ApacheIceberg • • Jan 03 '24

Hacker News discusses: Understanding Parquet, Iceberg and Data Lakehouses

Thumbnail news.ycombinator.com
2 Upvotes

r/ApacheIceberg • • Jun 21 '24

Snowflake: Open, Interoperable Storage with Iceberg Tables, Now Generally Available

Thumbnail
snowflake.com
2 Upvotes

r/ApacheIceberg • • Sep 14 '26

Unity Catalog: Pros and Cons

Post image
2 Upvotes

Apache Iceberg won the open table format war when Databricks acquired Tabular, followed by its subsequent adoption across the industry. Then, the catalog war began.

In a lakehouse, storing data in object storage and using the Apache Iceberg format is only part of the story. You also need a catalog that helps lakehouse query engines like Spark, Flink, or RisingWave discover tables, manage metadata, enforce access control, and work with governed data across different systems.

That is where Unity Catalog comes in.


r/ApacheIceberg • • Jul 20 '26

Debugging Apache Iceberg: A Production Guide

Thumbnail
itnext.io
2 Upvotes