jordepic

TL;DR: IceStream keeps a secondary index of iceberg data on object store to enable efficient conversions from equality deletes to positional deletes in a horizontally scalable manner. After benchmarking this a bit you can see order of magnitude improvements on large tables, which is hugely important for streaming pipelines!

A follow up to my previous post - icestream is an asynchronous compaction service for iceberg tables with many equality deletes (a symptom of frequent streaming writes on tables with "primary keys"). Now, instead of relying on Cassandra + Spark to index Apache Iceberg table data, Icestream uses Flink and Apache Paimon - enabling a separation between compute and storage and keeping an LSM tree style index on disk.

Icestream is an asynchronous table maintenance tool for iceberg tables that converts equality deletes (write optimized) to positional deletes (read optimized). It does so efficiently by leveraging an index over the table data stored in Cassandra, and the cassandra-spark-connector, which enables efficient distributed nested-loop indexed joins from iceberg data in spark to the index data in the database.