It's basically a workaround for DuckDB's lack of native support. I am very happy with the Pyicerbg library as a user, It was very easy and the native Arrow support is a glimpse into the future. Arrow as an interchange format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it!
Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For example, Clickhouse python also support arrow-based insertion:
https://duckdb.org/docs/stable/extensions/iceberg/overview.h...
Apache iceberg go ? Nope
https://github.com/apache/iceberg-go?tab=readme-ov-file#read...
Basically java iceberg is the only mature way to do this, it's not a very accessible ecosystem.
For a side project I'm using pyiceberg to sink streaming data to iceberg (using DuckDB as the stream processor):
https://sql-flow.com/docs/tutorials/iceberg-sink
It's basically a workaround for DuckDB's lack of native support. I am very happy with the Pyicerbg library as a user, It was very easy and the native Arrow support is a glimpse into the future. Arrow as an interchange format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it!
https://github.com/turbolytics/sql-flow
Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For example, Clickhouse python also support arrow-based insertion:
https://sql-flow.com/docs/tutorials/clickhouse-sink
This makes the glue code trivial to sink into these different systems as long as arrow is used.