Comment

Comment by mrlongroots

Opinionated take: ingestion should be treated as a streaming reorganization workload, separate from whatever a "database" is.

You do not even need to change the Iceberg spec, although you could. Just rejig things a bit to produce better statistics (think liquid clustering but on the write path). You still write the same dataset queried the same way, but automatically get better query performance. Workload owners should be able to decide the tier at which this streaming reorganization happens: they know their data staging and bandwidth constraints the best.

parent mrlongroots 3 months ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton