Dynamic Tables are interesting for declarative streaming. In the ClickHouse ecosystem, you might want to look at materialized views combined with streaming engines.
For real-time transformations, there are a few approaches:
- Native ClickHouse MaterializedViews with AggregatingMergeTree
- Stream processors that write to ClickHouse (Flink, Spark Streaming)
- Streaming SQL engines that can read/write ClickHouse
We've been working on streaming SQL at Proton (github.com/timeplus-io/proton) which handles similar use cases - continuous queries that maintain state and can write results back to ClickHouse. The key difference from Dynamic Tables is handling unbounded streams vs micro-batches.
What's your specific use case? Happy to discuss the tradeoffs.
For real-time transformations, there are a few approaches: - Native ClickHouse MaterializedViews with AggregatingMergeTree - Stream processors that write to ClickHouse (Flink, Spark Streaming) - Streaming SQL engines that can read/write ClickHouse
We've been working on streaming SQL at Proton (github.com/timeplus-io/proton) which handles similar use cases - continuous queries that maintain state and can write results back to ClickHouse. The key difference from Dynamic Tables is handling unbounded streams vs micro-batches.
What's your specific use case? Happy to discuss the tradeoffs.