Good question. I wouldn’t say this replaces Flink in general. If you already run Flink and are comfortable with it, it’s a very powerful system.
Where we saw friction with Flink was mainly:
1.) Operational overhead (jobs, state backends, checkpointing)
2.) Generic sinks not being optimized for ClickHouse (batching, small inserts, etc.)
We focused on making scaling a property of the pipeline itself (just add replicas) and optimizing specifically for ClickHouse ingestion patterns.
So Flink is more general, this is more opinionated and focused on this specific use case.
Where we saw friction with Flink was mainly: 1.) Operational overhead (jobs, state backends, checkpointing) 2.) Generic sinks not being optimized for ClickHouse (batching, small inserts, etc.)
We focused on making scaling a property of the pipeline itself (just add replicas) and optimizing specifically for ClickHouse ingestion patterns.
So Flink is more general, this is more opinionated and focused on this specific use case.