Comment Comment by ch2026 1) clickhouse async_insert would have solved all your issues: https://clickhouse.com/docs/optimize/asynchronous-inserts1a) If you’re still having too many files/parts, then fix your partition by, and mergetree primary key.2) why are you writing to kafka when vector dev does buffering / batching?3) if you insist on kafka, https://clickhouse.com/docs/engines/table-engines/integratio... consumes directly from kafka (or since you’re on CHC, use clickhouse pipes) — what’s the point of vector here?Your current solution is unnecessarily complex. I’m guessing the core problem is your merge tree primary key is wrong. parent ch2026 1 year ago
1a) If you’re still having too many files/parts, then fix your partition by, and mergetree primary key.
2) why are you writing to kafka when vector dev does buffering / batching?
3) if you insist on kafka, https://clickhouse.com/docs/engines/table-engines/integratio... consumes directly from kafka (or since you’re on CHC, use clickhouse pipes) — what’s the point of vector here?
Your current solution is unnecessarily complex. I’m guessing the core problem is your merge tree primary key is wrong.