Comment Comment by tingfirst Data sources are usually in Kafka, or other operational databases like Postgres or MySQL1. Table A : fact events, high-throughput (10k~1M eps), high-cardinality2. Table B, C, D : couple of dimension tables (fast or slow changing).The use case is straightforward : join/enrich/lookup everything into one big flattened, analytics-friendly table into ClickHouse.What’s the best pipeline approach to achieve this in real-time and efficiently? parent tingfirst 1 year ago
1. Table A : fact events, high-throughput (10k~1M eps), high-cardinality
2. Table B, C, D : couple of dimension tables (fast or slow changing).
The use case is straightforward : join/enrich/lookup everything into one big flattened, analytics-friendly table into ClickHouse.
What’s the best pipeline approach to achieve this in real-time and efficiently?