qoega

It is meant for single reader/writer workload so not meant to be used as a service

In ClickHouse it is just `INSERT INTO t FROM INFILE 'data.csv.gz'`. Any supported format, any encryption, autodetected from file name and sample data piece to get column types, delimeters etc. Separate tools to convert CSV are not necessary if you can just import to db and export as SQL Statements.

echo "name,age,city John,30,New York Jane,25,Los Angeles" > example.csv

clickhouse local -q "SELECT * FROM file('example.csv') FORMAT SQLInsert" INSERT INTO table (`name`, `age`, `city`) VALUES ('John', 30, 'New York'), ('Jane', 25, 'Los Angeles');

Are you guy comparing 16vCPU/32GB vs 8vCPU/32GB and say yours is only 1.6 faster?

Now you rarely use basic MapReduce primitives, you have another layer of abstraction that can run on infrastructure that was running MR jobs before. This infrastructure allows to efficiently allocate some compute resources for "long" running tasks in a large cluster with respect to memory/cpu/network and other constraints. So basically schedulers of MapReduce jobs and cluster management tools became that good, because MR methodology had trivial abstractions, but required efficient implementation to make it work seamlessly.

Abstraction layers on top of this infrastructure now can optimize pipeline as a whole by merging several steps into one when possible, add combiners(partial reduce before shuffle). It requires whole processing pipeline to be defined in more specific operations. Some of them propose to use SQL to formulate task, but it can be done using other primitives. And given this pipeline it is easy to implement optimizations making whole system much more user-friendly and efficient compared to MapReduce, when user has to think about all the optimizations and implement them inside single map/reduce/(combine) operations.

You can even migrate your zookeeper to ClickHouse keeper. It requires small downtime, but you will have all your zookeeper data inside and your clients will just work when your keeper will be back

Did not expect to see issue I created

Open-source ClickHouse also allows both real-time and large historical data.

It is nice to have an image of expected dashboard in readme.

Passion and experience

I don't know why exactly you need BigQuery in your scenario. If I wanted to have Postgres->ClickHouse it would take single INSERT SELECT from ClickHouse.

If you need just CSV result dumped to GCS you can use clickhouse-local mode that has all the features like integrations with Postgres, GCP, formats.

Only but it is not a service with UI where your data analysts can click and drag what they want to export. But SQL can be simple for them to write and you need nothing more than a trivial cron job analogue to run it.