Joined 14 years ago
:oof-1: CH needs more disk to keep up with merges.
We ran CH way too lean in those days.
And we are hiring. Come join us! https://posthog.com/careers
- Declarative, cluster aware migrations
- Iceberg and S3 backed merge tree's on ClickHouse (as performant as local MergeTrees)
- ClickHouse on K8s, but on Metal instances w/ NVMe used for caching and compaction.
- Load shedding query scheduler and eventually query optimizer
We have a broad charter when it comes to ClickHouse. It's one of the things that enables us to build so many products that share the same underlying data and context. Our goal is to enable all developers at PostHog to continue building these products quickly as the team and user-base scales.
It's honestly an exciting time to be working at this company and on the beating heart of what drives it. Chat with me if you have any questions!
Since we've been using ClickHouse long before this JSON functionality was available (or even before the earlier version of this called `Object('json')` was avaiable) we ended up setting up a job that would materialize json fields out of a json blob and into materialized columns based on query patterns against the keys in the JSON blob. Then, once those materialized columns were created we would just route the queries to those columns at runtime if they were available. This saved us a _ton_ on CPU and IO utilization. Even though ClickHouse uses some really fast SIMD JSON functions, the best way to make a computer go faster is to make the computer do less and this new JSON type does exactly that and it's so turn key!
https://posthog.com/handbook/engineering/databases/materiali...
The team over at ClickHouse Inc. as well as the community behind it moves surprisingly fast. I can't recommend it enough and excited for everything else that is on the roadmap here. I'm really excited for what is on the horizon with Parquet and Iceberg support.
To say big data is dead sounds to me like someone desperate for eyeballs.
I do think there is a huge opportunity for DuckDB - running analytics on 'not quite big data' is a market that has always existed and is arguably growing. I've seen way too many people trying to use Postgres for analyzing 10 Billion row tables and people booting up an EMR cluster to hit the same 10 Billion rows. There is a huge sweet spot for DuckDB here were you can grab a slice of the data you are interested in, take it home and slice and dice it as you please on your local computer. I did this just this weekend on DuckDB _and_ ClickHouse!
Disclaimer: I work at a company that is entirely based on ClickHouse.
Our mission is to build tools that will increase the number of successful products that exist in the world.
We are looking for Backend Engineers & SREs to take care of scaling to 1 Trillion+ events, both for our cloud offering and larger customers who are self hosting. If you are interested in processing data reliably at scale and love databases like ClickHouse, this is your role!
We have a culture of written async communication (see our handbook [1]), lots of individual responsibility and an opportunity to make a huge impact. Being fully remote means we're able to create a team that is truly diverse. We're based all over the world, and the team includes former YC founders, CTOs turned developers and recent grads.
To apply see https://posthog.com/careers or email us [email protected]
[0] https://github.com/posthog/posthog [1] https://posthog.com/handbook/
