jaysh
Joined 11 years ago
Joined 11 years ago
CREATE TABLE default.events (
`timestamp` DateTime
`event` String -- e.g. 'product.updated' or empty/null
`message` -- human readable message
`raw` -- the raw message - this is very useful when pushing logs that aren't JSON - you just let the `event` be null and dump the entire message here
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (timestamp, event)
TTL timestamp + toIntervalMonth(6)
ClickHouse is extremely performant even in the cases of e.g.: SELECT count(*) FROM `events` WHERE `raw` LIKE '%hello world%'Of course, the more columns you splat out (e.g. like correlation_id, user_id, order_id, etc) the better you can index and expect those queries to perform but in general I don't bother outside the obvious core domain ones (exampled above), the performance is so good that unindexed queries are significantly faster than indexed queries in Loki. I have reached the point where I JSON extract on-the-fly for the WHERE clause with very large queries with no meaningful performance issues.
http://highscalability.com/blog/2013/7/8/the-architecture-tw...
> Outliers, those with huge follower lists, are becoming a common case. Sending a tweet from a user with a lot of followers, that is with a large fanout, can be slow. Twitter tries to do it under 5 seconds, but it doesn’t always work, especially when celebrities tweet and tweet each other, which is happening more and more. One of the consequences is replies can arrive before the original tweet is received. Twitter is changing from doing all the work on writes to doing more work on reads for high value users.
If you're interested, an interesting measure for difficulty is how much companies charge to "buy youtube views" (e.g, $16 per 1000 views is ~8x the cost you'd pay for typical CAPTCHA solving).
Minor suggestion would be to remove the unlimited text messages from the plans. It's harder to change later when this takes off, is ripe for abuse, and (to me) makes it feel unsustainable: everyone know unlimited is not actually unlimited. I'd reserve that for for the "call us" enterprise pricing options.
I wouldn't assume that just because people vocalise brand loyalty, means that they stick with it when it's time to open their wallets.
After we were picked up, the driver helpfully informed us if we'd ordered the same ride via Ola (he drives both), it'd have been ~50% cheaper (as a new user).
Naturally, I installed it and now I'll switch between Uber and m Ola. This then expands to Bolt, Grab, and so on, and you realise it's virtually the same experience with different branding and the sole differentiating factor is price.
I definitely buy the convenience aspect: it's way more comfortable to pull out your phone and order a ride on a platform you're already familiar with. My trouble is the barrier is so low to install another app (and new joiner incentives are financially motivating) that this leads to virtually no brand loyalty.
Perhaps in a different world, where there are no up significant upfront financial incentives and perhaps drivers were exclusive to a platform (i.e. to pass their rigorous screening processes is a sign of quality), that'd certainly buy some loyalty from me.
> If I need to go somewhere, or get home, Uber has never let me down.
The real test of brand loyalty it staying with them after a few bad experiences.
- substantially better performance on the same hardware, even moreso for larger range queries (multiple days)
- no new query language to learn
- significantly more expressive as you said
- agents for scraping logs use way less CPU (I used to use grafana-agent which used about 80%, vector uses sub 5%)
- very intuitive to manage TTLs - I can keep some logs for 10 years, and some for 1 week based on the event in the JSON
- more compact storage, I didn't check scientifically but CH storage is better compressed at least 4-5x for us
- no running into maximum stream limits - struggled with these even on Grafana Cloud and didn't realise we silently lost a lot of logs
Honestly: why wouldn't you. Loki always felt like a mistake to me. A brand new query language, really counter-intuitive configuration, large ramp-up time for complex queries, lots of arguing about labels/cardinality etc. It all goes away when you drop it. I think logging should not be exotic or behave in unexpected ways.