This is nice. At my last job, we developed an extremely high performance HyperLogLog server (hlld). Take a look if you're interested:
https://github.com/armon/hlld
It processes over a 1MM ops per second on a basic MacBook Air. In production we saw it handle more. We never had to scale it over one server in order to handle millions of distinct metrics.
As an ad company, my last company had a ridiculous fanout of metrics for every request. Every ad serve API request (of which we got thousands per second) would fan out to over 150 metrics EACH. So for 1000 requests per second we'd have to modify 150,000 metrics. Raw SQL updates of course fall down with this (even if we tone back their durability). Instead, we opted for accumulated for 10 to 60 seconds in bloomd and hlld and statsite, then aggregating and doing a single bulk SQL update. Worked incredibly well.
We used it in combination with bloomd: https://github.com/armon/bloomd Bloomd is another super high performance data structure server, but this time for bloom filters.
You can read the READMEs to learn why they're better for their specific tasks. Both have been running in production for 2+ years. If you're not going to be using the data structures that much then Redis is a good choice. We personally used these servers alongside Redis for k/v.
https://github.com/armon/hlld
It processes over a 1MM ops per second on a basic MacBook Air. In production we saw it handle more. We never had to scale it over one server in order to handle millions of distinct metrics.
As an ad company, my last company had a ridiculous fanout of metrics for every request. Every ad serve API request (of which we got thousands per second) would fan out to over 150 metrics EACH. So for 1000 requests per second we'd have to modify 150,000 metrics. Raw SQL updates of course fall down with this (even if we tone back their durability). Instead, we opted for accumulated for 10 to 60 seconds in bloomd and hlld and statsite, then aggregating and doing a single bulk SQL update. Worked incredibly well.
We used it in combination with bloomd: https://github.com/armon/bloomd Bloomd is another super high performance data structure server, but this time for bloom filters.
You can read the READMEs to learn why they're better for their specific tasks. Both have been running in production for 2+ years. If you're not going to be using the data structures that much then Redis is a good choice. We personally used these servers alongside Redis for k/v.