Nilithus
Joined 13 years ago
Joined 13 years ago
I think there is a big jump from the tech literacy to install Arch and the tech literacy to maintain an Arch system over time. These derivatives feel like they are there to help solve the latter.
They don’t even mention their product till the very last section. Overall think it’s an excellent blog post.
Legit made me chuckle
I think one of the big tools is Shuffle Sharding. The article talks about standard sharding by itself as not being enough to provide robustness in multitenant queues. But Shuffle Sharding I.E. assigning users to virtual groups of underlying queues and enqueueing to the queue with the smallest size gets you pretty far. It can limit throughput for individual users but implementing some simple work stealing logic on the consumer helps make sure you keep your throughput up.
Has some good visualizations as well.
Since many of those use cases probably already run extensive on-prem infrastructure this could appeal to them. AWS outpost talks about industries like healthcare, telecom, media and entertainment, manufacturing, or highly regulated spaces like financial services. I've heard of media companies that process through things like IMAX cameras that have just tons of TB's of data sometimes just for 5 minutes worth of footage. That would simply be too cost prohibitive - in bandwidth alone - to try and move around in the cloud and you don't want to have to wait for things like AWS snowball or whatever.
While I think the space is "niche" those niche spaces are not small. Big companies with big budgets.
I saw in your socklimit project looks like `fasthash64` and `hashlittle` I'm not familiar with those any insight or recommended reading to understand these hash functions?
p.s. googling pairwise independent hash functions did get me some college class reading but doesn't mention any named hash functions developed out in the world.