realfun

I think it would take quite a long while to achieve human-level anti-entropy in Agentic systems.

Complex system requires tons of iterations, the confidence level of each iteration would drop unless there is a good recalibration system between iterations. Power law says a repeated trivial degradation would quickly turn into chaos.

A typical collaboration across a group of people on a meaningfully complex project would require tons of anti-entropy to course correct when it goes off the rails. They are not in docs, some are experiences(been there, done that), some are common sense, some are collective intelligence.

Agreed. It does need memorization. Anyone who don't think so can try to add and multiply in hexadecimal, what is 0xA + 0xC and what is 0x8 * 0x6? It becomes quite obvious memorization is required when you try it in any other radix other than decimal(which is memorized already).

Roaring bitmap works surprisingly better than bloom filter in Elastic-search as skip list plugin, completely beats the built-in skip list function.

Is there a reliable way to tell which airplane type before booking?

Thanks for sharing EGreg.

Elasticsearch has geo-indexing as well(based on geohash internally), and by default it does id hashing similar to what you said(murmurhash3), we actually leverages that for location based searches.

The challenge addressed in the blog is not in how to search/address(as said Elasticsearch handles it already), it is about how to distribute the load so calculation only happens on limited nodes, and reduce the index size so it can be more performant.

not just near the poles. Geohash distortion is already very obvious when you compare UK and countries in south Asia.

Same strategy can be applied regardless of ES version. ES6 does has index sorting and other optimizations, but not as good as pre-sharding for location based services, cases like Tinder.

geo-indexing is different from geo-sharding, search indexes normally have geo-indexing, but it is still one big index handles every single search. The alternative(geohash) is mentioned in article, which has high distortion around earth poles.

Thanks for the suggestion and there are definitely room for optimization.

In the original design we did consider approach similar to the "heatmap approach" you mentioned, it would reduce the shard movement for people who commutes in the city.

Later we figured that simply sharding along on the Hilbert Curve already give what we need, and the shard move is unavoidable, we would explain more details around the shard move in future blog.