Kerollmops
Joined 10 years ago
Karma
113
Submissions
69
Member since
Nov 1
Recent submissions
Comment · 4 months ago
Actually, you should take a look at [1]. It's made in Rust, inspired by LMDB, and supports a cool feature that allows close to anything to be implemented: allocating any page you want to store anything you want. The BTree storage is optional and you can implement whatever storage system you want. When storing a value to disk, you can allocate pages and decide exactly how you plan to store the bytes, allowing you not to store the length of them or to split your data into multiple pages, etc.
[1]: https://pijul.org/posts/2021-02-06-rethinking-sanakirja
Comment · 4 months ago
Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch).
But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9...
Comment · 4 months ago
Yeah, and I also added support for parallel read your own writes where you can write tons of entries and spawn multi children read-only transactions from your writes where transaction and read from them in parallel.
We use this in Meilisearch [1] to post-process cache for our most common prefixes i.e., "w" will match "work", "word"... and computing this requires doing large unions of the documents matching those words.
Being able to do it in parallel is necessary, especially when you have billions of entries to operate on.
Comment · 4 months ago
I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2].
I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compaction of the database and write performances are way better after that. We never had any major DB corruption... I mean... other than when using it on Azure. Azure never works, that's expected, I suppose.
[1]: https://mastodon.social/@hyc/116838499082046918 [2]: https://bugs.openldap.org/show_bug.cgi?id=10522 [3]: https://github.com/meilisearch/meilisearch
Comment · 10 months ago
So nice! That's an excellent extract and looks useful for benchmarking Meilisearch. I'll probably spend my Christmas holidays importing the tracks, albums, and artists into Meilisearch, while my CEO builds a beautiful front-end for it. I'll probably replace [the current music search demo](https://music.meilisearch.com) we have with this much higher-quality dataset!
That would also be a good fit for [the new delta-encoded posting lists I am working on](https://github.com/meilisearch/meilisearch/pull/5985). Let's see how good it can get. My early benchmarks showed a 50% reduction in disk usage.
Comment · 1 year ago
Someone reported it, and I answered today [1]. It's a rule that is too hard on the front end, and we will fix it by using a better Hybrid search setup (not only semantic). Thank you for the report.
[1]: https://github.com/meilisearch/meilisearch/issues/5504#issue...
Comment · 1 year ago
V1.14, released yesterday [1], ships with a search embedding cache. Most of the time you see is spent waiting for an OpenAI embedding answer. We also just shipped composite embedders to reduce the network latency when you need to respond quickly to user searches (by running embedders on the Meilisearch server) but still use external APIs to index many documents in batches. Note that it can only work with open-source embedders, the ones HuggingFace serves.
[1]: https://github.com/meilisearch/meilisearch/releases/tag/v1.1...
Comment · 1 year ago
HuggingFace is using Meilisearch, in production, on their website for a year now.
Comment · 1 year ago
You should try Meilisearch then, you'll be astonished by the quality of the results and the ease of setup.
Comment · 1 year ago
Right. We released a lot of new versions of the engine to improve the indexing part of it. V1.12 is improving the document indexing a lot! Have you tried the latest version v1.14 we released yesterday?
While Meilisearch is capable of limiting it's resident (actual mallocs) memory. However, it requires a bare minimum (about 1GiB).
Comment · 1 year ago
Meilisearch is faster when you reduce the dataset by filtering it. I wrote an article on this subject [1].
[1]: https://blog.kerollmops.com/meilisearch-vs-qdrant-tradeoffs-...
Comment · 1 year ago
35 GiB is probably a third of the data I index into Meilisearch just for experimenting and don't forget about the inverted indexes. You wouldn't use any O(n) algorithm to search in your documents.
Also, every time you need to reboot the engine you would have to reindex everything from scratch. Not a good strategy, believe me.
Comment · 1 year ago
> [..] to simplify the setup?
It would be simpler to keep Meilisearch and its key-value store out of Postgres' WAL and stuff and better propose a good SQL exporter (in the plan).
I have GitHub and Twitter accounts. I let you guess my handles.