* We imported ~1,000,000 messages of dynamic length (some very long, eg. emails);
* Once imported, the search index weights 20MB (KV) + 1.4MB (FST) on disk;
This is almost unbelievably succinct! If you encode the document features into 8 bits per document, and thus completely forego the need to store the document ID by indexing them implicitly, that alone is 1 MB.
Getting meaningful search out of on average 21 bytes per document seriously impressive.
* Once imported, the search index weights 20MB (KV) + 1.4MB (FST) on disk;
This is almost unbelievably succinct! If you encode the document features into 8 bits per document, and thus completely forego the need to store the document ID by indexing them implicitly, that alone is 1 MB.
Getting meaningful search out of on average 21 bytes per document seriously impressive.
[For reference, this sentence is 42 bytes.]