Joined 9 years ago
A lot has changed in two years!
That's the reason why the default maximum edit distance of SymSpell is 2.
Now, all your 6 out of 6 examples are chosen from that 1.1% margin that is not covered by edit distance 2, presenting a rather unlikely high amount of errors within a single word.
The third-party SymSpell port from Justin Willaby, which you were using for benchmarking, clearly states that you need to set both maxEditDistance and dictionaryEditDistance to a higher number if you want to correct higher edit distances. That you neither used nor mentioned. This has nothing to do with accuracy; it is a choice regarding a performance vs. maximum edit distance tradeoff one can make according to the use case at hand.
https://github.com/justinwilaby/spellchecker-wasm?tab=readme...
pronnouncaition IS within edit distance 3, according to the Damerau-Levenshtein edit distance used by SymSpell. The reason is that adjacent transpositions are counted as a single dit. https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_di...
Unfortunately, the comparison of Lexiathan vs. Symspell on your website regarding accuracy is misleading.
1. SymSpell has two parameters to control the maximum edit distance. Once you set both to 3, then also terms with an edit distance of 3 are accurately corrected:
pronnouncaition -> pronunciation
inndappendent -> independent
unegspeccted -> unexpected
soggtwaee -> software
maggnificntally -> magnificently
annnesteasialgist -> anesthesiologist
SymSpell accurately corrects all of your examples if used properly with the correct parameters and dictionary.
Apart from that, your methodology of comparing correction accuracy by cherry-picking specific terms without statistical significance, where your product seemingly performs better, is questionable.
One would use large public corpora to measure the percentage of accurately corrected terms as well as the percentage of false positives.
Because SymSpell is Open-Source, everyone can integrate it into their applications for free, modify the code, use different dictionaries in various languages, or add terms to existing ones.
The query "to be or not to be" that you mentioned, consisting solely of stopwords, returns complete results and perform quite well in the benchmark: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#be...
Both Lucene and Elastic still offer stopword filters: https://lucene.apache.org/core/10_3_2/analysis/common/org/ap... https://www.elastic.co/docs/reference/text-analysis/analysis...
https://en.wikipedia.org/wiki/Kademlia
https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia...
https://web.archive.org/web/20120128120732/http://www.cs.ric...
It would increase concurrent read and write speed (index loading, searching) by removing the need to lock seek and read/write.
But I would expect that the mmap implementations do already use io_uring / IoRing.
Yes, lazy loading would be possible, but pure RAM access does not offer enough benefits to justify the effort to replicate much of the memory mapping.
The SeekStorm server features an REST API via http: https://seekstorm.apidocumentation.com
It also comes with an embedded Web UI: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#bu...
Or did you mean a Web based interface to create and manage indices, define index schemas, add documents etc?
We just released a new OpenAPI based documentation for the SeekStorm server REST API: https://seekstorm.apidocumentation.com
For the library we have the standard rust doc: https://docs.rs/seekstorm/latest/seekstorm/
For SimilarityType::Bm25fProximity which takes into account the proximity between query term matches within the document, we have so far only anecdotal evidence that it returns significantly more relevant results for many queries.
Systematic relevancy benchmarks like BeIR, MS MARCO are planned.
You add the library via 'cargo add seekstorm' to your project which you anyway have to compile.
As for the server, we may add binaries for the main OS in the future.
WASM and Python bindings are on our roadmap.