olivernn
Joined 16 years ago
Karma
175
Submissions
20
Member since
Dec 9
Recent submissions
Comment · 1 year ago
Many years ago, when I was building Lunr, it was initially based on IndexedDB, but it was _much_ slower and the datasets I had in mind easily fit in memory so I didn’t pursue it. No idea if that has changed since though.
Comment · 9 years ago
Using `charCodeAt()` for numeric keys is an interesting optimisation, I'm definitely going to try that out with Lunr.js.
I'm also interested in _why_ that is quicker though, aren't all object keys converted to strings anyway?
Comment · 9 years ago
Lunr.js [1] used to use a trie data structure and memory usage was a concern. The problem is that although the data structure compresses prefixes, it does nothing for suffixes. Depending on the corpus this can lead to a lot of duplication.
Switching to a trie-like structure that compresses prefixes and suffixes can lead to significant savings. Building the structure can be a bit more burdensome, so there is a trade off there. There is a paper describing the approach [2] and, if you're interested, my JavaScript implementation [3][4].
[2] http://www.aclweb.org/anthology/J00-1002.pdf
[3] https://github.com/olivernn/lunr.js/blob/master/lib/token_se...
[4] https://github.com/olivernn/lunr.js/blob/master/lib/token_se...
Comment · 11 years ago
The last time this article was on HN I took a look at adding Okapi BM25 to lunr, from what I remember the changes don't seem to huge, its just a matter of getting the time to sit down and implement it!
Comment · 11 years ago
I'm fairly sure elastic lunr is a fork of lunr, I still seem to have the most commits even! [1]
I'll have to take a look and see what @weixsong added, perhaps there are some changes that I can merge upstream.
[1] https://github.com/weixsong/elasticlunr.js/graphs/contributo...
Comment · 11 years ago
I've come across these two:
* http://reyesr.github.io/fullproof/ * http://www.tipue.com/search/
Might be interesting to see how these also stack up.
Comment · 11 years ago
Nice. It'd be interesting to see the comparative performance of the three libraries under test, e.g. time to results and possibly memory usage.
Comment · 12 years ago
Who'd have thought the most English looking location would be in Wales ;)
Comment · 13 years ago
Why is it a red flag for a browser-based javascript library to require a browser for testing?
Comment · 13 years ago
That is almost exactly what lunr is doing. It tokenises the input text, stems the tokens and filters out any stop words. The index it can be searched, the order is not relevant, a prefix search is currently used so that you can find documents containing terms without having to type the whole term exactly. The matching documents are also scored as to how relevant they are to the search term.
In the future I want to add even more powerful querying, restricting search to specific fields, taking into account the distance between terms, and adding faceted search to reduce the total documents being searched over.
One of the original goals of the project was specifically to provide a better alternative to just using the browsers built in find-in-page functionality
Comment · 13 years ago
Since the library can be run outside of the browser (using node.js for example) the index could be generated server side, and then just passed to the client. I hadn't considered this before but it might be worth looking at.
Comment · 13 years ago
The example (http://lunrjs.com/example/) indexes 100 stackoverflow questions, some of which are relatively long.
If indexing performance starts to become an issue the whole search index can be moved into a web-worker, which prevents indexing from blocking the rest of the page.