I’m happy a lot of work is being done in this space. Lucene based search has long dominated, and search solutions like this, Vespa, Jina, etc are pushing the open source search space forward and at a minimum forcing Lucene to grow. In particular:
- Native, first class support for dense vectors first, no need to hack around the inverted index like this talk by Simon Hughes: https://haystackconf.com/2019/vectors/
- Using native programming languages, avoiding the JVM and all the memory and GC overhead
- Hopefully native languages might make it easier to deploy code that uses GPUs!
- A chance to focus on info retrieval when current Lucene solutions are either a bit crufty (Solr) or very focused on logs/analytics (Elasticsearch)
- a chance to have one system that unifies search and recommendations in one backend, not need to split them between two systems
- a chance to rethink the language used to specify query ranking factors. Solr has a very unique DSL that’s very write-only. Elasticsearch is very verbose to do anything non trivial.
Still I do wonder even if Lucene catches up a little, it’s current mindshare, reliability and install base will continue to dominate. For example see Lucene 9004 for adding dense vector search: https://issues.apache.org/jira/plugins/servlet/mobile#issue/...
The holy grail is a system that does a good job at classic full text search AND dense vector search. I wonder what system has the best chance of delivering that.
- Native, first class support for dense vectors first, no need to hack around the inverted index like this talk by Simon Hughes: https://haystackconf.com/2019/vectors/
- Using native programming languages, avoiding the JVM and all the memory and GC overhead
- Hopefully native languages might make it easier to deploy code that uses GPUs!
- A chance to focus on info retrieval when current Lucene solutions are either a bit crufty (Solr) or very focused on logs/analytics (Elasticsearch)
- a chance to have one system that unifies search and recommendations in one backend, not need to split them between two systems
- a chance to rethink the language used to specify query ranking factors. Solr has a very unique DSL that’s very write-only. Elasticsearch is very verbose to do anything non trivial.
Still I do wonder even if Lucene catches up a little, it’s current mindshare, reliability and install base will continue to dominate. For example see Lucene 9004 for adding dense vector search: https://issues.apache.org/jira/plugins/servlet/mobile#issue/...
The holy grail is a system that does a good job at classic full text search AND dense vector search. I wonder what system has the best chance of delivering that.