Comment

Comment by misterman0

FTS is dead. It didn't use to be dead but it is surely dead today. For years and years and years we were told to use stop words, integrate a stemming library, make sure term weights are normalised with TF-IDF, maintain a synonyms dictionary and use an index that keeps track of term positions to be able to cater for term proximity and if you did all that the relevance of your search results would be top notch. Top notch! Today none of that is relevant. Today you need to provide semantic search ("SS") or your search will be considered broken because the results will truly suck. To provide SS you need ML.

FTS is still commonly included in e-commerce where I guess it's not quite dead but only because truly relevant search results are irrelevant to retailers' bottom line.

SS is at least one magnitude more complex to set up, especially if you want to be able to refresh your "index" and even more so if your data is big. What a typical back-ender would wip up in a couple of days using ES/Solr you now need a proper Math Dev to do and for the MD's model to become useful you need them to hand it over to a distributed systems expert.

SS through ML is commonly a nasty, duct-taped work-flow that at best results in a system that looks more like a POC or DEMO than a proper production system, unless you work at Bing/Google that is (probably, but I have no hands-on experience of those systems).

I've been trying for years, I'd say at least ten, to try to "commoditize" this work flow, make it simpler, more usable for generalist devs (not just ML people) but no matter what I do I keep getting crushed under the weight of the data. To me it seems search is dead and we haven't appointed a new king.

parent misterman0 7 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton