TLDR: I made "poor man’s" DSSM (Deep Structured Semantic Model) — the count-based translation table that can enrich the inverted index for full-text search. This trick can improve baseline BM25.
So the idea is the following:
- You have supervised pairs (query, relevant document), e.g., MS MARCO or click logs.
- You tokenize both sides into some units (char n‑grams, wordpieces, words).
- You count cross‑pair co‑occurrences: unit u on the document side vs. unit v on the query side (not co‑occurrence within the same text).
- For each document‑side unit u, you keep the top‑k query‑side units v with the strongest association.
- At indexing time, each document gets postings not only for its own units, but also for the top‑k associated units of each of its units — i.e., document expansion baked into the inverted index.
It’s like mixing synonyms into the search query (but it’s not a synonyms exactly). The one difference from the DSSM is that it can only handle linear dependencies whilst DSSM can do the non-linear one.
And so it improves the performance over BM25 baseline.
I packed it as hf model repo: https://huggingface.co/mirth/msmarco-expansion-tables with a small usage demo script.
I am not claiming that this is a new idea. I made it because it’s fun and I’m planning to use it in my own search engine project.
So the idea is the following:
- You have supervised pairs (query, relevant document), e.g., MS MARCO or click logs.
- You tokenize both sides into some units (char n‑grams, wordpieces, words).
- You count cross‑pair co‑occurrences: unit u on the document side vs. unit v on the query side (not co‑occurrence within the same text).
- For each document‑side unit u, you keep the top‑k query‑side units v with the strongest association.
- At indexing time, each document gets postings not only for its own units, but also for the top‑k associated units of each of its units — i.e., document expansion baked into the inverted index.
It’s like mixing synonyms into the search query (but it’s not a synonyms exactly). The one difference from the DSSM is that it can only handle linear dependencies whilst DSSM can do the non-linear one.
And so it improves the performance over BM25 baseline.
I packed it as hf model repo: https://huggingface.co/mirth/msmarco-expansion-tables with a small usage demo script.
I am not claiming that this is a new idea. I made it because it’s fun and I’m planning to use it in my own search engine project.