ryptophan

Short version: I used a few retrieval benchmark datasets, and experimented with a 21 embedding model leaderboard. I held the benchmark corpus and relevance labels fixed, and transformed the queries: paraphrase, terse, verbose. I kept the “information need” the same under each query transformation, but the leaderboard order changed more than I expected.

I reproduced published benchmark scores for benchmark dataset + embedding model pairs, and then used that same setup with transformed queries as the input. I observed the reordering under each rewrite style to be consistent across 4 independently-generated rewrites (gemini-2.5-pro, claude-opus-4.8, gpt-5.1, llama-3.3-70b) rather than showing up in just one.