breadislove

yeah, but the bigger the corpus the more you are relying on the LLM guessing the right keyword, if there is no reranking in place, you will just end up burning through a lot of tokens.

coding agents is a very "niche" use-case where lexical matching works very well. code is a graph, by looking at few lines of code you can guess naming conventions etc. codebases are small compared to knowledge bases.

the bigger your index, the more you will need way better tools such as search.

ceramic is very very bad, the worst search provider of all. it feels like it is doing only lexical matching and the index is very outdated

One super important thing missing: Speculative decoding. Things like Dflash(2), Dspark etc. help to do one forwards pass and get 6-7 tokens out of it. (For completeness, the embeddings from the forward pass are passed into a diffusion model which predicts the next tokens, and the model just verifies it (very cheap operation)). So we can produce way more tokens for roughly a similar amount of compute.

I am curious if there is a way to speed it up. Would be very interesting to know how long a human expert takes to finish the task.

yes for the retrieval benchmarks. For officeqa pro v2 we used Codex (as databricks did) and for Harvey LAB we used the vanilla harvey benchmark. For these benchmarks we added minimal tools to use mixedbread search and toast 1.

there is full lore around the naming. i can guarantee you that we are pretty dedicated around our research and product.

the issue with smaller general models (see at the charts) are way behind the frontier models when it comes to search. we've found that there is huge uplift of having a fast dedicated model. from our perspective, having a very good index is the biggest lever and then having a specialised model.

we have not converged at all, if you look at how different the chinese models in terms of architecture you can guess that the labs are experimenting a lot as well. we are seeing all different types of hybrid architectures, different attention methods and so on. Of course on a high level its still a transformer but if you take a proper look we are seeing more divergence then a convergence.

On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?