adchurch

> I’m not really sure, I can’t say I trust any kind of synthetic AI benchmarks.

Fair enough but what exactly were you thinking when you said:

> I would like to see this (and any model router, frankly) benchmarked against two things

And thanks for sharing about failure cases! Those do sound like strange harness-level things, honestly we haven't seen failure cases like that crop up in our own usage & testing.

Great idea, we’d love to run those benchmarks! Any in particular you trust?

Also I’m very interested in the unique failure cases you’re referring to! What have you noticed?

Routing is the type of decision shape Jev is meant for. But from our own testing (plus externally run benchmarks) it’s much worse at routing than our purpose-built model.

I’d argue it’s similar to looking at the Meta algorithm for serving ads and asking “what about astra” - it could come up with an answer but it’s not the right model for the task.

Yes! We believe this is entirely possible. Frontier LLMs are trained to solve different problems. We’re training a frontier router :)

We support all the LLMs that Copilot sends data to!

My calculation exactly!

Not yet! Something we're interested in experimenting with though, hit me up at [email protected] if you have any thoughts here

For our hosted version we carefully select which providers we use (we don't use OpenRouter). For the self-hosted version though, OpenRouter does make it much easier to get started (at the cost of that variance potentially affecting quality).

Yes for both! We have some logic to put providers on cooldowns and/or deprioritize them.

To the point of using buckets of models: as long as there's >0 models available in a bucket, and we can order models in order of fit, we're resilient to different sets of models being available. With that said obviously cutting out some models has a much larger effect than others.

OpenRouter isn't strictly necessary but it does make it easier to not have to set up accounts/API keys with several different providers to get started.

Different frontier models are good at different things! We'll be the ones combining them optimally.

I haven't tried it as recently but last time I checked they only route once per session (or subagent). Imo this is basically impossible to do correctly. Consider the case where you start with one prompt "rewrite this in rust". Trivial in a 1 day old repo, extremely difficult in e.g. the VSCode repo!

Yes!

You can think of buckets as models with similar capabilities. So for example Deepseek 4.1 Flash will not be in the same bucket as Astra.

The latter is an interesting question! In practice because the session continues, we can see it's going down the wrong path and escalate. Basically no decision we make when routing is entirely unsalvageable (but we do have a performance penalty for every incorrect decision we make so of course we try to avoid it).

It is not sorry!

Absolutely!

Conceptually very similar to Cursor's auto mode. The key distinctions are:

- We plug into any harness (e.g. Claude Code, Codex, OpenCode, Pi)

- We aren't incentivized to route to our own model, we're incentivized to route to the best model whatever it may be