The open source ones- I downloaded a number and tried them and compared to Jev.
Anything that required knowledge / familiarity mmBERT and ModernBERT post-trains performed much worse.
So it seems like they did some kind of useful expansive pre-training.
Things that were Qwen or Gemma Diffusion did better at those kinds of tasks but were generally pretty inconsistent in terms of whether they could succeed repeatedly (and be stable + reliable) on the many types of tasks that are in the cookbook part of the Jev docs.
If you ask Jev similar input + questions, it's pretty stable. And does a reasonable job on a lot of questions.
This one public benchmark (the only I've seen) seems to give the open versions way too much credit. It wasn't my experience at all.
It gave my a false wrong sense of what might be required to get it working for something at work to avoid needing a new subprocessor as - at least on Cloudflare / OpenRouter Jev is third-party not hosted.
Anything that required knowledge / familiarity mmBERT and ModernBERT post-trains performed much worse.
So it seems like they did some kind of useful expansive pre-training.
Things that were Qwen or Gemma Diffusion did better at those kinds of tasks but were generally pretty inconsistent in terms of whether they could succeed repeatedly (and be stable + reliable) on the many types of tasks that are in the cookbook part of the Jev docs.
If you ask Jev similar input + questions, it's pretty stable. And does a reasonable job on a lot of questions.
This one public benchmark (the only I've seen) seems to give the open versions way too much credit. It wasn't my experience at all.
It gave my a false wrong sense of what might be required to get it working for something at work to avoid needing a new subprocessor as - at least on Cloudflare / OpenRouter Jev is third-party not hosted.