TypeSafe's new model, Jev, doesn't write text. You give it a situation and a question, and one quick call returns a probability for each answer.
The test: 770 posts from Reddit's r/AmItheAsshole. Each model had to predict the verdict Reddit actually gave.
Jev came second of seven setups. Sonnet 5 was a bit more accurate, though the lead is borderline once you account for how many comparisons were made.
Jev's median call was 6.3× faster than Sonnet's. Fast, but not 40–200×. That was measured on one laptop in one evening, so treat the exact number with some caution.
I have no affiliation with TypeSafe. Code, logs and method are open, so you can rerun it yourself.
The test: 770 posts from Reddit's r/AmItheAsshole. Each model had to predict the verdict Reddit actually gave.
Jev came second of seven setups. Sonnet 5 was a bit more accurate, though the lead is borderline once you account for how many comparisons were made.
Jev's median call was 6.3× faster than Sonnet's. Fast, but not 40–200×. That was measured on one laptop in one evening, so treat the exact number with some caution.
I have no affiliation with TypeSafe. Code, logs and method are open, so you can rerun it yourself.