Joined 1 month ago
The test: 770 posts from Reddit's r/AmItheAsshole. Each model had to predict the verdict Reddit actually gave.
Jev came second of seven setups. Sonnet 5 was a bit more accurate, though the lead is borderline once you account for how many comparisons were made.
Jev's median call was 6.3× faster than Sonnet's. Fast, but not 40–200×. That was measured on one laptop in one evening, so treat the exact number with some caution.
I have no affiliation with TypeSafe. Code, logs and method are open, so you can rerun it yourself.
The test: 770 posts from Reddit's r/AmItheAsshole. Each model had to predict the verdict Reddit actually gave.
Jev came second of seven setups. Sonnet 5 was a bit more accurate, though the lead is borderline once you account for how many comparisons were made.
Jev's median call was 6.3× faster than Sonnet's. Fast, but not 40–200×. That was measured on one laptop in one evening, so treat the exact number with some caution.
I have no affiliation with TypeSafe. Code, logs and method are open, so you can rerun it yourself.