I just ask our fleet maintenance agent to configure the variants of the new model and then run the bakeoff. In a few hours, I get a doc containing exactly how it performed on my real tasks, against the other models we've benchmarked.
We don't do any "prompt engineering" to make specific models work better or worse, so this gives us a direct comparison that lets us objectively compare how they'll work in practice for different types of tasks.
Good enough to have guided multiple model swaps and measured increases in productivity.
We don't do any "prompt engineering" to make specific models work better or worse, so this gives us a direct comparison that lets us objectively compare how they'll work in practice for different types of tasks.
Good enough to have guided multiple model swaps and measured increases in productivity.