However at this point - benchmark success is about as effective as results from someone who has been “taught the test”
If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place.
The best analogy I have right now is that improved results on benchmarks are like better acting from Hugh Laurie as House.
If you want to watch a show - great (generative work)
If you want to get a prescription - then not so much.
However at this point - benchmark success is about as effective as results from someone who has been “taught the test”
If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place.
The best analogy I have right now is that improved results on benchmarks are like better acting from Hugh Laurie as House.
If you want to watch a show - great (generative work)
If you want to get a prescription - then not so much.