gsandahl

i guess a model trained on this game will end up taking the #1 at some point

haha thats one way to look at it :)

very cool :)

Agree, this is where llms can uncover new perspectives!

Most of the tasks have assessed with ground truth, occasionally helped with an LLM as a judge to assess the answer if the answer is a sentence and not an exact result.

Example: Given a long travel journal How many cities does the author mention? GPT-5: 12 Expected: 17

Please do and give us some feedback!

I think just how far you can go with examples has been an interesting learning! As these models have become smarter, they are also getting better at reasoning from examples and understanding intent. We will be publishing some research in the next few days!

No up to date demo video unfortunately :(

Sounds like a great use case though!

We have been thinking a bit about this, and one option would be to have some form of locally hosted runner. You can optimize the task in the cloud and deploy it locally. Something like that. It is possible to plug in custom models so technically feasible.

Thanks for the shout out!

Co-Founder here thanks for taking a look at Opper! I’m hanging around the thread all day, so feel free to ask anything, share feedback, or tell us where you’d like the product to go next