Bootvis
Joined 16 years ago
Joined 16 years ago
> It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
We can agree that the pgrust project messed up and should have been more careful but your blanket statement lacks support.
In hindsight, this shouldn’t be surprising. The prompt requests to make a certain operation faster, the LLM will try to find a way and isn’t even informed on the wider context so it myopically chases the goal.
Even at higher levels, I think there could be use. The Dvoretsky example on my GitHub is real but it’s not a guarantee you get it and it could just be reiterating from training. What didn’t work for me was forcing it to look at lines in detail, but maybe with larger context and better engineering, you can get the LLM to extract themes from the search tree. With compare and contrast, it could maybe figure out why a move works in one variant and not in another and produce a coherent explanation of this nuance.
I’m not trying this yet because it would take a lot of tokens and I don’t have the time.
It integrates an LLM of your choice with StockFish to analyze games. Back then, the results weren't very good and I abandoned the project but a lot has changed (the LLM's got better and I did too) so maybe it is time to take another stab at it.
You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
Project: ibexlang.org // Fast data frames
Blog: bobjansen.net
Moderator at http://quant.stackexchange.com