chbint

Claude Code was used to do the reverse engineering.

Not hard to see how an AI agent could achieve something similar even as a step towards some innocently established goal.

Poor security + hacker-like capacities for anyone using an AI agent.

What could go wrong?

I suspect his diagnostic is pretty accurate, though. The bitter lesson came up when deep learning was already mainstream. The text discusses how that happened, and it can be the case that convenience beats accuracy. Accuracy is an epistemic value, but current AI is largely driven by market values. If accuracy manages to get along, great, but other than that, market-laden convenience reigns. Commercially, it is often more convenient to even change the world in order to make it easier for our models (consider how we're willing to create special places without pedestrians or human-driven vehicles for autonomous vehicles as a "solution" for their shortcomings).

I like RSS and I use it, but this sounds like wishful thinking. Even the amount of human produced content is just too big for one to be their own curator. We have those few authors or sites we keep up, but other than that we must rely on external help, such as HN or an agent.

Their argument is not sound, but it is informative paying attention to what they consider "evidence" for AGI. A nice instance of a problem that seems peculiar to AI: it tries to define both its target phenomenon and how well it is doing towards it.

I confess it was disappointing for me. Their main claim seems to be that thinking comprises pattern matching and pattern completion--allowing them to say that LLMs do resemble something we humans do-- but that's essentially the idea behind the connectionist movement from the 1980's - the one out of which current DNN models came from. Perhaps a friend of 1960's symbolic AI would be unhappy with that claim, but there are not many of these around anymore (Gary Marcus is misrepresented as one such, but his view is that models should be hybrid, not purely symbolic).

Nowadays, the question about whether LLMs are "actually" doing something similar to human thinking revolves around other dimensions, such as whether they rely on emergent world-models or not. Whether such world models would require symbolic reasoning or not is a different matter.

This has interesting implications for representations within AI models.