juliusjh

Yes, maybe what is hard for LLMs and what is hard for humans have a very small overlap. But calling it brute force or low-hanging fruits is still a bit absurd.

You can safely argue that it's statistics, because by construction it is. But then, maybe nothing is anything more than statistics.

Call these problems "low hanging fruits" may be the understatement of the century.

Not the one you are replying to, but an argument I heard is that

1) It is now much easier to work in a language with better security features such as Rust. Previously, there definitely was some kind of trade-off between accessibility, performance, and security, and some people argue that this is over. I don't know if this is true, but it seems plausible.

2) You can now translate your legacy projects to a different language much more easily. This is definitely true, but that doesn't mean it doesn't introduce new/other problems or that it is easy enough (again, I simply don't know).

True, it's a theory; basically a hypothesis derived from my experience with LLMs and their performance in CTFs/finding and exploiting vulnerabilities. Plausible, I believe, but I don't have any hard evidence of it happening.

Maybe the effort and required hardware of running an LLM locally have so far prevented widespread exploitation? Maybe open-weight LLMs are too far behind at the moment? Maybe it's happening but we didn't notice it for some reason?

I would strongly assume that it will happen at some point if it isn't already.

> I'm much less likely to get hacked in this new era of AI assistance, as I'm only a few days behind, rathe than a few months or years behind.

Yes, but also, attackers are much faster now and complex chains of vulnerabilities became much easier to carry out. In the past, you might have just been "secure" because nobody cared to put in the time to target you unless you were unlucky. Now, many people have the capability and can completely automatize it.

We essentially had security through obscurity for a long time. It was always dangerous. It's good that it ends. But the transition might be rough.

I would assume that in many (smaller, non-professional) systems/code bases, it is also possible to poison the backups and code base in general. This would have been a major effort in the pre-LLM age, but it should be pretty easy now. As soon as such instructions are out there, people (or rather LLMs) are going to find ways to make the threat persist.

I've been mostly reading here but created an account to disagree with this statement: The smartest people I know were always amazing at explaining. This was true for me as undergrad and graduate student where the smartest peers and the most renowned professor were always also the best at explaining, and it is true now at research level in a related field. When I am not sure if I really understood something to the core, I try to find a colleague who knows very little about it; if they understand my explanation well, that's a good sign.

Those who really understand a topic are usually also able (and great at) explaining it in very clear and "simple" terms. This may be part of my personal bias; I see theory builders as those who advance the field the most, and these are usually also amazing at explaining it. On the other hand, those who mostly "grind" through problems (approach them as complicated puzzles) with effort/time were often bad at explaining.

I observed the same for programming: the "architects" usually explain very well, the "debuggers" often don't. LLMs very much remind of the grind/puzzle approach. It does makes sense that RLVR, which in my understanding enables a lot of these results, would lead to a more mechanical approach.

Of course, I can't make any predictions on whether it will stay that way. But I strongly suspect that we need different ways of training for LLMs to write better text and explain better (I suspect the vagueness of LLM language is the result of RLHF as vague expression is less often incorrect).