throwuxiytayq

I find it breathlessly funny how people think it's going to be AIs vs AIs fighting it out with light sabers and plasma lasers in the cyber space or some shit.

Your intuition that this can be balanced out is wrong. When you have a CVE in your TCP stack, your vault unlocks and you're fucked, it's done. My dude, there is never going to be AIs scanning every incoming packet for signs of a 0day, and deciding whether to, uhh, cut it up with a lightsaber and shoot it with a laser pistol. Your options are to have a stack so secure that it can't be targeted, or to disconnect all network cables.

> No, they are not the same, and they require different tools

Yes they are, and no, they don't. Debugging can be done by writing/deleting/modifying code (e.g., write a test and run it, or add log statements, or remove a line and re-run the program, etc.), or by running shell tools (either as single tool calls, which you may to be a one-line program, or with larger compound commands, or entire shell scripts - throwaway or saved to a file), or by writing code that analyzes debugger output and logs, or editing code at runtime with hot reload, or - depending on the runtime - attaching to the running program and running ad-hoc code (REPL-like), and so on. It's literally impossible to enumerate all of the options because it's all mixed in one codey-debuggy-flavored soup. Are you saying you've never seen an agent do this?

It's also fun to write your own debugging tools by the way - just imagine any debugging workflow you think is good, write the tool, give the agent some tips how to use it, and they'll do great. It's so good, you should really try it. This includes debugging tools that are operated by writing code, it totally just works.

> you cannot write code in gdb or valgrind

I'm actually pretty sure these are extensible, and I'd bet quite a lot that an LLM could write that plugin.

> The fact that you personally do both things does not mean everybody and everything does it this way.

I don't, but LLMs certainly do.

> And no, debugging and code writing are not in same datasets. More to it, debugging and architectural planning is mostly represented in fine-tune datasets (prompt-response, agentic step prediction), while code writing is mostly (like 99%) pre-training data (refined internet scraps and regenerated code).

Programming and debugging is self-evidently part of the same training loop. Agents write code, validate it and adjust trajectory. I'm not saying that they don't train agents for debugging-centric tasks specifically, but these certainly often - if not usually - involve programming as well. The same way coding tasks often involve debugging.

> Training on purely LLM-generated data is known to give horrible results.

I don't know what to tell ya, it's how the LLM sausage is made.

> `c = a + b` and `c = a + d` are similar, but the difference is very small — it's about variable rename.

I think you're vastly underestimating the difficulty of real world programming tasks. It's a pretty common mistake, I usually see this in people who are struggling to downplay LLM capabilities, and their strategy is to pretend that everything LLMs are good at was trivially accessible in the training data all along.

> No, LLM does not learn to program same way people do, it generally does not learn things the way humans do.

Obviously, it doesn't. I said "LLMs are capable of learning and extrapolating knowledge to new problems". You can throw a novel problem at an LLM and you'll get a novel solution. This is obvious to anyone who works on interesting problems. You could literally hit CTRL+T and do it right now. It's such an easily and accessibly verifiable fact. What a weird hill to die on, and yet so many corpses there.

> You were talking about code writing, now you switched to debugging. Those are wildly different tasks

No, they're basically the same thing, intrinsically linked and inseparable. Likely represented together in training sets, too. I'm surprised anyone would say that. Are you a programmer?

> For debugging LLM-s reproduce fine-tuning datasets made by humans actually doing debugging on some code.

Not really, they are primarily trained on LLM-generated trajectories from executable training environments.

> And yes, the problems AI found in your code were similar to the problems humans found when writing those datasets.

So they're only similar now? I suppose, in the way that I learned how to solve bugs while learning to program and now I can fix similar ones? I guess we can now agree that LLMs are capable of learning and extrapolating knowledge to new problems?

I've submitted two PRs to fix bugs in open-source system tool last month. Both of these are targeted fixes to highly-specific bugs that existed in this software. The agents I used debugged the issue and implemented fixes and tests autonomously.

These problems looked slightly more difficult than bubble sort to me. Do you suppose these bugs existed in some other software and the agent only translated the fix between repositories? When the agent was debugging the software, were they simply following some tutorial they were trained on? Is this the 1% case? Or are the stochastic parrots regurgitating most probable tokens in approximately the right order to provide an illusion of work being done?

> In case you would say "but LLM-s learn on random source code from the internet"

Of course they don't, it's not 2022 any more.

You are completely discounting all of the original code being produced with LLMs. What percentage of LLM-generated code would you estimate is plagiarism? 1% 10%? 90%? 99%?

> It doesn’t matter who the boss is. It’s the same dynamic.

I'm guessing you haven't had the displeasure of working for a boss who doesn't code and has no idea what he's paying you for - lucky you.

> I was questioning the absurd 10x-100x company output, which is relevant to the op.

I'm talking about programmer productivity, not company output. For pure software companies, these may be closely related, but there's many other processes involved with converting code into money. Either way, the 10x+ productivity increase is obvious to everyone I work with.

Are you under the impression that LLMs take source from the internet and put it on your hard drive? That’s your good faith interpretation of what’s going on?

I don’t know anyone in this situation. No idea how it works in corporations, and I’d rather not find out.

I know some people who pay for their own tokens because it lets them do their job in 30 minutes a day, and spend the rest of their time creating a side project. I know several freelancers who didn’t make software before, and now they have custom productivity tools tailored to their jobs. Others work in small teams, where the boss doesn’t need to ask for proof because they’re burning tokens themselves.

Productivity != sales, especially in software, I think. This has been happening for just a few months, less than one project cycle. I think we’re about to see a huge wave of software products. But it’s possible most of these are competing for a slice of a constant-sized pie, though.

In my bubble, I’m seeing people write entire software stacks from scratch, single-handedly. At this point it’s clear to me that software engineering has taken off by a factor of 10x-100x, at least in some industry areas and in some circles.

But that’s not the point here - even if this is true and it continues, IT is not the entire world economy. I don’t think we’ll see the same productivity gains propagate more widely soon. This will take years. The real world is a ship that’s immensely hard and slow to steer, which is easy to forget when your CI pipeline deploys commits to production in minutes.

I have a general tendency to know things. I think it has something to do with my brain.

But also, I read about how this tech works. Have you? If so, how are you possibly still confused about this?

You’re confused. This isn’t a “maybe there’s more demand for flying cars than you think!” kind of situation. My dude, nobody is disputing the demand for flying cars. It is fairly obvious, however, that they’re completely infeasible technology, and even if not, then this certainly ain’t the way to do it.

I didn’t, because Waymo cars were proof of concept, not proof of wallet size.

It is very impressive engineering, just not very useful or ever going to result in any viable product. It’s a dead end. A waste of somebody’s engineering effort and talent for a one-time rich guy demo item. People hate on it because they get it.

Out of curiosity, what is your sad P(>500 people owning a flying Tesla by 2030)?