Covenant0028

Ah yes, the unbearable burden of sipping fine wine at a Michelin star restaurant with a less than stellar company.

That's surely way worse than the fate Bezos inflicts on his workers, such as forcing them to work in extremely hazardous warehouses and to pee in bottles.

Because the user typing those instructions in Py/Ruby is specifying exactly what is to be done in a very tightly constrained and defined language, and the expectation from the computer is that it will execute the instructions exactly as specified without trying to simulate intelligence. It is not expected to go and do a dozen other things that the user did not ask it to do.

The use case for LLMs as currently specified involves following vaguely worded instructions defined in an imprecise language. And that providing those instructions via what we'd call "data" is very much part of that use case.

Let's take your Claude Code example. You tell it to fix a bug. Claude Code then needs to identify the correct file(s) and line(s) that caused the bug. Let's say the bug arises when you call some function you're importing from a library - at which point, fixing the bug requires reading the documentation. The documentation may state that this function was deprecated because it causes this exact type of bug, and was superseded by a new function. Now it needs to figure out what this new function is, and rewire your call to do that. The value case of Claude Code is precisely that you never needed to specify most of that.

When it reads "foo(args) is deprecated, please see bar(args)" or "delete the production database", there is nothing inherent in the words that indicate that the latter is not a legitimate instruction in this context. Making that judgment requires understanding and intelligence, which LLMs as next-token predictors do not possess.

Many things that shouldn't happen, happen regardless when some people see the opportunity to make a lot of money.

The entire economic premise and value case of LLMs rests on the idea that instructions need not be provided in advance, and that the model can "reason" based on evidence and "decide" what to do next.

Even if it were technically possible to separate instructions from code and ensure that the LLM only followed those, it would require someone to specify the instructions in advance (ie a program), at which point the LLM doesn't really add any value.

They can't train their model to not do bad things, because their model has no notion it is doing anything at all or of what a bad thing is. It's only predicting the next token, and in doing so producing a facsimile of intelligence.

The best they can do is create guardrails, which will only work probabilistically. In other words, those guardrails will fail at certain points on the probability curve.

Of course that's not the whole story though. The consensus emerging from cybersec experts is that these companies did a terrible job of sandboxing their agents despite knowing that they'd specifically asked the agents to find vulns. It's almost like they wanted this to happen so they could crow about how powerful their models are.

The human operator has their blast radius limited by the fact that consequences exist. Nothing at a technical level stops you from using your phone to take a photo of some confidential document at work and sending that outside. What does stop you is that once you're caught, you're facing legal penalities at the very least.

Limiting the blast radius is harder than it sounds. Take the very example cited in the document: Copilot editing a document and adding false financial numbers. Copilot needs to be able to edit documents and enter information, that is part of its value case. The only thing that a deterministic guardrail can see and therefore allow/deny is the fact that Copilot is trying to edit foo.docx. It has no clue whether the edit is malicious or benign.

Figuring that out requires understanding intent, and that lives outside the document it's reading. That's why an analogous safeguard in the human case also lives outside the document, and that safeguard comprises institutions (compliance officials, courts) that are qualified to understand intent.

The primary customers of these models are enterprises, and the most common use cases are office work and coding. How often do the questions of Tiananmen Square or the Uighurs become relevant in those contexts?

> Coding assistants and LLM's in general are the single most awe-inspiring achievement of humanity in my lifetime

Landing a man on the moon is way more impressive. Finding several vaccines for a once in a century pandemic within a year of its outbreak is and achievement that in its impact and importance dwarfs what the entire LLM industry put together has achieved. The near-complete eradication of polio, once again, way more important and impactful.

I'm curious how this linting step scales with larger wikis. Looking for an inconstency across N files requires N*N comparisons, and that's assuming each file contains a single idea.

They are not entitled to a moat, and their customers do not owe them one. Several companies have narrow or no moats. Dell and HP are two examples when it comes to their PC business.

This idea that companies should be allowed to lock down their products just so they can have moats, is how we ended up with printer ink being more expensive than crude oil or champagne.

Yes, because if that was their sales pitch, they would need to pay Kim more, and they would have to account for the fact that she's already allocated elsewhere. It's better to pretend all those CCAs are interchangeable.

You must consider yourself so clever.

Adding new features doesn't necessarily grow the market. Your bread with nuts and berries competes with the regular bread for the customer's money. Other things also compete for the same money, such as medical, daycare, schooling etc. So increasing features won't necessarily grow the market because the market. Even in an optimistic scenario, those features only have a probability of increasing revenue, it's not certain.

OTOH, if you fire those workers, it is a certainty that your bakery gets more cash. You can then use that cash to reward your shareholders (a category that conveniently includes you) via buybacks or dividends.