Anamon

My #1 reason to never, ever do that is that it also gives company admins the power to remotely wipe your personal device. Spending weeks just to set everything back up the way it was, just becauce someone clicked the wrong button (or some "AI" agent was hallucinating a compromise) -- no, thank you.

It may be novel, but I agree with parent that it's mostly in the convex hull of previous knowledge. I like the term remix to describe that! Progress and truly novel things often come from "out of the box", and we know how quickly LLM output quality degrades when it leaves the space spanned by its training data.

So I would also expect a plateau. I guess future progress will depend on if we'll have enough people who still enjoy free thinking and brooding over problems, when there's the tempting option to just ask for an instant result that does the job of the moment.

Why are you so sure about that? Research communities are (used to be?) very tightly-knit with a lot of communication, exchange, and networking. Arguably, the more niche the research area, the more tightly knit and personal.

This is what I hear most about from friends who stayed in academia and research: the pain of losing that. Conferences stopped being interesting, because there's barely an exchange of ideas anymore. Nobody trying out new approaches, everybody just pulling the same lever in an endless loop. Nobody with anything interesting to share, and no potential to learn.

Looking at some of the materials, they seem to have been doing CGI work, too, in later years.

I do mourn the lack of physical special effects, models, and puppets, though. The studios try to convince us that we couldn't tell the difference, but it's pretty obvious.

I already liked Project Hail Mary a lot after having seen it, but my respect for it went through the roof when I saw the role names in the closing credits: model makers, metal workers, puppeteers, animatronics...

I agree, but I'm less optimistic about how far from that scenario we really are. I wonder what we'll look back on as the more damaging effect of the LLM hype: the economic bubble, or the carelessness it caused in people using the systems.

The knowledge about the nature of LLMs, their strengths and weaknesses, what they can appropriately be used for and what they should definitely not be integrated into, seems woefully lacking in many places. Just think back to the recent, disconcerting stories from the military world. We started with lawyers not understanding that an LLM can and will make up fake case law. Now we're at high-ranking military personnel not understanding that LLMs can and will make up fake target identifications.

Maybe the best hope for awareness about this spreading is a gradual slope of worse and worse, but not-quite-catastrophic-yet, highly publicised incidents.

Maybe it's cynical, but I feel similarly and also wonder, what do these people see that I don't? Smart people with supposedly a lot of inside information. But when I hear them talking about LLMs with words like "thinking" or "intelligence", it seems like it either has to be fearmongering for ulterior motives, or otherwise I live in a completely different branch of reality from them.

Their models are still, and based on this technology always will be, token predictors. How is a text generation engine going to directly endanger human lives?

It won't. It only can if someone hooks up tools to them that can perform damaging actions. Someone has to build that toolset. Someone has to give those tools the affordances to take harmful actions. And then someone has to write a "harness" that uses the token predictor as an authoritative input for instructions. Doing so falls somewhere on a spectrum between gross negligence and malevolence.

From a distant enough viewpoint it might not look like a big difference, but I think it crucially shifts the responsibilities. What is the actual difference between driving a potentially dangerous system with inputs from an LLM versus inputs from a fuzzer, or a website with injection vulnerabilities? I think it's the ones building and configuring the tools downstream that are responsible for knowing the risks and build under the assumption of the worst happening.

Heaven knows I'm not one to defend the big LLM companies, but I don't see how this is OpenAI's responsibility. They're building and selling an autocomplete system. How is it their fault if people ask the autocomplete for harmful commands and then feed them into an over-privileged system to run them? (I know the waters are muddied a bit by some labs acting like this themselves right now).

As much as I don't agree with the authors framing of and terminology around LLMs, reading the article generously, maybe this is exactly what he felt, too. That it's not at OpenAI where his concern and insight is needed most, because the token predictor isn't actually the dangerous part. Maybe he'll look where he can bring safety consciousness downstream, where people are hooking up these outputs into systems and in ways that they shouldn't. Don't let the LLM pick your bombing targets. Don't give the LLM sufficient rights to the hospital's file storage to be capable of ransomware-ing it. Don't strap the loaded machine gun to the dog.

My bank does this, too. They also only say that the browser has "outdated components" without bothering to tell you what particular thing offended it. Maybe it's some library they both use that doesn't return the details?

They compared with different "landmarks", and those are usually built around where people are, or people settle around where the landmarks are. I also found it strange that they didn't at least share a hypothesis of their own. But that would be mine -- settlement density and "modern lifestyles" may correllate with some negative health effects, or on the other hand with growing older and therefore being more likely to die of cancer? Maybe they'd have found a negative correllation if they tested for distances to deserts, open plains, and mountaintops.

Aren't you worried about missing stuff?

Maybe I'm farther out of the average than I think, but for me, planning and reminders are one of the few things I want 100% conscious control over at all times. Reminders are for me to make sure I don't forget the important stuff, so hooking in any kind of probabilistic system there would feel antithetical.

We're at the point of one such being mainstream-popular again.

"I get it, the brain is just an LLM!"

> We shouldn't be bullying people delivering trillions of dollars of value to society.

Surely, you mean trillions of dollars of negative value?

Sounds plausible. In Germany, which is on the list, the show runs in syndication on ProSieben. They have their own VOD service and people can probably stream it there.

The people I always loved working with most were the ones that were relentless in their PRs. But I understand it's a culture that's difficult to achieve. For one thing, I think it needs teams that are relatively stable long-term. It can be challenging for people to not see professional criticism like that as personal attacks. Much easier if you know and have a feel for the people you work with, and you go have lunch together after the disagreements in the PR. Everybody benefits, including the customer, but it's a work environment that's increasingly more difficult to maintain.

> Does anyone solder their own microchips at scale anymore?

That comparison doesn't work because up until now, it was only the execution we automated or optimised. Now, we're trying to outsource the understanding itself. No one hand-crafts a microchip, but we know how they're put together. We can debug and improve the process. The suggestion now is to just trust the magical statistics machine and not care that the artifacts become unfixable and unreproducible if and when the magical machine fails.

> If companies want to compete in the world of AI generated code, human code review has to go.

This is a HUGE non-sequitur. It only follows if by "compete" you mean producing more LoC. How often does that translate to market fit or economic success?

This is the mindset that makes me want to leave this industry immediately. Somehow, an industry that already annoyed me with how much "mediocre is good enough" was an acceptable stance, with the emergence of LLM coding tools suddenly decided that "absolute dogshit is good enough" was just as acceptable, as long as everybody else is also fine with eliminating the few quality standards they might have had.