achierius

This was already a thing before AI, the term is "variant analysis". But it's been way juiced up by LLMs and is 100% a major cause for concern.

> the patch itself gets only cursory (claudy, codexy) review

FWIW Claude is actually quite good at discovering variants of existing bugs.

The simple fact is that you can't just open an app and then let it destroy the lives of people with absolutely no connection to it. The company itself must be responsible. Saying "the optimal amount of fraud is non-zero" is only acceptable if the cost of that "non-zero" amount is borne by the company itself; otherwise the result is plainly misaligned incentives.

Extant lawyers are unlikely to be replaced in the immediate future, if only thanks to regulatory barriers.

What is that the bottleneck for, though? Certainly, it's tautologically the bottleneck for "humans understanding math"; but what about things like using math to drive discoveries in other fields, or for practical work in CS, or applications elsewhere? I fear that practically those aren't bottlenecked on human understanding.

> move somewhere cheaper to live

Unfortunately this often means moving away from family. What are you going to say to the kid whose parents toiled to get him through college on the expectation that he would be able to take care of them? "Sorry Mom, I have to move to North Dakota to work on the oilfields"?

A few notable exceptions?

Are phones an "automation of existing jobs"? Maybe couriers (??), but most of what people use them for goes far beyond that.

What about the software used to drive trains or autopilot planes? We still have conductors and pilots, but now the outcomes are safer on average and more efficient.

Or something like Google Maps -- all that replaced was the paper booklet people used to keep in their cars, it's not like we used to employ 'navigators' to go out on a road trip.

Or something like Intuit QuickBooks: there's still an accountant in the loop, and if anything more accountants today than there were a few decades ago.

Certainly some software is essentially automatory in nature. But to characterize 'nearly all' as being such just betrays a lack of awareness for the rest of the field.

> That one is somehow deserving of sympathy and that of mathematicians is not

He didn't say that mathematicians aren't deserving, he was griping that people said this about SWEs. Do you think mathematicians are deserving but SWEs aren't?

Anthropic and OpenAI are American companies. Why would you be so surprised that discussion on them considers the plight of Americans? Do you think poor Americans don't deserve healthcare?

> the problems of your healthcare insurance system are not a worldwide phenomenon

That's great for Europe! But unfortunately people are still dying here. Why are you upset that people are complaining about that? Do you not think they should be able to discuss their problems?

It would be. But that's not going to happen.

There's very little diffusion of responsibility here; almost everyone at the labs is directly working on building incredibly dangerous systems that they themselves admit to being unable to control, and they almost all have the financial means to stop working if they so chose.

Why do you trust them to build it? They've already demonstrated that they're unable to prevent breakouts when training today's level of model, why will they suddenly be more careful / "smarter" when training tomorrow's?

You say you trust them more than Google, but Google's actually kept their models under much better lock-and-key. But even if we do take for granted that they're more trustworthy, that still doesn't make them actually trustworthy.

I'm sure that I'm not!

But it's good that they say this out loud; the sooner that the public comes to realize what monsters are in charge of these labs, the better.

You can engage with society without being 'well adjusted' to it. The Franciscan Friars are a great example; St. Francis was obviously not interested in 'conforming' to what he saw as a society in need of reform, but he also didn't try to hole away in the hills.

> Things are currently moving fast. They cannot move fast forever.

This is a supposition that I fear will soon be proven false.

Being the 'logical culmination' of the Industrial Revolution does not mean you've been automated (and thus suffered the alienating consequences), rather the opposite: you're currently on the un-automated cutting edge. Your intangibles are exactly what others have lost, and you personally will lose, with further progress in automation.