fergal_reid

I think you are misunderstanding my point.

The technology to prevent plague was beyond the population at the time.

"Just use basic hygiene" is useless to a population who has no understanding of what that means.

It was a real virus that had very high mortality.

This makes it a serious counterpoint for people who argue such things can't be created in future.

"The AI designed killer virus using new science turned out to be not that easily transmitted, basic dynamically reprogrammable mrna implants easily prevents it, shame about the 95% who died"

I am not a biology person, but, candidly, this is a bad article, because the author is generalizing from the properties of viruses that tend to arise naturally, and using this to argue about the properties of deliberately designed viruses.

Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).

Even the existence of rare examples should be concerning, if attempting to argue against the possibility.

But even if there were no natural examples, it could just be because the ecosystem is in a balance, which them unlikely. That doesn't preclude someone from designing one.

To use an analogy. Natural predator prey ecosystems tend to be in balance. You could look around the world, before modern humans, and conclude that superpredators aren't a real threat, because predators that eat their prey too rapidly typically then die out. Thus most ecosystems you observe are in some balanced state.

But this doesn't mean modern humans can't come along, using their superior intelligence, and act as a superpredator. Ask the wolf population of western Europe.

If something is stopping us from doing this more often, it's alignment rather than capability.

You can't generalized from the typical behaviour that you see in a balanced natural ecosystem, and use that to say it proves there are no much more dangerous possibilities.

If the author made first principles argument about the inherent difficulty of doing something bad, even with abundant intelligence, I'd be much more reassured; or about defense offense asymmetry favouring defence.

What's here seems very naive.

I have no specific context here, but in case it's not obvious, I think you have to read the exchange in the context of British, and I guess RAF, indirectness and understatement.

I'm guessing the air marshal means "what the hell were you thinking"; Holden is agreeing, indicating that he has learned his lesson and acknowledges the error; and the air marshal is then graciously moving on.

Yes - but this will be the year of Linux on the desktop.

Well, you could conclude the world is now more dangerous for any kid let wander - even if fewer do, and even if the observed average risk to any kid is lower (given they are allowed wander a lot less).

I don't want to be on the 'overprotect kids' side of the argument, but I'm not sure the numbers argur cleanly in one direction or the other.

I also often think of selection bias whenever anyone says "I was allowed do a lot more and we were fine" in the context of child safeguarding; because it also sounds like a lot of kids were abused in the past, who don't speak up in that conversation.

I don't know. I worry I overprotect my kids, but I also am not sure how to price in small risks of massively negative events. I think that's the crux of it for parents - trying to weigh hard tradeoffs.

My story of kids growing up in a post AI world:

There's a gag in Star Trek 4 where Scotty goes back in time, and tries talk to a computer.

The gag is funny because he is from the future where you talk to computers normally. When the computer doesn't respond, someone hands him the mouse, and he tries use it as a microphone.

I watched that scene with my kids recently (9 and 6).

They didn't get the gag. They thought Scotty was completely reasonable to try and talk to the computer.

It took a while to explain.

Yes we're familiar with the terminology and framing of gofai. Fwiw I read (most of) the 3rd edition of Russell and Norvig in my undergrad days.

However, the point we're trying to make here is at a higher level of abstraction.

Basically most demos of agents you see these days don't prioritize reliability. Even a Copilot use case is quite a bit less demanding than a really frustrated user trying to get a refund or locate a missing order.

I'm not sure putting that in the language of pomdps is going to improve things for the reader, rather than just make us look more well read.

But your feedback is noted!

At Intercom we've also a lot of experience here.

I disagree, basically. In our experience actual real world processes are not compactly defined, and don't have sharp edges.

When you actually go to pull them out of a customer they have messy probabilistic edges, where you can sometimes make progress a lot faster, and end up with a much more compact and manageable representation of the process, by leveraging an LLM.

We've a strong opinion this is the future of the space and that purely deterministic workflows will get left behind! I guess we'll see.

You are correct that variable results could be a symptom of a failure to generalise well beyond the training set.

Such failure could happen if the models were overfit, or for other reasons. I don't think 'overfit', which is pretty well defined, is exactly the word you mean to use here.

However, I respectfully disagree with your claim. I think they are generalising well beyond the training dataset (though not as far beyond as say a good programmer would - at least not yet). I further think they are learning semantically.

Can't prove it in a comment except to say that there's simply no way they'd be able to successfully manipulate such large pieces of code, using English language instructions, it they weren't great at generalisation and ok at understanding semantics.

Strongly agree.

This seems to be very hard for people to accept, per the other comments here.

Until recently I was willing to accept an argument that perhaps LLMs had mostly learned the patterns; e.g. to maybe believe 'well there aren't that many really different leetcode questions'.

But with recent models (eg sonnet-3.7-thinking) they are operating well on such large and novel chunks of code that the idea they've seen everything in the training set, or even, like, a close structural match, is becoming ridiculous.

As an Irish person when I saw the article title, I was immediately sceptical.

I personally believe most articles about the famine shy away from the horror of it, and also from a frank discussion.

Going to give some subjective opinion here: people generally downplay the role of the British government and ruling class in it.

Why? One personal theory - growing up in the 80s in Ireland there was a lot of violence in the north. (Most) Irish people who were educated or middle class were worried about basically their kids joining the IRA, and so kind of downplayed the historical beef with the British. That's come through in the culture.

There's also kind of a fight over the historical narrative with the British, maybe including the history establishment, who yes care a lot about historical accuracy, but, also, very subjectively, see the world through a different lens, and often come up through British institutions that view the British empire positively.

It's often easier to say the famine was the blight, rather than political. (They do teach the political angle in schools in Ireland; but I think it's fair to say it's contested or downplayed in the popular understanding, especially in Britain.)

However that article is written by a famous Irish journalist and doesn't shy away from going beyond that.

Perhaps a note of caution - even by Irish standards he'd be left leaning, so would be very politically left by American standards; he's maybe prone to emphasize the angle that the root cause was lassiez-faire economic and political policies. (I'm not saying it wasn't.)

I personally would emphasize more the fact that the government did not care much about the Irish people specifically. The Irish were looked down on as a people; and also viewed as troublesome in the empire.

Some government folks did sympathize, of course, and did try to help.

But I personally do not think the famine would have happened in England, no matter how lassiez-faire the economic policies of the government. A major dimension must be a lack of care for the Irish people, over whom they were governing; and there are instances of people in power being glad to see the Irish being brought low:

"Public works projects achieved little, while Sir Charles Trevelyan, who was in charge of the relief effort, limited government aid on the basis of laissez-faire principles and an evangelical belief that “the judgement of God sent the calamity to teach the Irish a lesson”." per the UK parliament website!

It's not an easy thing to come to terms with even today. I recently recorded a video talking about how fast the build out of rail infrastructure was, in the UK, as an analogy for how fast the AI infra build out could be; and I got a little quesy realizing that during the Irish potato famine the UK was spending double digit GDP percent on rail build out. Far sighted, yes, and powering the industrial revolution, but wow, doing that while mass exporting food from the starving country next door, yikes.

Similar arguments to LeCun.

People are going to keep saying this about autoregressive models, how small errors accumulate and can't be corrected, while we literally watch reasoning models say things like "oh that's not right, let me try a different approach".

To me, this is like people saying "well NAND gates clearly can't sort things so I don't see how a computer could".

Large transformers can clearly learn very complex behavior, and the limits of that are not obvious from their low level building blocks or training paradigms.