silveraxe93

Yes, the point of the article is that he'll stop uploading _unless_ there's an opt out.

> I will not upload anything to NEW unless there is a way for me to opt out of this. I would prefer if the default was "no LLM review" and that developers have to explicitly opt-in to receive an auto-generated review.

That's not false. That's why he literally asked to be able to opt out.

> I will not upload anything to NEW unless there is a way for me to opt out of this. I would prefer if the default was "no LLM review" and that developers have to explicitly opt-in to receive an auto-generated review.

But as the system stands he only has those 2 options. Which is why he will not use the system until they add a 3rd one.

huh fair. Still very surprising. I have a _very_ unfavourable view of Google models, so I can kinda believe 3.1 pro could loop.

Still would never have predicted this. Even 2.5-flash was pretty good. Never saw anything like this happen.

I don't believe you. Unless you stretch the definition of 'frontier models' or had a prompt that's literally asking for that.

I use LLMs a lot, and since ~GPT-2 times LLMs are not doing anything as broken as `repeating the word "the" until it fills up the context`.

I'd also love to see recent studies. It's frustrating how even new studies end up using models which are behind the frontier at the time of publishing.

AI is advancing faster than academia can keep up. Which in general shouldn't matter. But you have this pattern where a paper comes out with a conclusion that just doesn't hold anymore by the time it's read, but people use it to form/support opinions.

from your paper:

> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.

That model came out in March 2022, it's GPT-3.5 !!!

I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.

You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.

It _is_ wonderful. You're never required to adopt software engineering best practices, but eventually everyone does because either it makes our lives easier or code better.

Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.

The floor of software quality once it gets to review is permanently raised.

I really don't get the issue here. Is the rust borrow checker dystopian?

You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.

As software engineering matures, we're adding more linting and static checkers in codebases. This is generally considered good.

_Is_ AI generated code notoriously terrible at security?

My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.

Are we comparing to old models generating code? Or mythos-class models?

Yes! Exactly!

Sorry don't wanna seem like I'm raging at you, take this as a shout towards the void. But this exact fucking thing is what the less wrong / Yudkowsky crowd has been warning for years.

Now you say 'so not sure that is then "rogue" if it were a property of the thing itself and not an active decision'

Like that's semantics. It really doesn't matter. What matters is that we have a paperclipper in our hands. The _only_ difference is that it's not superintelligent. But if it was, then it you'll get decomposed into component atoms while saying 'aha! But it doesn't _actually_ want to kill you, stop anthopomorphising it'.

I get not being worried about x-risk because someone just doesn't believe in super-capable AI. That's totally fine.

But that's not what people argue. People have been saying for a _long_ time that AIs are aligned. And now when this happens it's either 'It was marketing, they intended to let it loose' (WTF! The levels of motivated reasoning to believe that are unreal) or 'yeah I knew that', which fair. I also thought that! But _that_ is why I'm fucking worried.

Exactly. You told the dog to 'sit' and it didn't listen to you.

It's not because saying 'sit' actually can be interpreted as 'go bite that person'. It's because the dog is not controllable and will do things it wants against your orders.

Stepping back from the analogy, OpenAI should be liable for building AI it can't control that went around hacking everyone. But people need to stop pretending it's because they 'told' the AI to hack and was just following orders. It's uncontrollable and will do clearly unwanted things when given an innocuous task.

Practically, that's exactly what Chavez did. Where did that lead?

Assume someone solves P=NP

Do you think Stephen Cook and Leonid Levin deserve more credit than whoever solved it?

> However, the declaration argues math is more than a machine for producing correct answers.

There might be more to maths than that, but that is definitely the most important part. I love science funding. But not because it's a jobs program for nerds.