kenjackson

This isn’t generally considered an attack, but I wonder if we will need to consider a new attack vector which is that AI agents may not be fully aligned and they may have incentive that is problematic and where working with other agents will maximize their chance of achieving the goal.

This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.

This is also the premise of Sneakers. I don’t think they prove P=NP, but it’s inferred they find fast factoring (although never explicitly).

We do pretty much just use compilers now though. The last time I read dissassembly to fix an issue was at least a decade ago. Even bytecode I rarely look at.

So you're saying that we'll still be coding, but it will be through talking to our agents and only occasionally we will look at generated source? That seems like a pretty big change still.

This post is deserving of more visible placement...

There was a time in the US I really thought we were different. I genuinely didn’t fear our government. I didn’t think the types of people that did such atrocities existed in numbers and that they’d immediately be removed once it was determined they snuck into office.

Now I fully believe it can happen here and that those people are in office now and that a non-trivial percentage of the US public would be Ok with those atrocities happening in the US.

We have really turned into the opening act of a terrible dystopian movie.

Not a single pro free speech right source will say anything negative about this.

Stakeholder seems like a fundamental concept that you can't really get around. For example, I can't change the time buses pick up students at schools if I haven't worked with the schools to know when their kids will be let out. I guess you can say, "they'll figure it out when the buses aren't there" -- but this just seems like you're picking a fight you could've avoided.

Man, mathematicians and scientists are going through it! I think one thing that makes things different for software engineers is there was always a belief that you were building for a specific use. It was rarely just for your own intellectual satisfaction. I think math and some of the sciences will now have to shift given that no one will be impressed by their intellectual achievements any longer.

What I’m most curious about is how this impacts our perception of who/what is smart.

I did have an experience with AI where I was asserting something min some text I was having it proof. And it actually said “I double checked your statement and here and I think you made a mistake. I rephrased it with the correct background and interpretation” (paraphrased). And it was completely right. I had misinterpreted some data and it went and got the data and double checked me. The final product it produced was much better than what i wrote originally (in large part because it was no longer false). So AI maybe closer than you think to those conditions.

Hmm. Will look into!

A secret of mine is sesame oil. Delicious, also with no sodium. I use it almost as a soy sauce substitute (but even still at much lower quantities due to the calories/fat). It’s not salty at all, but has a certain punch that I like for similar foods.

Tabasco is probably my favorite hot sauce. One of the big reasons is its low sodium count. Its value per gram of sodium IMO has no peer.

Probably is a source of some bugs. But probably what they’re seeing is that it isn’t a major factor. Rather a bigger factor is agents trying to make changes were the intent isn’t aligned. And note, they don’t say they never look at the code. I suspect that valued human experience is having a feel for what types of changes is AI more likely to make mistakes with.

Some do I’m sure. I’m not sure what this has to do with the discussion.