hannasanarion

Probably not a clear divide, but it might be some kind of divide, and it might be the opposite of intuition.

The big advantage the LLMs have is their huge context windows, ability to parallelize and delegate, and no attention fatigue or need to sleep. They can think about random combinations of functions and modules all day long until they find a combination that's exploitable. That's hard for a person to do, and its the kind of exploit that probably would never be found otherwise except by another llm-powered thing. Not being a security expert myself, I gather that these AI vulns tend to be small needles in very large haystacks that normal people just would never have the patience to search.

But if you found something yourself, then it's something that's simple enough and involves a small enough number of moving parts that you can keep it in your head all at once and recognize the problem and correctly reason about its consequneces, which probably means it's easier to exploit, and therefore more dangerous if made public.

It feels a little backwards, and it might still be backwards, but if there's any benefit i think it's along those lines.

That "if" is load bearing and increasingly flat-out false.

Crowdstrike's average "breakout times", the time gap between a first successful infiltration and maximal exploitation, has gone from days or hours prior to 2024 to minutes or seconds now.

There are companies today that still keep Internet Explorer on their terminals because some of their intranet sites still need it in year of our lord 2026, you really think we're going to have 100% turnover of user interfaces within the next year just because AI makes the programming quicker?

Is principle the only valid reason to want to do something? Why is it unacceptable that people be motivated to do things because they want the results?

The actual physical reality of the status quo of reverse engineering and recompiling is that the vast majority of stuff never gets the treatment because they aren't lucky enough to catch the attention of one of the few magical individuals with extreme skill and dedication willing to spend worth hundreds or thousands of hours of their time on them. The consequence is rot.

The status quo is thousands of pieces of software, including but not limited to video games, are lost every year, with no feasible way to recreate the systems and conditions that are needed to run them, and no way to update and repair them to be able to run again.

An estimated 88% of PS2 games are already lost forever. There are thousands of people in the world right now who have in their possession the last copies of digital media with no way to play them and nowhere near enough skill to crack and preserve them themselves.

The alternative to REA isn't hand-made free-range organic recomps, it's nothing.

Is there any writing on it that was written by a human? This blog post is clearly AI.

It's somewhat concerning to me that none of the security conscious people in this thread seem to notice that they are changing their privacy practices based on the advice of a language model pretending to be a person.

It seems silly but I think there's a good use case for this:

Helping people deal with bad UX.

Everybody who has worked in a traditional corporate environment has encountered poorly tutorialized, poorly labeled, counterintuitive user interfaces where the only way to know how to get something done is to have somebody sit over your shoulder and tell you which buttons to click.

Oh, you want to figure out which OIM entitlements your coworker has that you don't to debug an access issue? You just need to know that you have to click the "compliance" button to get there. Why is it called "compliance" when what it actually does is let you export comparisons from the user tables? Because fuck you that's why. Memorize it.

An AI assistant that can read the procedures for you, or sometimes even read the source code, or the website code, can show you how to accomplish something that the UI on its own fails to do.

The fact that the utility doesn't give it the ability to do that?

It makes sense to not trust AI models with the ability to read and alter your screen for many many many reasons, but the developer of the tool knows that as well. A tool that lets your ai do something doesn't also let it do anything.

So, as always, it's a question of if you trust the software producer. You're giving them the ability to draw on your screen too. Would you have the same fear if there were no AI involved?

For stuff like this I imagine AI misbehavior as a novel failure trigger not a novel failure type. It can only do the harm that the software was able to do on its own already.

Smart bulbs.

You can make your artificial sunrise take as long as you want, and connect them to a sleep tracker so they're timed to wake you up during a comfortable phase.

I've been doing this for 12 years, and I don't think I have ever been upset to hear my alarm since, as long as I'm home and the lights have prepared me. Lately I've even been waking up before my alarm about half the time, even though I work on a schedule one time zone east of where I am.

It's shocking to me that it's not more popular for everybody. I guess the benefit is hard to explain.

I think this is a welcome overcorrection though. Any good businessman will tell you they'd rather be backed by an insufferable nerd than a yes-man.

Maybe it's just me.

For like, 90% of conversations, I don't want it to let technical inaccuracies and rhetorical flourishes slide. I want it to tell me that the point I'm making is technically wrong because an expert would recognize subtle misuse of terminology, or because there's an exception or edge case that I didn't proactively insert as a caveat, so that it is my decision to ignore that advice and be a little wrong on purpose to suit my writing goals.

What I don't want is for the AI to assume my writing goals, and be incorrect because it believes that is what I want. I want it to "well ackshually" me so I can say "shut up, nerd".

Like, there's another comment in this thread that I ran by claude to check my understanding about today's post-training methods and how they avoid sycophancy, and claude responded by splitting a bunch hairs over like, "well, technically this is still RLHF, its just that there's other feedback signals mixed in, and the preference is detected in other ways, and ai judges are involved as a filter for examples, this and that and blah blah blah". Shut up, Nerd. In the context of this conversation, RLHF is already being used as synecdoche for user preference feedback, readers understand that, and even if they don't, their misunderstanding is completely harmless. I will not be taking all the wind out of the sails of the point I'm trying to make inserting your three paragraphs of irrelevant clarification in the name of technical correctness, thank you very much.

As long as receiving nitpicks and technical minutiae implies 1. there are no larger structural problems and 2. the model isn't rolling over to please me with sycophancy, I figure this is ideal.

That's a great point about the emotional side of "not being convinced by a person, but an explanation machine" making fact-based rhetoric more effective. It feels more neutral.

I think there's a second effect that's a selection bias for the experiment itself and probably cuts against the supposed dangers of this result: a chatbot can tell you things only after you have asked it a question.

There is one thing that every single study participant (including the uninformed reddit users) have in common: they all knew that the person or entity was trying to convince them, and they read the words and thought about them enough to craft a reply.

This means that all of the people here were available to be convinced and self-identified as such.

Being open minded is work. You have to doubt your own beliefs, you have to disregard evidence that you previously found convincing, you have to crank up the empathy to put yourself in another's shoes, you have to listen intently to understand what is being told to you. Nobody is naturally doing that all the time, it's a state of mind that you have to intentionally activate, and it can't really be forced onto you.

All of the participants chose to engage in a conversation that was designed to convince them, which means they had all already accepted the possiblity that they are wrong a valid outcome. That's not really a state of mind you can trigger with a TV ad.

So, the warning that this could be weaponized isn't convincing to me.

If I found myself in a conversation with a person or chatbot who was clearly trying to convince me to flip a strongly held belief, like a major axis political affiliation, I would simply walk away because that's not a conversation I am willing to participate in.

So I don't think this is really weaponizable. Which actually means it's probably a good finding for society.

The study found that the models could convince anyone of anything, as long as it was allowed to cite facts. It could convince people of false things, but it needed to invent false facts in order to do so.

So as long as we keep training AI models to value facts and quality research (skills and values that are essential to be able to sell them as agentic workers), their influence on the opinions of society will tend to pull people away from beliefs that are unsupportable by facts. In the moments when those people are willing to accept a change in opinion, and they talk to a chatbot with doubts in their mind, even chatbots with no morals like Grok will tend to pull them away from conspiracy theories and similar ideologies and towards beliefs that are grounded in reality.

Could be! I don't recall turning things like those on, I use firefox for features not privacy, and mostly leave stuff like that at their defaults, which I trust Mozilla + uBlock to filter the invasive trackers while leaving actually useful functionality, but it's possible I fiddled with it years ago and forgot.

Some probably kinda strong counterevidence to that though is that this doesn't happen on all sites, only Amazon. I've never been prompted to make a new passkey after passkey login on the 12 other sites that I have passkeys in bitwarden for.

This currently is my team's only AI-use policy and I thik it's working out fantastically.

"No AI PR Descriptions" is a great rule because it does 3 things:

1. It's a hard binary that's easy to recognize and enforce

2. It establishes personal ownership for the submitter, making it psychologically difficult to submit something you don't understand. If a person has to describe what a change does, how, and why, then they must have looked at it and made a real attempt to understand it, because you can't map a territory you have never seen. This alone prevents "claude-code run amok" scenarios where devs have completely abdicated responsibility that are a real pain to clean up.

3. It informs the reviewer of the change's intent. A human PR tells reviewers what a thing is intended to do, as well as the authors belief about what it does, which they can use as a framework to critique the actual substance. It becomes easier to notice missed edge cases, code behavior that subtly differs from intent, or changes that would block off or complicate an intended future project direction or reduce maintainability.

And as a bonus, at time of writing, AI pr descriptions basically always suck. They are usually full of irrelevant implementation details, misplaced emphasis, weird presuppositions that seem to come out of nowhere and are really annoying to read, etc. That won't be true forever, they are always getting better, but it is true now, so that's a temporary 4th benefit.

Who said interpretability wasn't necessary?

Manuals, laws, and dictionaries are not designed to be read, but they are designed to be interpreted. In fact interpretability is what they maximize for. And interpretability isn't an absolute value, it is relative to an audience, this is why household equipment often comes with separate owners, installers, and service manuals, and why Webster publishes Elementary, Collegiate, International, and Mass-Market versions of its Dictionary.

Documents that are "meant for human consumption" in the way the top comment intended generally don't optimize for interpretability like code, manuals, and dictionaries do. Novels, letters, emails, internet comments, blogs, articles, nonfiction books, etc often forgo interpretaiblity for practical, expressive, or rhetorical effect. The best moments of these is even when they are confusing on purpose, because imprecise language is a gateway to deeper levels of meaning through fourth-wall breaks, metaphor, analogy, and humor. (all things that LLMs are notoriously bad at and don't seem to be improving on over the years)

Those other types of writing are intended to be read in full, every word, end-to-end, in order to educate, entertain, inform, or influence the reader who, and i think this is the critical part, does not drive the experience.

When I'm reading a news article, and you realize that you're reading the voice of chatgpt, I'm annoyed because it means that i've been strung along by a stupid machine that somebody else handed me off to to keep me occupied.

When I'm reading code, and I notice that it seems AI written for whatever reason, I might then choose to apply more scrutiny, maybe I try to suss out what the prompter intended to happen, so that I can validate whether that actually happened and if it was accomplished in an optimal way. But it's not insulting in the same way, because I'm still in the driver's seat of the experience, deciding on my own which parts to read and which parts to ignore as irrelevant to my investigation, because the only thing I want to get out of it an understanding of a part of the mechanical system that the code creates.