throwaway13337

Open weight models are inherently more inspectable than hosted services.

Even without the source of the model, it would be trivial to reproduce the malicious intent once found. You have the weights - you can find the prompt to produce the bad behavior. So where is that proof?

I'm much more concerned with both the private data I send and tokens executed on my computer doing god knows what from cloud hosted providers. Especially those that don't even show the reasoning traces. So the model's true intent can be hidden even while it's openly thinking of acting against you.

This is scare mongering that implicitly suggests something much less secure.

Both can be true.

This is a market for lemons.

As a market's trust lowers, buyers are less willing to pay a reasonable price for a square deal because of the probability being higher that it isn't. This, paradoxically, lowers the value of legitimate products from legitimate sellers because there's no way to differentiate. The risk adjusted value of the product lowers. Legitimate sellers leave. Things spiral. Everyone loses.

Trust is the real rare commodity.

I think I could have been clearer.

The PE firm creates market demand for the goods/services that the purchased company used to provide at a better value to the customer.

So yes, the 'creation' is a demand which is sort of a destruction of the value that the customers previously had. In a fair market, this demand can be met. But a PE buys strategically such that this demand is not possible to satisfy because the company they purchased is entrenched in some way (regulation/monopoly).

True. Though I don't like the term 'natural monopoly'. Most natural monopolies aren't - or don't have to be.

Maybe a term like 'natural markets' captures it better? The property being that natural markets/monopolies provide some sort of substrate on which a market can exist.

Good regulations seem to be ones that force open protocol and interoperability of these platforms that get large. This creates a new marketplace abstraction layer that enables new innovation to thrive.

I'm certainly glad that I'm not on AOL's internet. And also glad that internet exists in part due to Bell's telephone system being forced open.

Bad regulations do not seem to have that characteristic. It's too bad we do not have vocabulary to tell them apart. Public good type regulations are more muddy and can be used as a weapon more often than not.

These PE-owned companies create market demand by being shitty. Somehow they are able to keep their margins and their market so that their business model makes sense. That's a puzzle, right?

There is no shortage of investment looking for great returns. A market with huge demand not being met adequately is a dream to investors. Even more when you know the competition must continue to fuck their customers because they paid above market rates for the purchase and the business is saddled with debt obligations it must meet (Leveraged buyouts do that).

What could stop new competition from beating them out?

It's not capital.

I think that's true to an extent. And, especially successful people, often would rather be their own boss for less money than to be a corporate cog. Having the freedom to make ethical judgements yourself and build local community is a real luxury. We are, mostly, just social animals that want to build our status by helping each other out in the end.

Two things, I think, might make the decision for them, in spite of that:

  - The money is actually more than their business could be projected to make alone. This is because the PE firm isn't just buying the single practice but every practice in the area to kill competition. Get that captive market and you can charge more.

  - The personal ownership freedom can be already lost on these independent people if they feel they are already tied down too much. I know this is especially the case in small medical offices where insurance and regulation can over-rule them. Feeling like you don't have control in this way can cause a lot of depression (tangent but look up how we condition mice in order to test treatments of antidepressants on them). At that point, the monied exit starts looking like a better option.

That's how cynicism can win out. And we all lose.

There's a pattern to the kinds of companies PE buys and I think it points to the real problem.

They like companies with some kind of moat that makes it hard to unseat them. Basically, companies where there is no alternative for the consumer. That way, they can inflict abuse but know there will be nowhere to run.

There are two different ways to achieve this. Monopoly and regulation. Hospitals have both government granted locational monopoly and tons of regulations that make it impossible to compete.

Private equity is the symptom, not the disease.

Until we get at the disease, new monsters will be born with different name filling the same ecological niche. It's economic natural selection played out in the environment we created.

I think we're in the 'early digital camera' phase of AI art gen.

The real photographers knew they were shit photos. They were grainy, bad lenses, and ugly. A lot of people still liked them because they were much more convenient for a lot of uses than the film ones.

Well, I guess there's still a lot of film people. But film had a ceiling. Digital cameras kept getting better.

The adaptive film photographers eventually adopted the tool. A lot of their skills transferred well to the new medium. Most waited until the cameras got good enough, though.

I'm not sure when that'll be with AI art gen. Seeing scenes broken up into layers in the latest gpt image generation, though, made me think we might be near.

Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that.

AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.

1. They can be more powerful than any one human.

2. They lack liability as they are not people.

3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.

That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.

That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.

This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.

Solve one, and you solve the other.

That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.

It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.

I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.

The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.

It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.

Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.

There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).

This is cloud-era software philosophy. The device isn't yours. The software isn't yours. You are not a sovereign in your private space.

It was a choice, made by product designers and government regulators. They came from an ethos that had no respect for individual human dignity and self-determination.

The good news, though, is that it's mostly the software layer we're interacting with. And software has become so very much easier and cheaper to build. And it is easier still to take software that's built already and make it yours (as long as it's open source). These things can all come together to end the cloud age, if we want them to.

We can, right now, and individually, choose to use software that is ours. The chains that were real - that software required lots of people to build and proprietary formats were hard to interoperate with - are gone. We just have a learned helplessness problem from spending the last 15-20 years under those chains.

Well, Baby steps I guess?

Nearly everyone in the US has a giant car that is basically a tank.

They are dangerous. But it’s managed.

Centralized power never works. We have the worst times in history to look at.

Nationalization just centralizes to a different set of people. It’s personal ownership or oppression.

We all need open weight R2D2s.