aaroninsf

Every technology revolution needs its John Henry, I guess.

> Knowing these models’ decision making is inconsistently aligned with human preferences tells us we can’t just trust their decisions out of the box.

I'm not sure this is true.

What is the "human preference," in this assertion?

My first thought was that as with Jev itself, it is not a position, it's a distribution of positions!

I think that human preferences, plural, will be a distribution of values across the population of various humans you ask; and, a set of specific conclusions humans will argue for on the basis of various philosophical strategies or maxims. Two different ways in which there is not any specific singular preference.

(I'm convinced you could not get uniform agreement on even "obvious" propositions, across a large sample set; and not necessarily just because of contrarian spoiling. Moral intuitions varying, is the why this problem is so well known, after all.)

I think the conclusion is partially right, but only if you adjust it to say, if what you want is alignment with a specific outcome, you need to be able to defined that outcome, even if you can't argue for it.

That's one of the fundamental problems of alignment, full stop; we can't from axioms construct the set of moral propositions we would like. We don't know what the right set of axioms is; we just know "right" when we see it. (And our neighbor may or may not see the same.)

> The bad news about modern neural networks is that they're enormous. The good news is that the enormity helps...

I always catch on misuse of enormity

It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.

Never meta.

Do not run, do not support, do not use.

Hot take: the problem here is not the one implied or AFAICT assumed by the author.

The problem is not that LLM or, say, generative AI tools such as Midjourney, "have no imagination" and collapse upon the same solution. Anyone who has spent any serious time exploring what was originally called the "latent space" Midjourney provided a view into, is aware that the combinatorics around what these systems have ingested is inconceivably large and full of wonders and surprises... what we might well describe as unexpected, novel, dare I say creative, imagery [etc.]

The problem AFAICT is asking for examples of something without providing any process or instruction.

Vibe-coded single-shot prompts are inevitably (unless this is patched) going to converge on statistically-likely or contextually hinted examplars.

And this is true all the way down. It's true of secondary and tertiary features (what camera angle do we have on the pelican? Is it a juvenile or injured? Is it riding to the left or right? etc.)

The fact that humans are a heterogeneous pool with different experiences and capabilities, means if you sample humans, you get high variability. If you ask contemporary LLM, they are converging, indeed, but not in the way implied; they are converging because the corpus they are trained on is comprehensive and coextensive.

What you get is the centroids in the space.

Until you simply ask for something else.

If you want more "creative" examples, of pelicans or anything else, it's more or less trivial to get them. And if you detect durable secondary or tertiary qualities you want to see varied, you ask.

More seriously,

if what you want is to interact with large capable models/systems which by default exhibit variability , you do the (minimal) work to make that happen.

Midjourney has from very early on exposed explicit dials to turn in exactly this way.

I am surprised the author does not know this.

how do you find Voxtral quality?

This framing suggests that this is the sum of their business strategy, and strongly implies that it's being pursued cynically.

The risks posed by open access to highly capable AI are unprecedented.

The answer is not to ban open models; nor is it do nothing.

That we don't have answers technical political social or otherwise to how to navigate these risks, is not Anthropic's fault.

It's actively unhelpful IMO to amplify cynical takes which wave off or sideline the risk, or the reality that this is a very hard problem, among the hardest our societies have had to negotiate and navigate. It encourages a naive primary color tribalism that actively prevents process and progress.

Tao is behaving very much like a human.

He's got an understandable personal and professional grievance, and has found a very brightly lit soapbox,

and is now making many unforced errors in a very public way.

He should know better.

Few of us prove to when our time comes.

...and self hosted models are raw milk.

Not a bad metaphor: mostly fine, arguably legitimate benefits [when you're lucky], potentially really bad.

The biggest place the metaphor breaks is that when you drink bad milk, the consequences are local.

Note: this is not an argument that oligopolies or the state should control or have sole access to AI. It is a reflection of that fact that we as a society have no good answer nor even any process towards one, about how to balance the competing goods and risks of widespread access to capable AI.

Serious question:

Why would anyone believe this (also) is not simply example N+1 of this is the worst it will ever be, as opposed to recognizing this as what will almost certainly prove to be an awkward moment, soon to be replaced by another order of magnitude of cleaner, clearer, more intelligible, etc.?

Ximm's Law: every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon.

That would be a false similarity in some important ways.

This was a very specific single technical problem, requiring a silo from basic science to theoretical physics to advanced engineering, etc., with each layer having its own space of failure modes.

Something like the reverse is true with sufficiently-capable and sufficiently-efficient AI, which we are rapidly evolving to.

In our current situation, the attack surface is quite literally our entire highly interdependent civilization and its components. AI does not need to be autonomous, it just needs to employed, either will wilful bad intent, or, poorly stated goals, or, poor alignment, or, simple mistakes and errors, to break things.

The challenge with security is always that the attacker need only succeed once.

This does not mean doomerism has it right, especially in narratives based on poor understanding or extapolation beyond what is now konwn. The chance of "extinction" in particular is overstated.

The chance of one or more compounding "Chernobyl scale" gray swan events however is ever increasing.

Personally I don't think that's remotely contestable.

As a civilization and society and industry we need to have process, and public process at that, about the nature of the risks faced.

That process has not yet emerged in any coherent way; a hundred letters have been signed and such but to no actual effect, bar one performative "executive order" this week.