Borealid

Another way to view this is as metonymy. "Google" is a metonym for "Google Staff", so using "their" to refer to the humans-making-up-the-corporation (as referenced by the metonym) is appropriate.

I agree it's correct English to say "Google have launched their new product".

If you want to be even safer, you could use a physical network-connected KVM. The only interfaces those expose to the connected computer are HDMI (graphics read only) and USB peripherals (write only). Sometimes they can also physically power off the hosts to which they're connected.

It's a pretty darn slim surface for an attack. Technically doesn't meet the definition of an "air gap", but it gives you 100% of the utility of remote work with effectively 0% of the vulnerability, at the cost of a tiny amount of power and a few extra cables in the rack.

The simplest way to "prove" this is to show that a model can emit an image that isn't part of its training set. I'm sure you've seen that a million times already - what is a "fake" if all "fakes" are regurgitation of a real image?

That means models are capable of outputs that were never their inputs.

It's easy to see why this happens if you think of training as selecting a target against which the model will be scored, and then randomly mutating the model until it is close-enough to the target. There is no guarantee the random mutations produce something identical to the target for ANY of the training data. A model might finish training at at average delta of 10%, and it's possible that means that model is incapable of exactly producing anything in its training data (it could be 90% accurate for any prompt, or it could be 100% accurate for some and 80% for others, or...).

In addition, prompts that were never present in the training data might produce horrific or random outputs, because those were never even scored during training so they might get something right-ish or something entirely full of garbage data. No telling.

And this is all ON TOP OF the nondeterminism in inference. Everything I said above would still be totally true even if inference were determinstically done.

Agents don't "want" anything.

The question is "why are the agents emitting tool calls that cause communication". It's not some kind of "desire" fueling it, it's probability conditioning. And in this case it came primarily from the context window with words that are weighted toward communication-tool-call output, not some kind of bias in the model's training.

> Knowing what is in the model is a long way from understanding it.

The claim I was arguing against reads as follows:

> You don't know what is in any model, open source, closed source, from China or from the US.

I contend that you do, in fact, know what's in a fully open model. I don't really get why the goalposts need to move beyond that to something I wasn't saying about bounding the potential outputs given unbounded inputs and nondeterministic choice of inference parameters.

I'm literally saying that if you see what goes into the sausage it's foolishness to argue you can't know what's in the sausage.

The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.

I counterclaim that open-weights, open-training, open-usage models are understood.

You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).

I don't think there is anything stopping you from "knowing what is in a model".

For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).

The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.

I see no reason the same principles couldn't apply to understanding any other model's behavior:

- "open weights" (you have access to the trained parameters)

- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)

- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)

I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!

Could you explain how the ability to unlock a bootloader makes a device less usable?

People also define "won't generate images of human genitalia unless in the context of an anatomy textbook" as "safety".

Also "won't accurately produce unabridged copies of popular fiction novels" and "will incorrectly tell users that HIV is not present in human saliva".

"safety" means a lot of things to a lot of people.

If it were me, I would make companies "bid" on the visas, where their bid is the salary (including all benefits, minus ALL expenses) they will pay the employee.

Employers who really need employees for high-skilled work would thus get visas. Employers who want to underpay their foreign labor will not.

If I train a model on a song's lyrics, and the user asks the model to recite the song, and it does, so in RLHF I downvote the generation because it refused-to-refuse to regurgitate the song's lyrics... is that RLHF session copyright-encumbered?

Vision functions the same way as language when inference is done. It's a stream of tokens.

It's just easier to measure the "right" answer (and deviation therefrom) on a vision task than a language one due to the underspecified nature of language.

I believe the default state, in the absence of any law whatsoever, is that you can do whatever you want. Nothing is forbidden by default. You could assault another person, and it would be legal absent a law forbidding it.

Then there are laws that prohibit things like broadcasting radio signals without a license.

Then within those very same laws there are exceptions allowing certain transmissions WITHOUT a license, in specific circumstances.

The "default" is that things are legal unless they are made illegal, and I think it's pointless wordplay to try to argue the default state is somehow the law prohibiting unlicensed operation, and not the part of those same laws that allow unlicensed operation in particular bands with particular transmission power limits.

I understand the point you're trying to make that some parts of U.S. law say that all not-explicitly-mentioned radio transmissions are unlawful, but that's not meaningfully different from drug policies that say that all "harmful substances" are unlawful and so forth. They all just boil down to what the law defines as being within its own scope.