goldemerald

> Afaiu the projection works with the weight update. But here the vector is getting added to residual stream from engram lookup not as direct updat. So the approach is little different.

Residual stream steering is what the authors do, the orthogonalized weights are downstream of that. Those weights, of an obliterated model, are all computed w.r.t. the computed residual stream vector. The original paper focuses on steering, but the community loves the simplicity of not needing to make changes at test-time.

It's nice to see people actively working on this type of research, but OP's baseline is implemented incorrectly. You are not supposed to simply steer away from refusal, but compute the projected vector and subtract only that. The projected/orthogonalization approach is what's done by the original "Refusal is mediated by a single direction" paper.

It looks like they post-trained Qwen3.6. Interesting to see how far they could improve it with they harness/algorithm.

I was able to replicate OP's attack. Since ChatGPT generates images via a separate model, I was able to ask it to tell me what the inputs to the tool was. It's a null prompt: a completely unconditional image generation. What I'm not sure of is if these are the average image trained on that had no prompt in the dataset, or if they are the true average of the dataset during unconditional training step. Very interesting nonetheless, as typically researchers are only able to see the unconditional generation of open weight models.

Surprisingly when you ask ChatGPT to generate you an image with these tool params, the output is not the same; it's not remotely graphic.

  prompt: null
  size: null
  n: null
  transparent_background: null
  is_style_transfer: null
  referenced_image_ids: null

Edit: after more debugging the image generator does seem to look at the conversation as part of the input conditioning, so the one word change from OP makes more sense. There seems to be a hidden prompt rewriter that looks at the tool's prompt and the conversation to create the final conditioning for the t2i model.

Algorithmically served short form videos is clearly the smoking of our time. I cannot stand the conservative view of "well we don't know the videos cause mental health decline, or if it's simply those with a genetic inclination who seek out short form content.", exactly mirroring the skeptics about smoking causing cancer. I'm hopeful that in 5-10 years (but more likely 20) people will view this AI served, maximally engaging, content in the same way we view smoking now: disgusting and horrible, but adults should be allowed to do what they want. I can easily imagine kids/teens sharing their illicit access to shorts much in the same way they share vapes/cigarettes, which would be a much more preferable situation than the unlimited use we see today.

That is addressing the incomprehensibility of PCA and applying a transformation to the entire latent space. I've never found PCA to be meaningful for deep learning. As far as I can tell, polysemous issue with neurons cannot be addressed with a single linear transformation. There is no sparse analysis (via linear probes or SAEs) and hence the unaddressed issue.

Not quite. For an underlying semantic concept (e.g., smiling face), you can go from a basis vector [0,1,0,...,0] to the original latent space via a single rotation. You could then induce said concept by manipulating the original latent point by traversing along that linear direction.

This is an interesting line of research but missing a key aspect: there's (almost) no references to the linear representation hypothesis. Much work on neural network interpretability lately has shown individual neurons are polysemantic, and therefore practically useless for explainability. My hypothesis is fitting linear probes (or a sparse autoencoder) would reveal linearly semantic attributes.

It is unfortunate because they briefly mention Neel Nanda's Othello experiments, but not the wide array of experiments like the NeurIPS Oral "Linear Representation Hypothesis in Language Models" or even golden gate Claude.