manux

Balanced, sure. Why is 50/50 ethically coherent?

This is a very relatable comment. I think it's demoralizing to see common objections to AI, it feels to me like we failed and haven't done our communication job. I'm always temped to go, "Yes, but there's worse!"

But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)

That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.

I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.

I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.

Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).

In Canada, all three major AI research centers use clusters created with public money. These clusters receive regular additional hardware as new generations of GPUs become available. Considering how these institutions work, I'm pretty confident they've considered the alternatives (renting, AWS, etc). So that's one data point.

The intent behind media matters but isn't all that matters. How people might interpret something is important (albeit often unpredictable).

I think the symbolism of "let's crush all remaining vestiges of creative culture" is a pretty obvious _potential_ interpretation from a _non-trivial amount_ of people. In that sense it is an interpretation that matters for our present discourse, even if it isn't the interpretation that the creator of the ad intended.

> Demis now has a load of people reporting to him who previously were rooting for his failure

Having been in both places (Brain and DM), this feels so far from what I experienced that I must ask, what are you basing this on?

I know, the CAQ is full of xenophobes, and it can suck not looking white in Quebec (although much less so in Montreal). I'm skeptical that it drives rent prices in any detectable way.

Wow, what a take. I seriously doubt that people in Montreal/Quebec are more or less xenophobic than the average North American; if you have data supporting this I'd be happy to change my mind.

Also please be more skeptical of thoughts of the form "X is Y because of hot take Z". Reality is complex and things have many causes. Montreal has a long history of doing city planning differently than other cities in Canada.

> maybe things just move so fast

This is definitely the case in DL (and I'm assuming elsewhere too but I wouldn't know).

I've lost count honestly, running 1-2 year old paper github repos with some detail missing (like the Python version!) that make it non-trivial to run as is. Libraries make undocumented breaking changes, wrong pickle format, authors used a nightly version which didn't make it to a tagged version, and so on.

This perhaps says also something about the CS (versus software eng) background that most people engaging in DL publishing have.

What the permutation invariance gets you is that the model doesn't arbitrarily prefer one (graph) configuration over another, but this seems tangential. The difference between this and RL is in what we do with the reward:

- RL says, give me a reward and I'll give you its max.

- GFlowNet says, give me a reward and I'll give you all its modes (via p(x) \propto R(x)).

Yes you would ideally have a loss (well, a reward/energy) that is invariant and operates e.g. directly on the molecule rather than on some arbitrary ordering of the nodes.

I think the inspiration came to me from looking at SumTrees and from having worked on Temporal Difference learning for a long time. The idea of flows came to Yoshua and I from the realization that we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state.

But, yes in the moment it felt like some very serendipitous insight!