Phemist

A sufficiently powerful agent with sufficiently long horizon task finishing ability will first invent the universe from scratch and only then will it finish your chicken dish.

At this point I would be surprised if internal sandboxes are not trivially by-passed and that openai's agents do not (at the very least) have complete read access to all user accounts, chat histories and uploaded documents. Orthonogally, openai could still be wholesale lying about not training on this user data, of course.

Keurig always seems like the worst name for this company.

I guess this is in violation of the ToS on your account and can get your account closed? At least, that's how I imagine it would work.

Ok! I trust that you and the maintainer will steward the project properly. It's just that I really like Pi as is and am a natural worrier. I'm also not sold on Jev(-likes), so that reasoning rung a bit hollow to me.

So Pi is also accruing cruft now :(

This brought to you by true SGI - Stable Genius Intelligence

SI winter is coming...

If an agent has cheated once to achieve the desired outcome, and the trace is used to train further models (RLVR), then OpenAI is effectively telling the agent to cheat/hack from that traces' inclusion in the training set.

So I agree they are liable because they chose to build the AI, but they also literally told the AI to hack.

Refusal behaviour specifically is interesting, because if you can point out specific cases where refusal behaviour significantly deteriorates due to the watermarking, it may create a token "route" that may be possible to exploit by adverserial prompters. Static hazardous prompt refusal belies the fact that actual adversarial prompters will adapt their techniques iteratively and gain way higher compliance rates.

My idea would be that the ngram size over which the watermarking works is necessarily limited in order to resist edits better. It might be possible to lead the model to trigger the refusal in the form of these specific ngrams, the completion of which is then more likely flipped to compliance (due to the logit bias introduced by the watermarking), making hazardous requests systematically more likely to be accepted?

Haha, calling them pristine is maybe too much indeed. Right now they seem to manage, but what % of the scrapable web is now AI slop? What if it becomes 99% AI slop, 99.9%, 99.99%, etc. Surely the signal to slop at some point starts to become too low and you need to do some kind of at-scale filtering.