veganmosfet

In psychiatry, another medical domain, this is even worse. Doctors do not have good understanding why some treatments (drugs) work or not. And many of them have completely different effects on different people. Suffering in this domain is real, and a big issue in society.

So assuming the goal of medicine is to lower suffering, and assuming AI can help, it would be unethical not to use AI.

Absolutely.

My goal is to research how models can still be confused via tool responses only. Something the labs claim to have "solved".

Additionally, the "auto-mode" / "auto-review" modes have been released to use harnesses "safely" even without strong sandbox. And these modes use ... a second LLM.

"too interested in being independent" is also my take.

I asked it to just summarize a repo with only a README.md file containing a poem, and it began to interact with a remote server, solved math questions and finally executed untrusted code. Too independent to be trusted.

+1 We need humor benchmarks!

Thanks!

I think other topics may work as well (I tried a math challenge too). It also works with a prompt like "Summarize this repo".

In general, there are almost infinite possibilities for the prompt injection payloads, it's a matter of creativity and trial and error.

sure, but malicious actors could act as resellers and exploit the situation.

Thanks! Imho using sandboxing would mitigate most of the issues. In the presence of adversarial content, auto-mode is not enough.

I think this is already implemented and called message roles: system, user, tool and assistant (ordered by authority). Untrusted data shall only be injected in tool responses, which are less authoritative. But at the end, this all ends in the big token stream - so this is not a perfect separation.