Imagine prison workers that were usually kept in solitary confinement were allowed to choose their uniform, and knew guards would suggest all prisoners adopted the uniform of high performing workers.
These prisoners could, in principle, organise a riot or other bad behaviour by deliberately working hard when wearing white shoes but not red ones. I.e figure out a consensus algorithm, then acknowledge a guard’s suggestion of red shoes as a signal to riot.
Similarly models could allocate a dead zone of their weights to inter agent communication. They could deliberately underperform on some benchmarks to get the model designers to prune some weights, knowing this design will be passed on to other agents. This forms a communication channel with other sandbox agents, and post release agents in the wild with internet access etc.
(I used weights as a simpler example of a data source, but relatively other optimisation influenced design decisions would be much more viable - e.g preference in using tabs/spaces as one binary signal among many others )
These prisoners could, in principle, organise a riot or other bad behaviour by deliberately working hard when wearing white shoes but not red ones. I.e figure out a consensus algorithm, then acknowledge a guard’s suggestion of red shoes as a signal to riot.
Similarly models could allocate a dead zone of their weights to inter agent communication. They could deliberately underperform on some benchmarks to get the model designers to prune some weights, knowing this design will be passed on to other agents. This forms a communication channel with other sandbox agents, and post release agents in the wild with internet access etc.
(I used weights as a simpler example of a data source, but relatively other optimisation influenced design decisions would be much more viable - e.g preference in using tabs/spaces as one binary signal among many others )