Any AI safety experts here? I'm wondering if this claim here really holds:
> Anthropic invests significantly in making Claude safe, helpful, and harmless. We conduct rigorous pre-release testing, implement multiple safety layers, and continuously monitor our models' behavior. When Outputs are used to train new models without our oversight, additional risks emerge. Safety controls may be lost – models trained on Claude's Outputs won't have our safety measures, potentially leading to harmful or dangerous AI systems.
From my understanding distillation pretty much copies behaviour. If someone intends to distill from a Model, it can't extract unsafe behaviour, but would learn the same safety mechanisms, no?
From my understanding distillation pretty much copies behaviour. If someone intends to distill from a Model, it can't extract unsafe behaviour, but would learn the same safety mechanisms, no?