Comment by atty

There is a clear risk from these sorts of models as they get better - I mean recreating specific individuals’ likenesses in compromising images (or even worse, video). We’re not at that point yet, but these things are getting better fast, so it’s only a matter of time. The problem is that there’s no way to mitigate those risks except to keep the model behind an inference-only API, or not release it at all - as soon as the model is open sourced then fine-tuning the model can introduce whatever behavior you want. Holding the models back is virtue signaling at best and actively harmful at worst, because it draws attention from individuals who will take it as a challenge to find ways to misuse them, and cuts businesses and the open source community off from models that would otherwise be very useful to them, that they could help to improve. I’m concerned this is becoming a self fulfilling prophecy, where more companies will start to do the same thing, primarily because it’s what everyone else is doing.