> All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
This problem extends to humans, and that's why a complex regulatory system including but not limited to the FDA was created. That can still be manipulated (the Sackler Family, Purdue Pharma), but not in any way that machines are good at now--I don't think making thousands of malicious requests against Hugging Face until something succeeds is particularly transferable to anything that a human regulator wouldn't immediately notice.
There's also no reason why the trial would need to involve AI at all. The benefits of that would be more evolutionary (improved transcription of patient conversations with the doctors or whatever), so an AI candidate could simply be tested through the usual human-directed trial process. As the grandparent notes drug discovery already works through optimization, just not purely in silico. So that problem of gaming your optimization target already exists, and the trials are already designed to mitigate that.
This problem extends to humans, and that's why a complex regulatory system including but not limited to the FDA was created. That can still be manipulated (the Sackler Family, Purdue Pharma), but not in any way that machines are good at now--I don't think making thousands of malicious requests against Hugging Face until something succeeds is particularly transferable to anything that a human regulator wouldn't immediately notice.
There's also no reason why the trial would need to involve AI at all. The benefits of that would be more evolutionary (improved transcription of patient conversations with the doctors or whatever), so an AI candidate could simply be tested through the usual human-directed trial process. As the grandparent notes drug discovery already works through optimization, just not purely in silico. So that problem of gaming your optimization target already exists, and the trials are already designed to mitigate that.