Misaligned to exploit the trial process is actually something to consider carefully. I would not put it past a drug company to design a phase 3 trial that has some subtle bias towards commercialization. And if we're just getting cocktails and trial protocols from the AI, it would be very wise to be skeptical.
For example, if the prompt is something like "give me a cocktail that is most likely to succeed in this specific phase three trial" there are any number of ways the AI might find ways to game specific features of the trial design.
For example, if the prompt is something like "give me a cocktail that is most likely to succeed in this specific phase three trial" there are any number of ways the AI might find ways to game specific features of the trial design.