Exactly. Tao is a mathematician discussing complex fields which aren't math and he's showing his ignorance. And I don't say that in a negative way, just a factual way. I think a lot of people would be shocked to know how little professionals actually understand about the stuff they work on, including (and especially) medical professionals. It's not like math lol.
But just to highlight the issue with his statement: if I have cancer and I'm dying of cancer and there's a promising treatment which hasn't been fully understood by even a single human researcher but which has demonstrated clear positive outcomes, then I'm gonna take it. I agree that this is really just a balancing of risks, so I likely wouldn't take such a cocktail if it was just to, say, cure my sniffles. But when the alternative is terminal illness, you'll find that people become willing to accept absolute quackery for cures, so an AI produced cure that, again, has enough evidence to make me believe it is worth a shot is not a stretch.
But just to highlight the issue with his statement: if I have cancer and I'm dying of cancer and there's a promising treatment which hasn't been fully understood by even a single human researcher but which has demonstrated clear positive outcomes, then I'm gonna take it. I agree that this is really just a balancing of risks, so I likely wouldn't take such a cocktail if it was just to, say, cure my sniffles. But when the alternative is terminal illness, you'll find that people become willing to accept absolute quackery for cures, so an AI produced cure that, again, has enough evidence to make me believe it is worth a shot is not a stretch.