My concern about effectiveness about the "AI safety" stuff is that it kind of sidesteps the whole effectiveness analysis: if you're nominally working on risks you claim to be existential or apocalyptic, then the work is almost automatically "effective" even if you don't do anything really, just because the risk is
so big that an infinitesimal decrease amounts to massive impact.
I'm sure just because of the nature of the movement there's been lots of thinking about the likelihood of various future scenarios and accounting for this in less flippant terms than I've just described it, but it seems to me that this is a group of people who are skewed to think AI is more likely, more potentially dangerous, and more interesting to work on and fund.
Putting aside the likelihood of AGI materializing at all, it's not really clear to me that paying for a bunch of research on this necessarily affects the actual AI that would be produced anyway, as that's being done by different people with different interests and incentives. You see people discuss the numbers of people affected by something like malaria prevention versus eventual extinction at the hands of AI as a justification for the need for AI work, but less talk about how we actually know what kinds of things we can do to do malaria prevention and really not so much the AI stuff. Obviously you have to start somewhere but...
From an outsider's perspective, the whole area just kind of feels like a pit you can throw money into while pretending you're being rigorous about spending.
I'm sure just because of the nature of the movement there's been lots of thinking about the likelihood of various future scenarios and accounting for this in less flippant terms than I've just described it, but it seems to me that this is a group of people who are skewed to think AI is more likely, more potentially dangerous, and more interesting to work on and fund.
Putting aside the likelihood of AGI materializing at all, it's not really clear to me that paying for a bunch of research on this necessarily affects the actual AI that would be produced anyway, as that's being done by different people with different interests and incentives. You see people discuss the numbers of people affected by something like malaria prevention versus eventual extinction at the hands of AI as a justification for the need for AI work, but less talk about how we actually know what kinds of things we can do to do malaria prevention and really not so much the AI stuff. Obviously you have to start somewhere but...
From an outsider's perspective, the whole area just kind of feels like a pit you can throw money into while pretending you're being rigorous about spending.