Don’t get me wrong, I very much agree with the danger. As I highlighted - I saw it this morning when someone used Claude to draw the wrong conclusions.
I’m saying there is a way in which they can be used where there isn’t scope for numerical hallucinations at all. They can write sql queries, for example, without ever being allowed to even see numbers.
What invariably does and will happen though is they’ll inner join instead of left join and some data will get missed. Or there will be some missing context (users in this set already have a certain class of property by virtue of some selection bias and that will be mistreated as some signal etc).
I’m saying there is a way in which they can be used where there isn’t scope for numerical hallucinations at all. They can write sql queries, for example, without ever being allowed to even see numbers.
What invariably does and will happen though is they’ll inner join instead of left join and some data will get missed. Or there will be some missing context (users in this set already have a certain class of property by virtue of some selection bias and that will be mistreated as some signal etc).