Oh yes, absolutely. We only make claims for "natural distributions and models" (^).
It's almost certainly possible to break this by pathological choice of a data distribution or model/initialization/optimization scheme.
But, I don't think this is interesting -- what I think is interesting is that this seems to hold true in real life, in "natural settings".
---
(^) Whatever "natural distributions" means... (which I think is a good research direction in itself).
It's almost certainly possible to break this by pathological choice of a data distribution or model/initialization/optimization scheme. But, I don't think this is interesting -- what I think is interesting is that this seems to hold true in real life, in "natural settings".
---
(^) Whatever "natural distributions" means... (which I think is a good research direction in itself).