Comment by InclinedPlane

You have to consider instead the probability of detecting such objects that don't come very close to Earth. And that probability is actually pretty low (currently), so the proximity tells us very little beyond some general estimate of how common such objects are.