We've gotten very close to physical limits. Quick back-of-the-envelope estimate: if electrons traveled at the speed of light through silicon (they don't), then in 3GHz, an electron could travel .1 meters. In reality, electrons in silicon travel quite a bit slower than that. Net result: electrons can barely cross the diameter of the chip in one cycle, even without gate propagation delays and other factors limiting work done per cycle.
So, if you want data from a tiny distance away, such as a local register, you can grab it and do something simple with it in one cycle. If you need data from any further away, forget about it; cache takes longer, another core takes even longer, and memory takes far longer.
So, if you want data from a tiny distance away, such as a local register, you can grab it and do something simple with it in one cycle. If you need data from any further away, forget about it; cache takes longer, another core takes even longer, and memory takes far longer.