It's totally separate from the topic of efficiency gains or code reviews, but the idea of 100% pairing would take most of the enjoyment out of programming for me. Working in isolation (for the most part) to solve a hard problem and drive it to completion is incredibly satisfying. A lot of that goes away when you're working side by side.
You're never in "the zone" if you're pair programming.
You're never in "the zone" if you're pair programming.