The reasons are twofold. Like you mentioned a pitcher's aim isn't perfect. Also a batter isn't perfect at reading pitches. A pitch aimed at the bottom of the strike zone might miss accidentally. Or a pitch like a curveball can be aimed below the strike zone but it appears to the batter to be a strike only for it to drop and end up outside the zone. Not swinging at these seemingly borderline pitches results in a lot more pitches being called balls.
Also pitches thrown in these tough to hit locations will often result in bad contact. Bad contact is generally worse than a strike. A ball that is weakly hit into play will almost always result in an out. A strike will only result in an out when there are already 2 strikes. Therefore swinging at fewer pitches means the batter sees more pitches and gets deeper into counts. The batter will therefore have more late counts in which they are ahead simply due to fewer at bats being ended prematurely.
Also pitches thrown in these tough to hit locations will often result in bad contact. Bad contact is generally worse than a strike. A ball that is weakly hit into play will almost always result in an out. A strike will only result in an out when there are already 2 strikes. Therefore swinging at fewer pitches means the batter sees more pitches and gets deeper into counts. The batter will therefore have more late counts in which they are ahead simply due to fewer at bats being ended prematurely.