The GPL does not limit freedom, it only limits power. Distributing a binary to a user while denying them the source code, while not strictly a malicious act, is an attack on the user's freedom because it puts the user at a position where the author hold power over him/her. The author allows himself/herself to freely study and modify the program in its preferred format (the source code), but he/she has intentionally made it impossible for the user to do this. Copyleft is about authors willingly removing the powers granted to them by copyright and "redistributing" them so that all users are granted freedom.
It's worth noting that the GPL does not say you have to give out source code all the time, you just have to give it out when people ask for it, and you have to make the offer to give it out clear. So the actual issue that you believe is "a limitation on freedom" is the lack of the ability to say no when someone asks for the source code, despite that they have already been distributed a binary. However, there are no practical reasons to do this that don't involve intentionally keeping secrets from people, or intentionally making studying and modification of the program difficult.
It's worth noting that the GPL does not say you have to give out source code all the time, you just have to give it out when people ask for it, and you have to make the offer to give it out clear. So the actual issue that you believe is "a limitation on freedom" is the lack of the ability to say no when someone asks for the source code, despite that they have already been distributed a binary. However, there are no practical reasons to do this that don't involve intentionally keeping secrets from people, or intentionally making studying and modification of the program difficult.