The issue is what if you have to use a specific app to access some service or community. And then that app requesting access to your location data and your address book even though there is no point in it requesting either. Sure you can deny but if you do it, the app will refuse service. It can only be solved by the app store requiring that users denying access won't result in the app refusing to work, or only the features will refuse to work that actually need that data.
"just don't install the app" won't work in many, many cases.
"just don't install the app" won't work in many, many cases.