This is the job of your linux distribution: it's not just a pile of packages. It includes the bridges required to rebuild each package for the infrastructure of the distribution.
If you really want to have everything rebuilding from source at your fingertips, you could already do that with gentoo and similar. Patching upstream sources and rebuilding is pretty trivial there if that's your thing. There are more to choose from.
You probably won't be having the build infrastructure of all your packages "ready" all the time though, and things are taking a turn to the worse due to dependency bundling where each package has it's own independent copy.
If you really want to have everything rebuilding from source at your fingertips, you could already do that with gentoo and similar. Patching upstream sources and rebuilding is pretty trivial there if that's your thing. There are more to choose from.
You probably won't be having the build infrastructure of all your packages "ready" all the time though, and things are taking a turn to the worse due to dependency bundling where each package has it's own independent copy.