The most comprehensive make alternative in python I've seen is Scons (https://scons.org/)
It would be worth to see how they tackles some of the challenges you're looking into.
Blurb from the website:
SCons is an Open Source software construction tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.
It would be worth to see how they tackles some of the challenges you're looking into.
Blurb from the website:
SCons is an Open Source software construction tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.