Comment by bacr

There has been a lot of work in this space, and it’s a lot of fun to read around and play with. The Magenta team at google developed a differentiable digital signal processing system that is what you are describing. Here it is doing some tone transfer: https://sites.research.google/tonetransfer/about