Comment by joshmarlow

No - a differentiable program would still be running on a real computer (which is a finite approximation of a Turing machine). So for a differentiable program to solve the halting problem, that would imply that the underlying machine could solve the halting problem (which it can't).