I am not sure when you are using neural nets how much knowing the underlying math helps you. Yes we are doing matrix multiplies and using gradient decent. That is stuff you probably learned in high school. Usually you just learn partial derivatives in two dimensions instead of 10s or 100s of dimensions.
Even if you understand the math you are training tens of thousands of parameters inside your model. You are also passing them through non linear elements like sigmoid of ReLUs. I am not sure what insight knowing calculus or linear algebra will provide you in building the model unless you can process more than 3 dimensions non linear elements in your head. I am sure there are people that can do it, but how many can do it?
Even if you understand the math you are training tens of thousands of parameters inside your model. You are also passing them through non linear elements like sigmoid of ReLUs. I am not sure what insight knowing calculus or linear algebra will provide you in building the model unless you can process more than 3 dimensions non linear elements in your head. I am sure there are people that can do it, but how many can do it?