The non linearities are fundamental. Without them, any arbitrarily deep NN is equivalent to a shallow NN (easily computable, as GP was saying), and we know those can't even solve the XOR problem.
> nothing but linearity
No, if you have non linearities, the NN itself is not linear.
The non linearities are not there primarily to keep the outputs in a given range, though that's important, too.
> nothing but linearity
No, if you have non linearities, the NN itself is not linear. The non linearities are not there primarily to keep the outputs in a given range, though that's important, too.