that's a great point you made, but the goal of this research paper is not to improve the performance, but to show that you can train deep neural networks without the need of activation functions, normalization layers, deep neural networks.
one simple usecase for them is physics-informed neural networks and neural ODEs, where using activation functions is discouraged, mainly because they aren't infinitly differentiable, and they use the tanh or the sin most of the time,
this kernel i introduced works better then the neurons followed with a tanh to solve different PDEs
one simple usecase for them is physics-informed neural networks and neural ODEs, where using activation functions is discouraged, mainly because they aren't infinitly differentiable, and they use the tanh or the sin most of the time, this kernel i introduced works better then the neurons followed with a tanh to solve different PDEs