mlnomadpy

yes, since you can learn to represent the same problem with less amount of params, however most of the architectures are optimized for the linear product, so we gotta figure out a new architecture for it

they are one of the reasons neural networks are blackbox, we lose information about the data manifold the deeper we go in the network, making it impossible to trace back the output

this preprint is not coming from a standpoint of optimizing the inference/compute, but from trying to create models that we can interpret in the future and control

that's a great point you made, but the goal of this research paper is not to improve the performance, but to show that you can train deep neural networks without the need of activation functions, normalization layers, deep neural networks.

one simple usecase for them is physics-informed neural networks and neural ODEs, where using activation functions is discouraged, mainly because they aren't infinitly differentiable, and they use the tanh or the sin most of the time, this kernel i introduced works better then the neurons followed with a tanh to solve different PDEs

basicly the real "non-linearity" in deep learning have always been the orthogonality, squashing functions make it easy for the neurons to tap into the orthogonality, while most of the activation functions "lie" about their orthogonality by setting the dot product score to "0", and a dot product of 0 between two vectors means they are orthogonal (linear indep)

what i did was rely on both the angular information and spatial information between the input x and the weight w to measure how "similar" they are.

the lower bound of the yat-product is 0, and it is achieved only when two vectors are orthogonal and away

indeed, there is an extensive work done in kernel learning that is facinating and one of the applications that still do these transformations are satellite imagery/multispectral imagery, you can get more information just from calculating the ndvi from the different bands of your image, which make it easy for your models to make decisions

We built a Notion-style alternative to Jupyter notebooks with local AI built-in. Connect to your own Jupyter server and run code seamlessly.

It's a PWA that runs entirely on your local machine. No vendor lock-in, no data sent to the cloud overlords. WebLLM is supported out of the box.

Fork it, customize it with your own extensions, and deploy it for free on Netlify. You can also host your own hub on Firebase.