I’ve always wondered why we don’t add a few frozen “layers” to models that perform complex computations 2/3rds deep in the model. Some set of steerable fft + some common vector math operations could be reusable units of capability that the model learns to leverage during training.
Given that models learn to use residual connections optimally, it can’t be hard for them to steer around the fixed computation layer when it’s not needed.
I’m sure giant models invest a few layers in reproducing these functions, but surely smaller models can benefit?
Given that models learn to use residual connections optimally, it can’t be hard for them to steer around the fixed computation layer when it’s not needed.
I’m sure giant models invest a few layers in reproducing these functions, but surely smaller models can benefit?