Is this applying the learnings from vision transformers to language transformers?
If I understand correctly, vision models split an image into tiles and append a positional encoding to each so the model can understand the relative position of each tile.
I admittedly only read the abstract - a lot of this stuff goes over my head - but it seems like this paper proposes a similar idea, but for 1D instead of 2D?
If I understand correctly, vision models split an image into tiles and append a positional encoding to each so the model can understand the relative position of each tile.
I admittedly only read the abstract - a lot of this stuff goes over my head - but it seems like this paper proposes a similar idea, but for 1D instead of 2D?