Comment by killerstorm

There are many alternatives to attention which work well for small sequence sizes.