Comment Comment by qigtyofhp The attention mechanism started as a simple trick to not use recurrent neural networks.Read the intro in the original paper "Attention is all you need" (https://arxiv.org/abs/1706.03762)This video explains the drawbacks to RNNs and how transformers solve that: https://youtu.be/S27pHKBEp30?t=394Andrej Karpathy explains attention here: https://youtu.be/kCc8FmEb1nY?t=3719He explains how attention is seen as a communication network: https://youtu.be/kCc8FmEb1nY?t=4298 parent qigtyofhp 3 years ago
Read the intro in the original paper "Attention is all you need" (https://arxiv.org/abs/1706.03762)
This video explains the drawbacks to RNNs and how transformers solve that: https://youtu.be/S27pHKBEp30?t=394
Andrej Karpathy explains attention here: https://youtu.be/kCc8FmEb1nY?t=3719
He explains how attention is seen as a communication network: https://youtu.be/kCc8FmEb1nY?t=4298