For those who have no idea what's being discussed, quick background.
Discussing: Transformer [1] memory issues and approximate attention [2] in machine learning training.
Specifically: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. [3]
As a side comment, this entire industry is sorely in need of at least intros. The entire space has moved so fast in the last year I need an entire new dictionary and thesaurus for all the terms they've created. Notably, because of this, found out Google has a glossary of machine learning terms. Actually somewhat handy.
Discussing: Transformer [1] memory issues and approximate attention [2] in machine learning training.
Specifically: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. [3]
As a side comment, this entire industry is sorely in need of at least intros. The entire space has moved so fast in the last year I need an entire new dictionary and thesaurus for all the terms they've created. Notably, because of this, found out Google has a glossary of machine learning terms. Actually somewhat handy.
[1] Google Machine Learning Glossary (Transformer): https://developers.google.com/machine-learning/glossary/#tra...
[2] Same (Attention): https://developers.google.com/machine-learning/glossary/#att...
[2] arXiv: https://arxiv.org/abs/2205.14135