Comment Comment by algo_trader Infact, as stated in the paper, this is bad news> We therefore leave the attention layers untouchedMeaning, presumably, that the GPU memory remains the bottleneckFlops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !! parent algo_trader 3 years ago
> We therefore leave the attention layers untouched
Meaning, presumably, that the GPU memory remains the bottleneck
Flops really are quite cheap by now, e.g. vision inference chip ~$2/teraflop/s !!