Comment by apsec112

I haven't run the code, but... how is this even possible? I've done PyTorch profiling on QLoRA Llama-2-70B fine tunes, and the runtime is dominated by the large matrix multiplies in the MLP layers, plus a bit for attention. Under the hood, this repo calls the same torch.matmul() for MLP and flash_attn_func() for attention as HuggingFace does. So how can it be that much faster? They have a few Triton kernels, but there appears to be no Triton on MLP or attention and that's most of the bottleneck.