Comment by Taek

Looks like you need about 120 GB to fine-tune the 65B model with this code at a sequence length of 512. How does the memory usage scale as the sequence length grows?