kouroshh

Hey, I am one of the co-authors of the post. So the training data for ViGGO has about 5.1k rows which we trained with a block size of 512 (you can lower the block size if you want but we didn't do so because it was just easier to not change code :)). On 16xA10Gs for 7B it took ~15 min per epoch and on 13B it took ~25 min per epoch. So the on-demand cost per epoch is ~$7.2 for 7B and ~$12 for 13B. This is based on the time only spent on the training part and does not take into account the cluster startup time and shutdown time.

Would be good to see a rigorous analysis of these PEFT methods on quality. There still seems to be a debate on whether these methods sacrifice quality or not.

Llama-2-chat models have been overly fine-tuned to be like this. You can give a few-shot prompting a try, but they still don't gurantee a desired output. The best way to guarantee is to fine-tune on small (~1k) data points and go from there.