ActivatedAI
Joined 2 years ago
Karma
13
Submissions
2
Member since
Aug 2
Recent submissions
Comment · 2 years ago
Page 9 on the Llama tech report has an interesting graph that predicts task level performance from the cross-entropy loss. The sigmoidal model fits well, and at the steepest part of the S, a .01 change in NLL is worth about 5% task level accuracy.
Here is a quick screenshot if you are lazy.
https://snipboard.io/C6mipQ.jpg
And here is the paper if you want to dig deep into the highest quality published research on LLM frontier.
https://ai.meta.com/research/publications/the-llama-3-herd-o...
Comment · 2 years ago
There is some good published research about doing multiple passes over the training data, and how quickly learning saturates. The TL:DR is that diminishing returns kicks in after about 4 epochs.
Cofounded by a ML eng at Google with 16 years of experience, and an LLM obsessed brilliant hacker who was using the GPT3 API on the first day of its release.