I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memorization from LLMs. I've been working with neural nets for 30 years, and fine-tuning language models since 2017, and this behavior is most surprising to me! Other folks have seen it in LLMs too, although I haven't seen a analysis of this kind before (although we might have missed something).
Let me know if you have any questions or thoughts.
I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memorization from LLMs. I've been working with neural nets for 30 years, and fine-tuning language models since 2017, and this behavior is most surprising to me! Other folks have seen it in LLMs too, although I haven't seen a analysis of this kind before (although we might have missed something).
Let me know if you have any questions or thoughts.