ModelForge
Joined 1 year ago
Karma
713
Submissions
34
Member since
Dec 18
Recent submissions
Comment · 3 months ago
Good question, it's 100%. I.e., the developers usually share a reference implementation with e.g., Hugging Face transformers to load their weights, and from there on you can read the code and, if you have time, reimplement and check everything. It's actually a great learning exercise where you can self-check whether you reimplemented it correctly by comparing the LLM outputs to the reference implementation. Made a video about that workflow a while back here: https://www.youtube.com/watch?v=TXzQ7PGpO6w
Comment · 3 months ago
And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"