anuj0456

the only way i am aware of is to cross check with what deployed on huggingface/transformers repo

yes, that is correct. after chinca came into picture the advancement in this field sky rockted

fixed

i will see what i can add here. contributions are welcome

thanks for pointing out. its a bug

Thanks for sharing

yes, most of them are similar. but implementation of GQA, MLA, mHC, Sliding Window changes the implementation drastically because of which the overall model effeciency changes.

yes. this is just raw implementation of the model arch as described in papers. for complete model training with back propogation we need training pipeline with optmizer and loss calculation.

will try my best, but open for contribution.

Thanks, Yes PyTorch is a framework largely used to create neural network models.

I have been studying modern LLM architectures and started implementing them from scratch in PyTorch to better understand the design choices behind each model.

OpenArch is a collection of these implementations, including Llama, Qwen, DeepSeek, Gemma, Kimi, GPT-OSS and others.

The goal is to keep the code readable and useful as a reference when going from the paper to an actual implementation.

Would be interested in feedback from people working on model architecture and training.

Sure, we will add comparison matrices. this is helpful