I have been studying modern LLM architectures and started implementing them from scratch in PyTorch to better understand the design choices behind each model.
OpenArch is a collection of these implementations, including Llama, Qwen, DeepSeek, Gemma, Kimi, GPT-OSS and others.
The goal is to keep the code readable and useful as a reference when going from the paper to an actual implementation.
Would be interested in feedback from people working on model architecture and training.
OpenArch is a collection of these implementations, including Llama, Qwen, DeepSeek, Gemma, Kimi, GPT-OSS and others.
The goal is to keep the code readable and useful as a reference when going from the paper to an actual implementation.
Would be interested in feedback from people working on model architecture and training.