You are right that the field doesn’t have a theoretically sound explanation for the architectural choices aside from “A works better than B”. However, I would argue this is an ideal opportunity for the “gentleman scientist” or eager 17 year old.
Basically every part of the original transformer was replaced with something more efficient or better:
LayerNorm -> RMSNorm
Sinusoidal position encoding -> RoPE
MHA -> GQA
ReLU -> GELU
What this means is that there is ample opportunity to improve on what we’ve done thus far.
Basically every part of the original transformer was replaced with something more efficient or better:
LayerNorm -> RMSNorm
Sinusoidal position encoding -> RoPE
MHA -> GQA
ReLU -> GELU
What this means is that there is ample opportunity to improve on what we’ve done thus far.