A3B refers to 3 billion parameters active during a forward pass, not the size of an individual expert.
The 3 billion includes all active experts, and then other shared weights, attention components, embeddings.
In the qwen 3.5 35b a3b series, each expert is ~3 million parameters. There are 40 layers which each have 1 shared expert and 8 routed experts. Roughly 1.1B of the 3B active are experts.
The 3 billion includes all active experts, and then other shared weights, attention components, embeddings.
In the qwen 3.5 35b a3b series, each expert is ~3 million parameters. There are 40 layers which each have 1 shared expert and 8 routed experts. Roughly 1.1B of the 3B active are experts.