There’s a measurable performance tradeoff versus gqa so there’s reluctance.
For the most part though the new deepseek v4 tech is hca and mhc and people are still catching on like with moe and rl. Wait for 6 12 months, minimum time for next pre train.
There’s a measurable performance tradeoff versus gqa so there’s reluctance.
For the most part though the new deepseek v4 tech is hca and mhc and people are still catching on like with moe and rl. Wait for 6 12 months, minimum time for next pre train.