I am really surprised you’re seeing half the generation performance of 3.6 with 3.8 at the same parameter size and quantization (and same prefill performance to boot) - is there just an optimization in the stack somewhere for 3.6 that hasn’t landed yet for 3.8?
Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.
Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.