Can someone walk me through exactly how powerful distillation is?
I've never seen a good blog post or metric establishing what % of a new model's "intelligence" can be captured. I'm not disputing the document's claim. Widespread effort at distillation is clear evidence that it is highly effective. I guess I just don't have an intuitive or technical sense for how powerful distillation is.
I've never seen a good blog post or metric establishing what % of a new model's "intelligence" can be captured. I'm not disputing the document's claim. Widespread effort at distillation is clear evidence that it is highly effective. I guess I just don't have an intuitive or technical sense for how powerful distillation is.