Comment Comment by umeshunni > in that a distilled model of an LLM is like a JPEG of a photoThat's an interesting analogy, because I've always thought of the hidden states (and weights and biases) of an LLMs as a compressed version of the training data. parent umeshunni 1 year ago
That's an interesting analogy, because I've always thought of the hidden states (and weights and biases) of an LLMs as a compressed version of the training data.