Is there any measure, not of size or token amount, but of diversity in the content of the text?
Did that metric meaningfully change when the amount of required memory dropped?
If the amount of diversity is lowered, I would expect that to lower the amount of patterns to be modeled from the text. If that is the case, then the resulting model size itself would be lowered, during and after training.
Did that metric meaningfully change when the amount of required memory dropped?
If the amount of diversity is lowered, I would expect that to lower the amount of patterns to be modeled from the text. If that is the case, then the resulting model size itself would be lowered, during and after training.