Comment Comment by cschmidt I'm curious if you're aware of some papers from around 2005 on using contextual entropy to do unsupervised word segmentation on Chinese, and other languages that don't use spaces for word boundaries.https://aclanthology.org/Y03-1017/ https://aclanthology.org/I05-1009/ https://aclanthology.org/P06-2056/Exactly the same approach of segmenting a word when the entropy goes up compared to the previous byte. parent cschmidt 1 year ago
https://aclanthology.org/Y03-1017/ https://aclanthology.org/I05-1009/ https://aclanthology.org/P06-2056/
Exactly the same approach of segmenting a word when the entropy goes up compared to the previous byte.