It used to be that pre-DeepLearning tokenizers would extract ngrams (n-token sized chunks) but this doesn't seem to exist anymore in the word embedding tokenizers I've come by.
Is this possible using HuggingFace (or another word embedding based library)?
I know that there are some simple heuristics like merging noun token sequences together to extract ngrams but they are too simplistic and very error prone.
Is this possible using HuggingFace (or another word embedding based library)?
I know that there are some simple heuristics like merging noun token sequences together to extract ngrams but they are too simplistic and very error prone.