Comment by bobm_kite9

Yes, you're right. There could be multiple ways to tokenise a sentence. Shouldn't all the valid tokens be included in the vector?