The tokenization is done by the tokenizer which can be thought of as just a function that maps strings to integers before the neural network. Tokenizers can be hand-specified or learned, but in either case this is typically done separately from training the model. It is also less frequently necessary unless you are dealing with an entirely new input type/language.
Tokenizers can be quite gnarly internally. https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt is a good resource on BPE tokenization.