Comment by anentropic

FWIW there is an example of building a tokenizer/scanner using documented features of the re module here: https://docs.python.org/3.11/library/re.html#writing-a-token...

re.Scanner looks more succinct though...!