Comment by brennvin

Tokenizing everything in one go is usually the right approach. The typical case is the error-free case, so you won't actually tokenize more characters than necessary, but you will simplify the implementation and avoid jumping back and forth between parsing and tokenizing (thus improving locality).