Comment by alcover

Question to compiler writers : when is it useful, if ever, to tokenize the whole input beforehand (as done here) ?

You wouldn't catch an early syntax error and would go on tokenizing till the end for nothing.