The output tokens are identical in either case, but there are quite a few additional settings and formats that huggingface compat mode can generate. In general it also requires inputting Python lists of Python strings.
The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.
The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.