marcelroed

Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.

The output tokens are identical in either case, but there are quite a few additional settings and formats that huggingface compat mode can generate. In general it also requires inputting Python lists of Python strings.

The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.

Thanks for the kind words, Craig! I'm planning to do a technical writeup+paper and a presentation video on the project in the near future. Will make sure to share it with the Discord!

Running the numbers now

It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.

The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.

Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.

Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.

I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.

Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.

You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.

In practice every training project I've worked on does tokenization in a separate data processing phase.

We have autograding for code through tests written by hand, and additionally do manual code audits if we see suspicious behavior. We also do grading the old-fashioned way for writeups.

We do indeed catch students who don't follow the honor code. It's very obvious from how the code looks, as well as the rate of progress. Since we use Modal for class submissions, we have code deltas for every time they run something on B200s. The diffs often contain something like 300 lines in 5 minutes, in which case we review and report based on how egregious/provable it looks.

TA here. Noted! I now have more resources to test more environments, and will do so whenever possible. I think freezing due to memory overuse is going to be a problem with anything you code yourself, but I do think we could be more rigorous with guiding people to achieve limited memory use for the tokenizer task.

IMO the cost of renting GPUs is a bit overstated in these comments. Generally almost all of the development can be done locally, and then ran for a short period of time using on-demand GPUs. For assignment 1, you can run everything on your local machine, even if you don't have a GPU. For A1 and A2, you can do (most of) the tasks with only a few hours of renting. Without being too careful using rental GPUs throughout will net you around $200 of a compute budget, but you can easily get this under $50 if you're willing to scale down many of the problems. I think we could work on making this clear and charting what these changes are.

If you have further feedback or encounter problems, feel free to open issues in the repos so we can resolve them! It's hard for us to fix issues we're not aware of.