Comment by yorick

It looks like you can use the gemma tokenizer to count tokens up to at least the 1.5 models. The docs claim that there's a local compute_tokens function in google-genai, but it looks like it just does an API call.

Example for 1.5:

https://github.com/googleapis/python-aiplatform/blob/main/ve...