Comment by gtoubassi

Token counting is importing when you are injecting fetched data into the prompt to make sure you don't overflow the prompt size (e.g. in retrieval augmented generation). You want to give the LLM as many facts as will fit in the prompt to improve the quality of its response. So even with billions of dollars... token counting is a thing.