TPUs aren't transformer ASICs. The Ironwood TPU that Gemini was trained on was designed before LLMs became popular with ChatGPT's release. The architecture was general enough that it ended up being efficient for LLM training.
A special-purpose transformer inference ASIC would be like Etched's Sohu chip.
A special-purpose transformer inference ASIC would be like Etched's Sohu chip.