How is the Token/second calculated? I ask it a simple prompt and the model generated a 150 word (about 300 tokens?) answer in 17 seconds, then mentioning the speed of 408T/s.
Also, I guess this demo would feel real time if you could stream the outputs to the UI? Can this be done in your current setup?
Also, I guess this demo would feel real time if you could stream the outputs to the UI? Can this be done in your current setup?