The issue is most of these play-throughs happen on “normal” response speed. Way too slow for these LLMs to send back and forth and process this many tokens in a quick enough reaction time.
I’ve yet to see running these LLMs on Ultra-fast (8x speed on Sol). Wondering if that’s fast enough to do real time
I’ve yet to see running these LLMs on Ultra-fast (8x speed on Sol). Wondering if that’s fast enough to do real time