Comment by gbxk

Amazing project. Have you guys thought about running it or adapting it for FPGA-like approaches like this one below that reach Taalas-speed like 20k tok/sec on cheap hardware?

https://www.mikeayles.com/blog/on-chip-llm-kv260/

500 tok/sec is great but this would be 40x and it changes the kind of tasks that it could even be used for if you link it to fast tools too.