Hello, while I was learning LLM Inference basics I realized with few tricks I could use my ESP32 for inference if I stream the weights only when they are needed from a host over WiFi.
Speed? Really bad. Lol. But this is just a technical demo of how it can be done.
I really enjoyed while coding it. It has a sliding window which allows infinite generation ( of course meaningless :) )
Speed? Really bad. Lol. But this is just a technical demo of how it can be done. I really enjoyed while coding it. It has a sliding window which allows infinite generation ( of course meaningless :) )
Hope you will enjoy!