I'm on team open source. To me the exciting thing was ollama downloading the 7B and running it on a 5yo cheap lonovo and getting a token rate similar to the first release of ChatGPT.
Running local on CPU opens so much possibilities for smart and privacy focused home devices that serve you.
In my test it hallucinated confidently but my interest is in simple second brain like rag. "Hey thingy, what is my schedule today?"
Need it to be a bit faster though as the thinking part adds a lot of latency.
Running local on CPU opens so much possibilities for smart and privacy focused home devices that serve you.
In my test it hallucinated confidently but my interest is in simple second brain like rag. "Hey thingy, what is my schedule today?"
Need it to be a bit faster though as the thinking part adds a lot of latency.