I'm convinced the path to ubiquity (such as embedded in smartphones) is quantization.
I had to int4 a llama model to get it to properly run on my 3060.
I'm curious, how much resolution / significant digits do we actually need for most genAI work? If you can draw a circle with 3.14, maybe it's good enough for fast and ubiquitous usage.
I had to int4 a llama model to get it to properly run on my 3060.
I'm curious, how much resolution / significant digits do we actually need for most genAI work? If you can draw a circle with 3.14, maybe it's good enough for fast and ubiquitous usage.