Comment by kevmo314

Neat idea. Regarding the performance, I think you could get a lot better performance by training a small classifier model, essentially an embedding model, and using the LLM as the distillation source. This would both be much smaller, addressing your desire for it to run in browser, while also being much more performant, addressing your quantization need. Using the full LLM is a bit overkill and you can extract the core of what you're looking for out of it with something a little custom.