soycaporal

mainly this is a portability and compatibility solution.. I think with GPU available deployments, it wouldn't benefit from ternary distillation. It would be a different problem domain.

amazing.. glad to know this integration path worked fro you!

very cool, I'll look into this. Thanks for sharing.

the base model I clocked it at 5 ms per embedded on my mac studio. There is a mini variant (the demo version) that is sub - 2 ms. It could be a SIMD issue.. I'll look into this for better runtime support (also fee free to file an issue)

super cool use case! Hopefully it can provide quality embeddings + retrieval. Would love to learn to results/issues or feedback. Please feel free to file for issues on github

The corpus is mainly trained in english, unfortunately no other languages have been included in the distillation training. Yes it would work like fuse.js, but unlocks semantic search.

Source code has the entire embedding distillation pipeline includes dataset preparation. You could run the same distillation training (but not sure if teacher model used a multi-language dataset).

Open up an issue on the github repo and I can reply with details

awesome, noted, looking for capable teacher models to distill other architectures

that's great! let me know if there is anyway I can support, or any specific use case a roadmap could address!

It's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact.

The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.

CPU cycle maxxing, who said GPUs were special?

thank you! hopefully it can unlock some novel applications, that would be cool

ohh thanks for the report.. probably has to do with wasm runtime.. Will note this as a known issue

love the idea! Will think of a way to host it probably on huggingface

I think standardizing the runtime is pretty effective, it then open up portability

gte-small outscores all-MiniLM-L6 on MTEB (~61 vs ~56 avg per the GTE paper). MiniLM is ternlight's teacher (ternlight holds 0.84 Spearman fidelity to teacher). I haven't run a head-to-head yet; STS-B/MTEB numbers are on the roadmap. Also on the roadmap is to distill gte-small as teacher.