I love these smaller models for the added capability they can bring to older hardware.
Recently picked up a 2013 Mac Pro (the trashcan one) on eBay for $150, and the official 6-bit quant of 2.6B runs at 25-30 tok/s under llama.cpp on one of the D500 GPUs. After installing Ubuntu LTS I have myself a lovely (if somewhat inefficient) little homelab server.
Recently picked up a 2013 Mac Pro (the trashcan one) on eBay for $150, and the official 6-bit quant of 2.6B runs at 25-30 tok/s under llama.cpp on one of the D500 GPUs. After installing Ubuntu LTS I have myself a lovely (if somewhat inefficient) little homelab server.