Thanks for sharing! I would love for some go to source for this type of insider info from the people who actually have tried. (The best I've found so far are some guys on Twitter/X like Alex Fateev (https://x.com/superalesha) and AJ (https://x.com/ItsmeAjayKV))
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.