Really great write-up, thank you John.
Two naive questions. First, with the 4060 Ti, are those the 16gb models? (I'm idly comparing pricing in Australia, as I've started toying with LM-Studio and lack of VRAM is, as you say, awful.)
Semi-related, the actual quantisation choice you made wasn't specified. I'm guessing 4 or 5 bit? - at which point my question is around what ones you experimented with, after setting up your prompts / json handling, and whether you found much difference in accuracy between them? (I've been using mistral7b at q5, but running from RAM requires some patience.)
I'd expect a lower quantisation to still be pretty accurate for this use case, with a promise of much faster response times, given you are VRAM-constrained, yeah?
Two naive questions. First, with the 4060 Ti, are those the 16gb models? (I'm idly comparing pricing in Australia, as I've started toying with LM-Studio and lack of VRAM is, as you say, awful.)
Semi-related, the actual quantisation choice you made wasn't specified. I'm guessing 4 or 5 bit? - at which point my question is around what ones you experimented with, after setting up your prompts / json handling, and whether you found much difference in accuracy between them? (I've been using mistral7b at q5, but running from RAM requires some patience.)
I'd expect a lower quantisation to still be pretty accurate for this use case, with a promise of much faster response times, given you are VRAM-constrained, yeah?