> Too much context confuses the LLM. Reduce unnecessary context and performance improves. Not to mention, too much context and you just run out of vram.
I don't think you can reduce context enough on a local LLM to get performance comparable to a 500k context without those reductions...
I don't think you can reduce context enough on a local LLM to get performance comparable to a 500k context without those reductions...