Comment by darkteflon

Context constraint is a cheap way to keep the model on-topic. So rather than relying on an ever-growing context window to stuff/mapreduce more undifferentiated “context” (the entire chat history), interposing a vector search engine that only returns relevant context tends to get you better overall model performance, in addition to being scalable in a way that increasing context window size is not.