Summarization is much more expensive than vector db's. Assume you have 1m tokens of context. You could run all through GPT-4 and summarize the information, but it would cost $60 (based on current prices) and take 10's of minutes of GPU time to do the inference.
Disclaimer: I work for a16z and on the infra team, so consider me biassed.
Disclaimer: I work for a16z and on the infra team, so consider me biassed.