Agreed it could be faster for uncached queries. The embeddings retrieval itself is actually pretty fast (uses pgvector). However, I found that having a LLM rerank results + generate summaries related to the search query made results more useful, which is what accounts for much of the latency.
Maybe I should make that a user-customizable setting!
Agreed it could be faster for uncached queries. The embeddings retrieval itself is actually pretty fast (uses pgvector). However, I found that having a LLM rerank results + generate summaries related to the search query made results more useful, which is what accounts for much of the latency.
Maybe I should make that a user-customizable setting!