This is super helpful. I'm building a document question-answering service over a custom data corpus (related to Saivism, a sect of Hinduism). So far the first pass has been to manually chunk the text (based on headings, chapters etc.) and then I've used OpenAI's embedding service and storing the embeddings in Pinecone. All stiched together using LangChain. To ask a question, the question is again embedded, then searched against the vector store, then the related documents are provided as context to the LLM along with the question.
So far it was really easy to set up the prototype, but the results weren't as great as I had hoped, so I'm excited to see how I could improve it.
So far it was really easy to set up the prototype, but the results weren't as great as I had hoped, so I'm excited to see how I could improve it.
Edit: wow, I didn't see this before. LangChain implements one of the featured article's suggestions (HyDE) - https://python.langchain.com/en/latest/modules/chains/index_...