Ask HN: Storing a large amount of emails fast in a Vector DB
Hi HN,
I am working on a personal project where I'm trying to create an email GUI that I can use for myself. For this, I want to store all my emails in a pinecone index. I want to figure out what's the best way to embed and store all the emails extremely fast - I tried threading in Python and batching but not getting great results. Let's say I have 10k emails I want to store - what's a way I can get these done really fast? I'm new to optimization and pinecone so I apologize if this question is easily answerable. TIA!
I am working on a personal project where I'm trying to create an email GUI that I can use for myself. For this, I want to store all my emails in a pinecone index. I want to figure out what's the best way to embed and store all the emails extremely fast - I tried threading in Python and batching but not getting great results. Let's say I have 10k emails I want to store - what's a way I can get these done really fast? I'm new to optimization and pinecone so I apologize if this question is easily answerable. TIA!