LoCoMo has an open world question category. Which is non factual/subjective retrieval (96 questions). Like "Based on everything you know about Maria, what birthday gift would she appreciate?"
These are challenging as the retrievers need understanding not just keyword matching or vector similarity.
Although subset of questions is less but a lot of systems fail at this.
These are challenging as the retrievers need understanding not just keyword matching or vector similarity.
Although subset of questions is less but a lot of systems fail at this.