Benched cold start recently: 19k LongMemEval sessions laid down in the real ~/.claude and ~/.codex layouts, 100 questions whose answer sits in exactly one session, scored by whether that session comes back (bias: I built deja, one of the six)
deja: 29s to index, 24ms query, 18/100 hit@1, 67 found@50. Plain BM25, no vectors
agentmemory: 95s import, 14 hit@1, 65 found@50, plus a worker and engine on four ports
MemPalace: ~3h mining, 2.6s query, 14 hit@1
CASS: 56m index, every NL query fails with "query fuel exhausted" on the release build (fixed on their main)
claude-mem: no-op out of the box, only records forward from install
funes: the documented 1 min first pass indexed 189 of 19k sessions (0/100); full index still embedding, ~3 sessions/s
Numbers look low because 19k sessions is brutal; on the standard 500-session LongMemEval-S the same BM25 gets ~85% hit@1
The funny thing is BM25 basically ties embeddings here at 1/100th the cost. The real cliff is reranking (found@50 67 vs hit@5 35) and staleness. Vector search has zero concept of "superseded info" only fix I found was letting explicit user corrections outrank the transcript.
Repro scripts and corpus: https://vshulcz.github.io/deja-vu/guide/day-zero.html
@skeledrew: cross-agent across 23 harnesses, but yeah, it's an index over logs, not a source of truth :)
deja: 29s to index, 24ms query, 18/100 hit@1, 67 found@50. Plain BM25, no vectors
agentmemory: 95s import, 14 hit@1, 65 found@50, plus a worker and engine on four ports
MemPalace: ~3h mining, 2.6s query, 14 hit@1
CASS: 56m index, every NL query fails with "query fuel exhausted" on the release build (fixed on their main)
claude-mem: no-op out of the box, only records forward from install
funes: the documented 1 min first pass indexed 189 of 19k sessions (0/100); full index still embedding, ~3 sessions/s
Numbers look low because 19k sessions is brutal; on the standard 500-session LongMemEval-S the same BM25 gets ~85% hit@1
The funny thing is BM25 basically ties embeddings here at 1/100th the cost. The real cliff is reranking (found@50 67 vs hit@5 35) and staleness. Vector search has zero concept of "superseded info" only fix I found was letting explicit user corrections outrank the transcript.
Repro scripts and corpus: https://vshulcz.github.io/deja-vu/guide/day-zero.html
@skeledrew: cross-agent across 23 harnesses, but yeah, it's an index over logs, not a source of truth :)