roampal
Joined 11 months ago
Karma
10
Submissions
10
Member since
Dec 1
Recent submissions
Comment · 10 months ago
You're right, it is a form of tagging technically. The difference is you're already saying "thanks that worked" or "nah that's wrong" anyway. No extra step, it just listens.
Comment · 10 months ago
Ran a 4-way comparison test across 200 query-memory pairs:
- Baseline RAG (embedding similarity only): 10%
- RAG + reranker: 20%
- Outcomes only (no reranker): 60%
- RAG + outcome scoring (mature memories with 20+ uses): 60%
"Accuracy" = correct memory ranked #1 for the query. The outcome scoring uses Wilson score lower bound - memories that consistently get positive feedback from the "user" get boosted, ones that fail get demoted.
Test methodology: https://github.com/roampal-ai/roampal/blob/main/dev/benchmar...
Comment · 10 months ago
Fair point, the install instructions at the end were meant as a "here's how to try it if interested" but I can see how it reads as pushy. The core of the post is about the outcome scoring approach itself. Should've led with more depth on the methodology. Thanks for the feedback.
Comment · 11 months ago
Absolutely, go for it!
Run whatever benchmarks you have. I would love to see how it stacks up against RL methods.
Ping me if you need help with anything.
Thanks!
Comment · 11 months ago
Dude, you literally wrote the exact motivation paragraph for Roampal right around the same time I posted this
Thorndike's Law of Effect is the entire reason I built the outcome-scoring (+0.2 for worked, −0.3 for failed) and shift weighting toward proven memories. You're not half-baked — you're 100% right. I just happened to ship the PoC first.
Would love to hear your take on the cold-start problem and whether those reward magnitudes feel right in practice. Shooting you a connection request on LinkedIn if you want to swap notes.