A1aM0
Joined 10 months ago
Joined 10 months ago
The thing I’m testing is not “new tests,” but a tighter system: derive the rubric from real team tasks, apply the same rubric to both public evidence and live assessments, and make every score traceable (with lower weight on easy-to-fake AI-era signals).
If that doesn’t improve consistency/speed/quality vs current hiring loops, then it’s just old process with new branding.
You’re right that many strong engineers can’t legally share employer code, and “has OSS time” is not a universal signal. So I’m now thinking of this as a dual-path system:
1) Public evidence path (for people with OSS/public technical work), where existing contributions are treated as reusable evidence. 2) Structured assessment path (for people without public artifacts), using scoped tasks/pair debugging/incident reasoning mapped to the same rubric.
So OSS should be an advantage when present, but never a requirement.
Also agree on AI confounders: raw public activity can’t be trusted at face value anymore. We need to weight traceable process signals (review back-and-forth, bug-to-fix chain, consistency over time) higher than easy-to-generate text/code volume.
If you were hiring with this, what would be your minimum bar for “credible evidence”?
However, my hypothesis is that matching on 'Problem Vectors' might actually help break the cycle of 'only getting hired for what you've already done.'
Transferable Complexity: A traditional recruiter sees 'Game Dev' and ignores them for a Fintech role. But a vector model might see that the candidate solved a 'distributed concurrency' problem in a game that is mathematically similar to the 'payment sync' issue in the JD. It matches on capability, not just domain keywords. Signal for Juniors: Currently, ATS filters reject Juniors based on '0 years experience.' If a Junior has tackled a complex logic problem in a hobby project or Hackathon, this system highlights that specific signal. It gives them a fighting chance based on code reality rather than resume keywords. That said, I agree this model naturally leans towards Senior/Specialist roles where specific technical gaps need immediate filling. It's not a silver bullet for 'hiring for potential,' but I hope it's a step up from the current keyword-soup approach.
When you expose a database via a protocol designed for 'context', you aren't just exposing data; you're exposing the schema's complexity to an entity that handles ambiguity poorly. It feels like we're just reinventing SQL injection, but this time the injection comes from the system's own hallucinations rather than a malicious user.