smnscu
Joined 10 years ago
Joined 10 years ago
- strategy document
- "sprint" document with technical implementation
- actual implementation
- e2e testing scenarios updates
Every step involves iterating with Claude on it with me in the loop (setting the direction then resolving the "founder questions" as they appear), and importantly a different model for review/code-review, be it Codex (usually, it's great at it) or Antigravity/Gemini (sometimes finds novel things, its precision and recall are abysmal but on the odd occasion it has good accuracy). This iteration on the high-level plan then on the implementation plan is essential to me, and IMHO part of why people are surprised that I tend to get solid results from LLMs. At the very least, it allows me to fill gaps in my own knowledge (primarily front-end development) and be more productive than writing the code myself. I cannot stress enough how nice it is to have a partner in the high-level system design – yes, it often suggests utterly moronic ideas, but the overall experience is still net positive and getting better every quarter.
There's still hope for a second season FWIW. At least the story is mapped up, if not more preprod stuff, it's been a while since I looked into it.
MTP - https://blog.google/innovation-and-ai/technology/developers-...
MLA - https://machinelearningmastery.com/a-gentle-introduction-to-...
CSA - https://deepseek.ai/blog/deepseek-v4-compressed-attention
I didn't set out to do this. After I got laid off in December, a client quickly fell in my lap: a small startup in the middle of a massive investment round that needed to hire 25 people immediately, with only a CTO available for interviews. I created their content and ran their interviews while building the software at the same time. It started as Google Meet + CoderPad + Calendly and gradually became an in-house system. Unlike Proton (lol), I'm not pretending I built my own video call solution from scratch, it's just an off-the-shelf 100ms integration.
The content is all versioned and structured, which makes it fast to iterate on and easy to reason about. We use major.minor versions and only bump the major for backwards-incompatible changes, or changes big enough that comparing interviews stops making sense. Otherwise, any combination of question versions inside an interview format is considered comparable if the major versions are identical.
The interview itself is highly structured: once you define a format from the content library and the various knobs you can adjust, you can schedule interviews and run them using our integrated "room" (video call + multiplayer code editor, both recorded, with transcripts and playback) and "rubric" (the tool the interviewer uses for content, scoring, and notes during the interview). Once you submit/publish the interview, a report is generated immediately. Example: https://coderba.se/sample
Two interesting AI bits:
- "AI linting": a way to benchmark interview questions by running a candidate model and an interviewer model against each other. The candidate closely follows a defined skills profile, then we compare actual vs expected performance. More here: https://coderba.se/blog/product-update-unit-testing-the-inte...
- "AI draft": once an interview ends, it takes ~30s for the video and transcript to become available. Then we use basically every relevant artifact from the interview, with a PII redaction pass first: questions, scoring, incomplete rubric, transcript, code editor history. We send that through our LLM gateway, currently mostly using DeepSeek because the quality/value is insane, though I may switch to Mistral to stay on the better side of privacy. It sends back recommended scoring + writeup, which we present as Cursor-like suggestions you can accept/reject/edit.
I've spent the last 7 years conducting over 3,000 technical interviews. Along the way, I made the (utterly insightful) observation that most hiring processes are broken, often relying on vibes and ad-hoc trivia. Coderbase is in large part my attempt at treating hiring and interviewing as a rigorous systems engineering problem rather than a simple outsourcing of the thick side of the hiring funnel.
I'm essentially attempting to distill thousands of hours of interview data into sensible processes that can lead to meaningful improvements to hiring. The platform underneath it all is early, but we're getting close to being able to scale the system to e.g. 1000 interviews per week without (hopefully) breaking a sweat.
I plan to publish these weekly, as working on this technical challenge is exhilarating (I already have enough material for a second post but it's got to wait until next week). For now, let me know what you think of my attempt to weaponise AI slop (aka "synthetic interview benchmarking").
hit me up at google's email service, same username