domrdy

It has Sonnet 5 as a usable model. Interesting.

For sure true for specialized ones like MedGemma (healthcare). In my testing, the 27b model is at least on the same level as frontier, and in some cases outperforms them. 4B is insanely good too for some lighter workloads. Thanks G for working on this!

Congrats Andrej. Let me know if you are looking for someone to take over Eureka Labs. We were in the same WoW classic guild. domrdy on x. We can duel for it :). Also, I need $10M if any VCs are reading this.

Wow, this is fantastic. Keep it up, Banaxi-Tech, so much more to learn :) Clearly great parenting, too.

Agreed, the way it is implemented is that candidates do the recording after they submit their work.

I agree with you. I’ve failed many live-coding interviews because I start focusing on how the interviewer perceives me instead of the task at hand haha. Hiring managers still want to hear the candidate talk through a technical problem. I built a way for candidates to screen-record their take-home solution using only their microphone and screen share, with as many takes as they want. This lets hiring managers hear how candidates communicate and explain their technical implementations, a skill that matters on teams. But it means managers have to watch those videos. Many of our customers use it and candidates like it, but it still takes extra effort on the candidate’s and hiring managers part, with no guarantee that it will "pay off".

I think certificates are interesting in theory, and there’s an entire industry built around them. AWS does a solid job with its certifications: if you earned one five years ago, it’s still more or less relevant. I'm not sure how certifications would work for proprietary technology.

Labor laws in many places are less “flexible” than those in the US, and they don’t support what you’re proposing, for good reason. I wouldn’t quit my job or uproot my family just to convince a manager I’m worth keeping.

Thank you. I'm not a native speaker, what I was trying to say was, that it depends on the "use-case".

As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia.

I’ve worked in the assessment space for 6 years and have seen many hiring processes, from Fortune 10 companies to startups hiring their first engineers. The range of signal required, "how much time can my engineering team spend with a candidate", and how much candidate experience you can get away with, is huge. I’ve also been a candidate myself and failed many live coding interviews. It made me feel terrible about myself. The last time for a role at Ycombinator (the interviewer was super nice).

When I work on my product, I try to view it through the lens of empowering candidates to show their skills and potential. I encourage our customers to use assessments that somewhat resemble on-the-job skills. I don’t like the phrasing “real work” anymore. An assessment shouldn’t be unpaid labor, it should be a way for candidates to demonstrate that they can do the job and handle future work thrown at them, and for hiring managers to feel confident extending what are often very high salaries in tech.

With AI, unfortunately, short take-homes (what I prefer as a candidate, using my own tools and editor) are becoming harder to maintain as a fair signal due to AI assistance. I’ve seen companies move back to onsite, and competitors deploy all kinds of proctoring and invasive monitoring.

The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, with their own editor and configuration, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve. I think about it daily and have not come up with a solution.