mfkhalil
Joined 3 years ago
Karma
148
Submissions
85
Member since
May 24
[email protected]
View on Hacker NewsRecent submissions
Comment · 5 days ago
we still browse issues to check for anything critical but this is really useful for uncovering patterns / symptoms of the same root cause.
also it gets issue body + comment/reaction metadata so it can generally intuit whether something is important or not
Comment · 1 month ago
It’s a mix. On benchmarks, yeah, we route a request, then run the same request on models within a range of capability and compare the results.
We also have our entire team using it for their day-to-day work, and they’re devs who are pumping out tens of PRs a day, so they’re a pretty unforgiving test group and are always giving feedback.
We also work with companies that have it either running on real traffic or running shadow evals in the background (which do a version of the +/-1 method you’re describing), and they’re a huge source of feedback.
At the end of the day, most of our focus is on how it performs on production workloads. As useful as benchmarks can be for initial testing, I think it’s dangerous to over-optimize for those kinds of tasks (and most frontier models have already learned how to solve those problems optimally during training, so results can be misleading there).
Comment · 1 month ago
Hey, I'm on the team at LiteLLM that's building the auto-router and our goal right now is to abstract that decision making away from the end user. The biggest thing we're trying to figure out right now is how do we do that without frustrating the end user - as a developer myself I would hate for my agent to be dumbed down below the threshold needed to complete a task.
In theory though, there is a minimum viable model for any given task, and we think that is a problem that the big labs will avoid because they profit from charging more per task. We're trying heuristic and LLM-based approaches but it's still a work in progress, so if this is something you'd be interested in trying would highly recommend trying ours out -- any and all feedback at this point is extremely valuable to us.
Comment · 3 months ago
Very salient point. My take on this (and I know it’s not very popular on HN) is that trying to preserve every pre-AI coding skill is probably a fruitless endeavor.
AI-assisted coding is here and it’s not going anywhere. In the same way most modern software engineers aren’t writing assembly anymore, I don’t think future engineers will need to understand exactly how every part of a codebase works under the hood. It’s just another layer of abstraction.
Having said that, I do think knowledge of architecture/performance/security remains a pretty large part of prompting correctly, although more conceptually than actual implementation. As long as you’re still using those concepts in your prompting, you can keep that judgment sharp, even if some of the implementation fluency fades.
For example, with the boxing analogy, yes, you lose some boxing skills, but you’re also becoming a way better boxing coach by consistently teaching it. I think that’s the tradeoff.