htormey
Joined 17 years ago
Karma
1,297
Submissions
386
Member since
Sep 30
Recent submissions
Comment · 2 years ago
This has been down for multiple hours now. During this time we have migrated off of fly to coolify/digitalocean.
This is the second time this happened in recent months. The last straw was one of our customers reaching out and letting us know our site was down. We had paid for two machines with fly to have redundancy.
Pretty sad as apart from the outages we really liked fly. Hopefully they fix things and learn from this experience.
Comment · 2 years ago
Pretty handy collection of prompts to do basic things with LLMs. I’ve had good results with using Claude to explain code, tag sentiment and extract emails or other specific content from free form text.
If you plan on using any of these at scale I recommend investing in a good evaluation test harness to check for regressions when you tweak prompts.
Comment · 2 years ago
I don’t trust anecdotes on twitter because every time I’ve tried an agent that’s been hyped up it’s been more expensive and time consuming than just using GitHub co pilot with Claude/ChatGPT and putting up a PR myself.
Hence I’m skeptical of people making claims about a product I can’t try out myself. It’s unclear if the tasks they are doing and the way they are using Agents is relevant to the work I do. Which is usually working on a team of engineers shipping code on a complex code base.
For AI I tend to put a lot more weight in benchmarks, such as SWE-bench, which is why I wrote an article about:
https://www.stepchange.work/blog/why-do-ai-software-engineer...
SWE-bench is mostly small python tasks evaluated solely by unit tests which require less than 15 line changes to a single file. Most of those it fails at and the ones it gets right it ignores all sorts of libraries and conventions used in the rest of the code base.
I’m Optimistic that agents will eventually agents will improve dramatically in a few years but today Devin is not good at making larger changes that build on one another like features.
Comment · 2 years ago
AI software engineers like Devin and SWE-agent are frequently compared to human software engineers. However SWE-bench, the benchmark upon which this comparison is made, only applies to Python tasks, most of which involve making single-file changes of 15 lines or less and relies solely on unit tests to evaluate their correctness. My aim is to give you a framework to assess if AI's progress against this benchmark is relevant to your organization's work.
Comment · 2 years ago
This article is about Cognition Labs Devin and what’s in the benchmarks used to assess it and AI on coding tasks.
React Native at Coinbase, based in SF, formerly Apple/Facebook.
Twitter: @htormey
For consulting/contracting enquiries email: harry dot n dot gale @gmail