Joined 9 months ago
Even with the same model and the same input, the output is inconsistent. And what I've observed is that as the size of the input and output grows, the consistency and accuracy of the output seems to decrease. It gets more complicated when you don't control the full input, such as a chatbot with customers.
I think the problem remains even if it can be mitigated by freezing the model and the hardware, which carries the tradeoff of requiring a model you can download and run on your own so you can't use the SOTA models.
Very cool! What's been the hardest part? Have you successfully automated non-trivial communications? (for example with prospects, customers, vendors, partners, etc.)
> maybe an agent starting a business from scratch would have a better time
This describes the project I've been working on since last year, with agents in company roles, deciding on strategy, collaborating, and making progress (admittedly nonlinear). Some parts of the platform are stronger than others. So far the company agents have developed a company strategy and plans for executing it, launched a website and blog, and built and operate two products, one free and one paid.
But I would be wary of using a third party like Pion as a platform for running a business. It's one thing to know that your prompts and responses will be used to train models in the future, by companies that have a huge sea of relatively unstructured data. But it's another to hand over to another company every aspect of your strategy and operations, available to that company in real time and structured in such a way it's quick and easy to understand what you're up to and where you are going. Especially if it's being offered for free and at scale. With a free offering, value creation will likely come from either customer data or from escalating prices once customers are locked in. And even if neither of those happen, you now have a huge single point of failure for your entire organization. Seems like there are lots of strategic risks in there for the users.
I refactored the tool out of my application and it's available now at https://endpointevaluator.com . There's nothing there someone couldn't build themselves, but then you have to spend that time building and maintaining it. I think the ongoing and long term cost of maintaining all this LLM-generated software is under-appreciated. So hopefully there is still space for outsourcing tools that are generic enough to be used by many and provide good value for cost.
But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.
LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.
I am also building LLM knowledge/memory systems and I've been surprised how bad LLMs are, even SOTA models, at summarizing non-trivial input batches of text. They get things wrong, distort the underlying meaning or data, etc.
My company recently built a tool that is closer to your first category, but it's an API so it doesn't have the security (supply chain) concern of being embedded in your application.
It's built to help people manage the risk of LLMs changing underneath them and drifting from their designed behavior. Traditional deterministic testing probably won't be sufficient for apps that provide nondeterministic output, like a chatbot backed by an LLM.
The point in the linked article about the challenge of selling developer tools to developers is a good one. I think the first reaction to coding agents is "let's build everything ourselves!" but the long tail of maintenance is still there and the pendulum will probably swing back to "let's stick to our knitting."
