technoplato

> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: [email protected]

Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".

In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.

Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:

L1: Show a number on the screen L2: Allow a user to click plus or minus and the number responds accordingly L3: Show that number on any device running your software, and keep it in sync with all other devices L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished. L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa. L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with. L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application. L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well) ... ... ... And so on and so on.

Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.

I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.

I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.

Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.

Best, Michael

I deeply empathize with the goals of bridging the gap between the incredibly vocal minority opinion holders at both ends of the curve. I’ve been thinking about how to solve this problem and working on my own take, knophy [1], for a couple years now in my free time. Knophy is a content aggregation platform that creates an innovative new economic model whereby voting power is earned by first reading, watching, or listening to the content on which one desires to comment or “upvote” themselves. After consuming a piece of content, a user can then answer community moderated evidence-based questions to earn said voting power. Whenever questions are answered correctly, the creator, moderators of the question bank (the community) and the answerer all earn relative amounts of points with which to vote on content. Should a user develop a surplus of points, they may “burn” those points in exchange for a fungible, cryptocurrency token that they can either sell or speculate against.

This model has implications in solving the problem of content monetization, copyright, siloed walled gardens who rely on riling up their users to increase usage, moderation, and many more major problems which I would love to discuss in further detail if you find them intriguing.

I realize I haven’t done a great job introducing myself in the above paragraphs, but feel that the ideas we have in regards to solving this problem are more important than me blabbing about myself. I’m a full stack engineer specializing in React / React Native and am very excited about using the GRANDStack (GraphQL, React, Apollo, Neo4j Database) to create highly connected and innovative new applications at scale.

I'm currently very happily employed but would love to help innovate in this space in any capacity that would be welcome!