andyk

> bingeineering

i'm gonna start using this

because it seems like the human mind does this and it was easy to start with. The human mind obviously also compresses by other dimensions so we'll keep experimenting

suggestions for what to try next?

fair point. the headlong agent running on my mac mini, cleo, doesn't pause between calls to "generate next thought". for a while the implementation didn't support pausing between thoughts. a `thinker`, which is where the thinking loop lives, only had a boolean variable `self_trigger`, which caused it to recursively fire itself to run again (ie immediate generate the next thought). We added the ability for it to slow down its thinking and prep for sharing it with the world since it makes it easier to leave it running and not have it spend a few hundred dollars a day and you can still observe the same mechanism, which is the heart of what we want to get at here:

I don't think the blog post fully conveys this, but for me it's not whether there's a pause between thought generations, but instead whether the agent is designed to stop after some human-set goal is achieved to wait for more instructions

In contrast the idea here was to force the agent to pick its own next goal (even when it receives no explicit tasks from a human) -- even if that goal is implicit -- and then watch and see what it picks (or at least what it does with its time). and then figure out what sort of mechanisms we can introduce to the architecture at that outermost life/goal picking loop

Should maybe have said more loudly in the posts that this is not intended to be disciplined engineering (or used in prod!) we did think about the architecture quite a bit and we've tried a bunch of different things out. We are fully optimizing for velocity of research experimentation and iterations on the design vs engineering hygiene

i've been playing with the project and idea for years now. the codebase started as javascript plus supabase before reasoning models were a thing (i incorrectly predicted the world would call reasoning models Large Thinking Models - bahaha). at one point the focus was almost entirely on a really easy-to-use human prosemirror UI for editing the agent's thought stream--because the models were still so bad at reasoning at that time.

then we pivoted to all bash at some point since it simplified things and i have this hypothesis that the LLMs will do better if as much of its world as possible is command line operations (inspired by terminus the agent we built as part of the terminal bench project)

the bash implementation is entirely written by coding agents - the human time goes into iterating with an AI agent on the design docs in the ./design folder

re security model - i think of the project as a research experiment about what it's like to treat an agent more like a person - so for secrets you have to trust its judgement about what it will share or if it will betray your trust. I don't know the OpenClaw architecture in great depth but my sense is that none of the LLM based agents that can interact with multiple people have strong guards in place that prevent the agent from leaking secrets between people. Definitely an interesting area for research though!

so yeah security is not a priority in this research per se - easier to assume you don't share sensitive things with a headlong agent

Along those lines, because we keeping it research and optimizing the velocity with which we can try new things out, the current aim isn't necessarily to get a bunch of community contributions. but if we do start to get any i'm thinking we will actually require people to only create PRs with design docs (no code allowed) and maybe even also require the logs (or at least the human prompts and the agent's responses) from the coding session where they created the design doc.

maybe i should write up the history, philosophy, and big picture aims of the project at some point

andy here (headlong post author). terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them. w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho

andy here - happy to answer questions

I hope the competition will inspire people to make breakthroughs in the open, so I won't take any rights to the IP, instead the winning solutions must use open source code and open weight models.

yes the prize money is from me to the winners

That has a double meaning - half tongue in cheek.

1) since we are creating a contamination-free version of SWE-bench (i.e. scraping a new test set after submissions are frozen) it is guaranteed that agents in this contest can't "cheat", i.e., models can't have trained on the benchmark / agents cant memorize answers.

2) as a general rule in life, don't cheat on things (not that there aren't exceptions)

(reposting from locallama and lower down here) yep that's true.

one of my goals is to inspire and honor those that work on open source AI. Those people tend to be motivated by things like impact and the excitement of being part of something big. i know that's how i always feel when i'm around Berkeley and get to meet or work with OG BSD hackers or the people who helped invent core internet protocols.

those people are doing this kind of OSS work and sharing it with the world anyway, without any cash prize. i think of this as a sort of thank you gift for them. and also a way to maybe convince a few people to explore that path who might not have otherwise.

yeah i agree. one of my goals is to inspire and honor those that work on open source AI.

people who work on open source tend to be motivated by things like impact and the excitement of being part of something bigger than themselves - at least that's how i always feel when i'm around Berkeley and get to meet or work with OG BSD hackers and people who helped invent core internet protocols or the guys who invented RISC or more recently RISC-V

those people are going to do this kind of OSS work and share it with the world anyway, without any cash prize. i think of this as a sort of thank you gift for them. and also a way to maybe convince a few people to explore that path who might not have otherwise.