notahan

These tests are interesting but I can't help feeling that these are benchmaxxing results. "We made our harness make sqlite in rust" sounds like a much easier probelm statement than having to recreate an application like facebook. I think the concept of having to integrate is the real challenge. Re-writing code in new languages based on decent docs is a difficult but less useful measure of how AI can help replace engineers.

Imagine this: you need to build a new CRUD app but your AI agent spends 20M toks on just writing it's own custom login function, that's a waste of time and money. I would not like to have a new system developed for me in such cases. I think we need a test where the agents are more specifically tasked to find technologies worth integrating and hence write the fewest lines of code.

Isn't that the core fundamental of AI? Constrain the results to a latent space which has all the probable answers and then pick up whichever token makes sense based on your random sampling?

I feel like the whole concept of harness and LLMs IS to pick random tokens which make sense. Think about this as a solution space (a very complex one), then the harness and the LLM are moving towards the local minima at which the code is complete. This is akin to how we think about AI anyways.

This is so cool, what are the system requirements needed to run this?

The tab model was a lot of fun, you felt like you're getting a speed boost while coding. I think vibe coding (or agentic engineering) is a different paradigm altogether.

I have tried out some of the popular tools and I'm using opencode on desktop and I use pi via termux on android for when I'm on the go. I think the current direction of PRD -> review -> execute -> debug is in many cases the right mindset.

Working with a team of fresh graduates, I see that working with any vibe coding tool is like being a manager, not a developer. I think that's what you miss, you miss being a developer but the vibe coding tools make you a manager which isn't something that you might enjoy.

Nonetheless, I do think that there are some interesting things to do with pi. I'm just getting started, if anyone has an interesting workflow in pi, I would be interested in trying it out!

Oh god, you can imagine how governments are going to look at this and freak out about espionage. Or worse, use this tech for espionage!!