pjm331

great writer, starts forgetting instructions around 80k context in my experience

markdown specs are certainly the funniest way to claim to distribute software

I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself

so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive

who out there is running evals of giving agents these kinds of tools and how are you doing it?

I’m guessing they’re referring to jamal khashoggi

Yes I have just started dabbling in some harness evals where the task is translating the RealWorld “mother of all demos” app, maybe one of the most translated apps of all time, and they cannot reliably 100% it

I don’t write the same program over and over again so I am always confused by how exactly this industrialization metaphor is supposed to map on to software development

I appreciated this post, had never heard of codemode and I think the name is bad but it makes perfect sense

I have wanted something like this ever since I hooked up my first database MCP

Originally I had thought to set up a python runtime with like a standard data science toolkit and the db mcps available as functions, and maybe I still will but JS has been working fine for now

I’ve been slow blinking at every cat I meet for years, I swear by it haha

callback urls are all localhost, there’s no domain to whitelist, just client names

Another thing they do that I find equally frustrating is their MCP can do things you cannot do via API so you are forced to use theirs and cannot implement your own

this is how i've been solving the problem before some of these newer techniques around deferred loading etc. came out

a subagent per mcp toolset, basically sub agent per saas in the configuration at $WORK

I don't have any hard evals on this but it feels like it works more than it doesn't, especially now that subagent use has gotten better in general, but you do feel drawbacks if you ever need to do something that involves coordinating multiple sub agents, there's some communication overhead that wouldn't be there if it was just one agent with all the tools to do the whole job

yes codemode is when you want to utilize this ability AND you want MCP tool calls as part of your scripts

codemode lets you execute scripts in a runtime where your MCP tools are made available as function calls

this matters for cases where the MCP tool is the only way to do something and you do not have an equivalent CLI, API, whatever to script with