derekcheng08

They have a public dataset of 6000, real world coding agent sessions. About 80% of them are Claude Code. Lots of cool analyses to be done on these.

I built Prism this morning. It runs a tiny 500M parameter LLM in your browser on a given piece of text and visualizes the entropy of the probability distribution computed for each token -- effectively how confident the model is in predicting each token.

I've been wanting to build this for a while. I had a crude version of this when I first started working with LLMs and it really helped with my intuition of how the model worked. When you run it on a block of code, you'll see that the model is unsure when it needs to pick an identifier or start a new line. It's a fascinating glimpse into how models operate -- what is "easy" for them and what is uncertain.

After analyzing thousands of agent trajectories, we’ve seen all sorts of bizarre LLM quirks, this being one of them. Would love to hear other war stories also!

Pretty cool! Always struck me as odd that the IDE has largely remained single-player for 40 years.

That was an insane time. The pace was unreal. I remember Netscape 2.0 had at least 6-7 beta releases prior to the full release. And each one just dropped something massive and fundamental to the Internet - JavaScript (then called LiveScript IIRC) being one of those things. Just casually dropping what would dominate the entire industry in a browser beta.

The only other period I have experienced that comes close is what is happening now. What an incredible time to build.

Super interesting how this arc has played out for Microsoft. They went from having this massive advantage in being an early OpenAI partner with early access to their models to largely losing the consumer AI space: Copilot is almost never mentioned in the same breath as Claude and ChatGPT. Though I guess their huge stake in OpenAI will still pay out massively from a valuation perspective.

The biggest omission that immediately stands out to me is: "provides a clear sense of direction".

I've seen so many examples of teams and organizations that experience a lack of clarity, with all sorts of negative downstream consequences - muddled strategies, moving goalposts, fatigue/low morale. Having a leader that can provide that clarity is so important.

I do really like Tonkotsu :)

But beyond that, we wanted a playful, accessible brand. We think dev tools (particularly ones like Tonkotsu) are consumer products and we didn't want the staid/corporate branding many tools have.

Forgot to mention that an interesting behavior we see emerging is “human as editor”: let the agents make several commits in a branch, and then the human does a single refinement pass over it before raising a PR. Curious if others use this workflow or something else?