bojo

I love this anecdote. It highlights what our industry continues to forget: The end user doesn't care.

Don't get me wrong, tech is why I am here. But if it works, Alice and Bob don't care one bit about how the product exists.

While neat, I don't get consistent scores if I retry it a few times. If it leads with a series of greens first, my score is more green oriented, and vice versa.

This is what I have been doing. I don't think I've made any amazing breakthroughs, but at the same time I can't help but feel like I've come across some white paper-worthy realizations. Being able to correlate across a lot of domains I feel like I intuitively understand but have no depth of knowledge has been a fun exercise in LLM experimentation.

That's a solid point.

I knew my flake setup could be better but never bothered. Then one day earlier this year I threw Claude at it. Not only did it improve everything, it fixed a small bug that had been bothering me.

My confidence in doing this came from exactly what you said: If it blows everything up I can just rollback.

Perhaps? I just used it to analyze one of my 96k Zig codebases using Claude Code and here is (part of) what came back. (I snipped out the deeper analysis above as it exposes my private project - but it was all correct).

  Head-to-Head

  ┌──────────────┬─────────┬─────────────┬────────────┐
  │    Metric    │  Opty   │ Traditional │   Ratio    │
  ├──────────────┼─────────┼─────────────┼────────────┤
  │ Input tokens │ ~13,500 │ ~39,408     │ 2.9x fewer │
  ├──────────────┼─────────┼─────────────┼────────────┤
  │ Tool calls   │ 21      │ 61          │ 2.9x fewer │
  ├──────────────┼─────────┼─────────────┼────────────┤
  │ Round trips  │ 5       │ 9           │ 1.8x fewer │
  └──────────────┴─────────┴─────────────┴────────────┘

I had it run a separate analysis using traditional vs. opty and count the actual tool calls and input token counts. My prompt was basically, "do a full analysis of this entire codebase."

I appreciate the feedback. I have been talking to a lot of people about the very same point you make and had a lot of good brainstorming sessions.

I introduced functionality to do it once per week, or specific days, and not just every day to partially alleviate the process if it gets tedious.

More importantly, I see two things:

1. If people have a constant score over time, that should lead to a discussion. I'm not sure what, but it aligns with the goal - get them talking to each other and asking if everything is truly ok

2. If a manager doesn't invest themselves in the process, then yes, it just turns into a "keep the lines aligned" game. I have no fix for this, but those people probably weren't the target of this product anyways

I keep going back and forth on it. In certain lights it genuinely seems useful. In others, hard to say.

It was fun to build. I'll keep tinkering with it for now and see where it ends up later this year.

Sorry to be that guy. I love nix, I want to read more about how people use nix.

The font presented under NixOS + Firefox is near unreadable on this website.

What kind of costs are you thinking?