Tenemo
Joined 8 years ago
Joined 8 years ago
I'm not saying corporations execute on this flawlessly, just any time when I wonder what I would've done myself, I end up with a similar structure (maybe a bit flatter)...
With LLMs the problem is more complex, it's people getting used to how a model works and to the ecosystem. Sure, you can make all your skills harness-agnostic and deal with Anthropic's stubborn refusal to adopt the common naming/directory structure. But most people don't. So then you end up with something closer to the ancient Android vs iOS discussion. Can you prove, in isolation, that iOS is more energy efficient, the hardware is faster? Yeah. But that won't speak to someone who has been on Android for 10 years and would have to migrate and get used to iOS to experience that, first.
I've noticed myself how I get used to common failure modes of particular models in my projects. GPT5.5 tends to create some checks/booleans I don't need, it heavily overcorrects on error handling, etc. While Claude 4.7/4.8 doesn't do those as often but gets derailed on our E2E test suite, forgets to run linting despite guidance. So even assuming fully harness-agnostic working setup, a new LLM model with its own quirks can be a lot friction for heavy users who might be used to Claude specifically and all their skills/guidance pre-address common failure modes.
E.g. I might be a Prius owner, then you gift me an objectively better, more efficient, safer, newer, same-size, physical knobs car ...and I might still swear by my Prius! I'm used to how it turns, how it feels, I can repair some issues myself. Isn't that a normal reaction then?
Couldn't imagine using YouTube on my phone without it, it's night and day difference – that's despite being a premium subscriber anyway.
Is there anything wrong here? Not sure I understood your comment
- Opus 4.6 (Extended thinking): "Drive it! The whole point is to get the car to the car wash — you can't wash it if it's still in your driveway."
- Gemini Pro Deep Think: "You should definitely drive. Even though 50 meters is a very short distance, if you walk, your car will stay where it is—and it's pretty hard to use a car wash if you don't bring your car with you!"
- ChatGPT 5.2 Pro (Extended thinking): "You’ll need to drive the car—otherwise your car stays where it is and won’t get washed. That said, since it’s only ~50 m, the most sensible way to do it is often: 1. Walk over first (30–60 seconds) to check if it’s open, see the queue, confirm payment/how it works. 2. Then drive the car over only when you’re ready to pull into a bay/line."
A pretty reasonable answer by ChatGPT, althought it did take 2min4s to answer, compared to a few seconds by the other two models.
We make 10+ gameplay changes a day in our custom kitchensink mod that steers everything. 70%+ are bugfixes (we have plenty of mods that conflict/are buggy), the the rest are QoL improvements, small new features and rebalancing. If we see some playstyle is too strong or boring, we spice it up the same or the next day. Mobs too weak/strong? Update the same day. A ton of performance fixes with actual in-game benchmarking. All this while I keep playing.
I built sort of an automatic modding pipeline (including public release and all) with parallel test game clients + servers used by LLMs to develop and test the changes and then merge before releasing a new mod version. We have had very few issues, I'm still shocked how reliable this is. They follow guidance well and test everything in real game clients on wipeable clones of our live world and our actual characters. Josh has a bugged chest at his place? The LLM will reproduce it, develop a patch, yes, but then also log in as him to his actual in-game base and test the fix using game inputs + OCR. At the end I get a .html page report on each change with screenshots and technical details if I ask for them. I don't care if it takes 4h to do a simple fix because it's not my time and game stability is critical. And up to 8 of them can work in parallel, so there isn't a bottleneck caused by excessive testing/re-testing.
I think this will be the future of gaming for many sandbox/easily moddable games. It's a bit too technical at the moment for regular players to handle the updates and models' quirks, but we're only going up from here. Just crazy.