Joined 6 years ago
I'm not certain if that's time/budget versus "simplifying the game for everyone".
It's easier to balance a game with 10 "sets" of armor + weapon - as opposed to having a myriad options of armor per body part and weapon combos/etc. DA:VG was the most beautiful but shallow Dragon Age game.
Nowadays if your game has even one thing wrong and the wrong influencer makes fun of it, suddenly it's trash. At the same time, your game can be trash and if an influencer laughs their ass off at the bugs (in a good way), everyone wants it. It's a sad culture. /rant
Anecdotally, in game benchmarks I've run, OpenAI models seem particularly adept at finding the meta game and exploiting it...
Outside of that, behavior is quite good, a bit slow, and definitely different than both GPTs and Claudes -- which is appreciated.
With the advent of AI, I've been _trying_ to use AI to accelerate game development and have come to a few conclusions.
1. AI models want to make everything deterministic. This sounds like a truly sound way to write a verifiable program. Games are not verifiable simulators except in the niche of ...simulators. Even in those cases, the vast infrastructure devoted to this leads to a lot of churn, bugs (AI forcing things to align with the verifier), and rigidity.
2. Making a game != Fun. This one is obvious but needs to be stated. Anyone on earth right now can say "make me doom" or "make me a super mario clone" and Opus/Astra will dutifully comply and make something generic and bland. Once the novelty of "oh wow it actually built it" wears off, you're left with a mediocre clone, lacking depth, that no customer would want hoisted upon them.
3. Content and novelty are the hard parts. AI models always create the mediocre middle. There's real work (and interesting work) involved in both inventing novel gameplay systems which LLMs will never stumble upon despite infinite 'brainstorming prompts'. But, AI is 'OK' at taking your novel idea and proposing directions -- that you can then strike down viciously until something interesting emerges.
On the content side, this has always been a truism for an independent developer. AI art tools aren't quite there yet. There's image-to-3D and there's Mixamo for animation but we're not there yet. Nearly though! But again, will every game look like a bland generic AI generated image with bland AI generated models?
4. AI models are crap at building re-usable large-scale architectures. Even building with Opus 5.5 and Astra they will happy design themselves into a corner. Yes, they can _also_ design themselves out of that corner at speed - with a lot of pain and code churn and mess. I've found it vastly better to iterate slowly, using your experience (if you have it) to structure the core architecture and systems the way you want (AI writing the code) -- and then build upon that solid base -- taking special care to reject updates that butcher your architecture. I think this applies to most coding projects actually.
5. Game dev currently surfaces very clear differences in model capabilities with Opus 5.5 leading the pack (in this domain) and Fable 5.1 and Astra/GPT-6.1 Sol following a decent distance behind. Part of this is wrong usage of the tool. If you say "make a game" -- Opus will outperform the others by a large margin -- in making a generic "game" that was undefined (a paragraph of text is not a defined game). In these cases, the model has filled in all the gaps in undefined areas with average 'what probably looks good' content. You're left with something devoid of creativity. This is the problem with the myriad three.js tech demos.
If you instead have a clear understanding of what you're building, models (nearly all SOTA) will get you there cheaper by building what you've asked without speculation. I don't believe you should hand off the one human worthwhile trait, creativity, to the AI -- so "Opus build a game" is indeed the wrong way.
6. It seems hard (and pointless) to collaborate with other human developers when AI coding on smaller/indie projects. Each AI model is trying to "solve the whole thing". The speed also makes it hard to stop at arbitrary boundaries and promotes a rushed feeling or conversely a sluggish feeling (AI did all the stuff, now I'm waiting to integrate with a friend/collaborator). This is probably easily solvable but has thus far felt weird when I've collaborated.
7. The return of custom engines? While models do great with Godot and Unreal, this is the chance to build custom engines like the golden years of the 2000s. OK, this one is a bit of nostalgia but I miss the world of dozen of game engines each with their own unique look and feel. Otherwise you get XBOX 360 era where every "realistic" character looks like a Gears of War meathead. Every adventure game has the same mechanics, you're hooking something, you're scooting under something, you're scooting between walls or things, you're on collapsing things that require either button sequences or hitting your 'hook' (every adventuring has some hook of infinite rope of course). OK, this entire bullet is just a rant about samey-ness. LLMs should unlock our human creativity and allow experimentation outside of that cost imposed same-ness.
All things said, it's _clear_ that the acceleration is here if you know what you want. You can go from concept to prototype in a few hours. You then have to be disciplined enough to step back and rebuild that thing (with AI) correctly and towards a useful objective that you've thought up. I'm looking forward to went content creation catches up. Or perhaps that's a sad day when creativity vanishes.
OpenAI needs to improve AI Safety --- OpenAI Employees have a responsibility to retain corporate secrets and do not have blanket freedom to share with 3rd parties.
Their job is twofold, they have to balance being an agent of the company they work for, with their role and responsibility for safety research.
This is the case for anyone in any company. You can "believe" that an external party needs access to something - that doesn't make it right, or allowed. As someone senior, you're expected to strike a smart balance, in-favor of the company you're working for. That doesn't mean hiding things, it does mean being thoughtful, ensure your leadership is comfortable with what you're planning to share/disclose, etc.
They work for OpenAI, not METR. It's a corporate vs academic mindset. They can believe METR needs x information to best research/audit something - that doesn't mean that is allowed/or the best option for OpenAI.
The question is whether these researchers exceeded clear, reasonable sharing boundaries or were penalized for carrying out expected safety collaboration.
That then made me realize that they lower the bars of tied scores so on the site it looks like Astra in second place. Weird. Anyway, yes, so many of these composite benchmark sites are irrelevant if they're not trimming the fat and sticking to the most up-to-date variants.
I think it's important to have a solid benchmark where you KNOW there's a difference in model performance.
I have one around 3D modeling that models really land in the same space each time I run it. It's visual, and it's super clear. Sol has perpetually generated low quality work regardless of reasoning level. Astra was the first OpenAI model to suddenly leapfrog the pack and generate content that was production ready, beating out any other provider.
I haven't seen Astra regress (yet).
I think, if you want to be consistent and scientific about it, then you'd have to use the models via API and lock to a specific version. Via the subscriptions, you are floating on whatever the latest version is, vendor to vendor.
To try to add something to this discussion -- I think that while I've seen these sort of loops less --- what I have seen is "overly helpful".
Models nowadays want to double-triple-quadruple check things. I'm being silly but it verges on "I have a working solution but let me write a variation in Rust to ensure a convergent solution and prove this works".
I've had to stop models nowadays mostly because they're being agonizingly pedantic in their validation. Opus is actually one of the most pedantic and "off track" here. But again, not in a bad way. I'm usually like "stop testing latency between 50 runs of this app... this is version one.. we're going to make a million more changes.. you're not buying us anything".
Fable is without question the larger and more thoughtful/intelligent model. It also gets out performed by Opus on many/most benchmarks. So we can say that while Fable is more intelligent, Opus is more capable. I'd still opt for Fable in nearly every case if tokens were free.
So it can be true that the "smarter" model is perhaps not the smartest in every single niche dimension that its cousins have been fine-tuned for (yet!).
Gemini is great via the Chat interface and decent via Github Copilot.
I honestly hate it via Antigravity CLI because their sandboxing system frankly doesn't work. Every other harness has mastered "don't ask me if you're working in this one directory and using common commands". Agy instead either tries to pull a global elevation or wants every tedious variation of a command string whitelisted. Madness - circa 2023.
Agy _really_ needs to make the out-of-the-box experience cleaner and hassle-free. Heck, even Grok CLI "just works".
This may reflect a global mind-shift from "approve and validate everything" to "just do the stuff and only ask permission if it's outside the folder or a command that actually requires elevation". Maybe that's not for everyone, but for those that do want to perform unattended agentic work -- Agy is painful.
