pocketarc

It does; I play games backed by Easy Anti Cheat on a VM Windows running on Nix. It does require fiddling with the settings to make it look like you’re not in a VM, but completely doable.

Your game example is a great one: If a game didn’t come with its 3D models, sound, level files, game data, or assets of any kind, but its engine was open source, I don’t think anyone would say “hey this game is open source”. They’d say the engine is, but that’s it. All of the above things are not code, but they are part of a game and yes, expected to be included in an open source game.

This absolutely should be a thing, but it'll have to be a per-community thing, them building their own dataset and creating their own evals (similar to how people do for production workloads).

Although:

"(idiomatic, etc.)" I don't think that should be part of the aim (or it should be under-weighted), because... you can just provide guidance on how to do things more idiomatically, rather than depend on that knowledge already being encoded in the model. I'd be more curious about verifying that it can work through gnarly bugs / features in an Elixir codebase. After all, what use is a model that by default does everything idiomatically if it can't figure out some small concurrency bug.

You’re running 48GB now but imagine a future where everyone has 512GB RAM or 1TB RAM in their computers (it might sound like a lot but also 20 years ago we had 512MB PCs).

It’s not hard to imagine what 5-10 years of pressure to increase RAM will do to specs, on top of the normal tech improvements.

That’s worth bearing in mind when thinking about local models.

Plus, local models keep getting better and better; 2 years ago what you could get out of those 48GB of RAM was embarrassing compared to what’s doable today.

We’re getting there. Just takes time.

In the olden days, I enjoyed Opus 3 because it was easy to have it sound way more human than GPT.

Nowadays, with the focus on agentic use and coding, it seems models have all been RLHF’d to death, it’s so incredibly hard to have them write in a different voice than their default. I put together a skill to review its writing and have it edit its own output (e.g. code comments), which does make a difference, but isn’t perfect.

What, if anything, do people do for writing? That feels like a neglected side of LLMs. They’ll make 100 Bash calls referencing ancient commands without batting an eye but heaven forbid they use something other than “load-bearing” while talking. For something trained on “all the human knowledge” it’s incredible how limited their default vocabulary seems to be.

When people imagined AI/AGI, they imagined something that can reason like we can, except at the speed of a computer, which we always envisioned would lead to the singularity. In a short period of time, AI would be so far ahead of us and our existing ideas, that the world would become unrecognizable.

That's not what's happening here, and it's worth remembering: A caveman from 200K years ago would have been just as intelligent as any of us here today, despite not having language or technology, or any knowledge.

In Carolyn Porco's words: "These beings, with soaring imagination, eventually flung themselves and their machines into interplanetary space."

When you think of it that way, it should be obvious that LLMs are not AGI. And that's OK! They're a remarkable piece of technology anyway! It turns out that LLMs are actually good enough for a lot of use cases that would otherwise have required human intelligence.

And I echo ArekDymalski's sentiment that it's good to have benchmarks to structure the discussions around the "intelligence level" of LLMs. That _is_ useful, and the more progress we make, the better. But we're not on the way to AGI.

Yeah like I remember prior to reasoning models, their guidance was to use <think> tags to give models space for reasoning prior to an answer (incidentally, also the reason I didn't quite understand the fuss with reasoning models at first). It's always been XML with Anthropic.

I think I've read the reference a hundred times throughout my time on HN, but had _never_ actually seen it, thank you for the link!

I'm a dev, not a salesperson, but let's be realistic. A company tells you "yeah we're interested in signing at $1M/yr, but we really need this feature, when will you have it by?", to which saying "eh we don't know - it'll be done when it's done" will lead to the company saying "ok well reach out when you have it, we can talk again then" (or just "eh ok then not a good fit sorry bye"), and in the meantime they'll go shopping around and may end up signing with someone else.

Having a promised date lets you keep the opportunity going and in some cases can even let you sign them there and then - you sign them under the condition that feature X will be in the app by date Y. That's waaaay better for business, even if it's tougher for engineers.

I use this approach for a ticket based customer support agent. There are a bunch of boolean checks that the LLM must pass before its response is allowed through. Some are hard fails, others, like you brought up, are just a weighted ding to the response's final score.

Failures are fed back to the LLM so it can regenerate taking that feedback into account. People are much happier with it than I could have imagined, though it's definitely not cheap (but the cost difference is very OK for the tradeoff).

I agree - and it's not just what gets you promoted, but also what gets you hired, and what people look for in general.

You're looking for your first DevOps person, so you want someone who has experience doing DevOps. They'll tell you about all the fancy frameworks and tooling they've used to do Serious Business™, and you'll be impressed and hire them. They'll then proceed to do exactly that for your company, and you'll feel good because you feel it sets you up for the future.

Nobody's against it. So you end up in that situation, which even a basic home desktop would be more than capable of handling.