orbital-decay

"We patched every vulnerability we knew". Meanwhile, prompt-injecting Argon (or any other model) is still trivial, and Carbon won't be an exception. Because the attacker is not limited to things Google has evaluated.

AI shops are blind to anything that's not a metric.

Comment · 3 days ago

If you want an OS to handle edge cases like booting from the network (that can be arbitrarily complex), reusing an existing one makes a lot more sense than writing a custom one from scratch. Make a very basic bootloader that will init the minimal amount of hardware and then load a user-supplied boot image from the flash. That's it. Linux typically boots this way on non-embedded systems.

Comment · 3 days ago

As I implied with "proof of concept", the purpose was to see if it works at all, not to make a game. It made mistakes and autonomously fixed them. Just a few months before, agents weren't able to do this. Now they're able to reach the goals you set, as long as they're definite, you're not making immovable objects collide with unstoppable forces, and the search space is not very diverse.

Comment · 3 days ago

I mean, 8 months ago Anthropic made Claude build a C compiler autonomously, and the crowd came to the autonomous repo and threw a shitposting party making fun of the build failing for some time (it fixed itself in a few days but nobody cared anymore, partying was the entire point). Half a year later we had people porting huge million-LoC projects to different platforms, they run and literally have bug-to-bug parity. It's tedious work that doesn't need deep thinking but has clearly defined goals and can be iterated upon, so it's a good fit for current AI. If your expectation of vibecoded software is it doesn't actually work (as you said above) then you really need to update it. Internals might be a horror, but it does work and it's not broken. This wasn't the case a few months ago.

Comment · 3 days ago

??? I don't see anyone claiming they made a game, let alone a "fun" one, you're replying to something that is not in this thread. The claim was that the agent wrote a program with clearly defined constraints and it works according to spec because it dutifully tested it and iterated on incorrect code. It's not bugged, it doesn't crash or anything like that.

Comment · 3 days ago

>I don’t really view 30 hours of stacked exposures as “a photo I took”, rather as the outcome of a lengthy signals processing chain

Is there a difference? It's still a photometrically correct capture of the scene, unless you're using narrow-band filters or multispectral compositing you would see the same if you had huge eyes. In the same way a stacked smartphone photo is still a photo unless it's also using AI, and even then AI denoisers are pretty much accepted in photography as long as they don't meaningfully affect the subject (as well as more traditional things like CPLs or extra lighting).

Nothing in Nikon's challenge is natural, it's still a photo as long as it represents something real.

Comment · 4 days ago

There are many wildly different definitions of AGI to begin with, but sure, "deep" cognition would be able to accomplish a ton more than the current "wide".

Comment · 4 days ago

And yet here I am, playing a full-fledged first-person shooter written from scratch by an LLM as a proof of concept. It has enemies I sketched on paper, everything else is 100% procedurally generated in code. In what way does it not work? It's not very good, it's not optimized for maintenance or extension (that would take a complete rework which is also cheaper with it), but it does work and is not broken because the agent tested the most obvious scenarios. It took it a few hours.

As soon as your spec has a clearly defined reachable goal and non-contradictory constraints, and doesn't require taste or judgement, current agents are pretty good at reaching it, because it's all they're optimized for. Writing that spec in a non-contradictory manner is not trivial, though.

Comment · 4 days ago

Anthropic consistently claims the opposite since Opus 3 (early 2024) which was notoriously better than their modern models at understanding overarching intent. That was literally their official prompting advice, roughly "in our tests Opus-written prompts were better than prompts written by humans, so you should let Opus write them for you". But it was painfully obvious that the model lacked depth to do that, and it's obvious modern models lack it too. It will probably not be solved until reaching AGI, whatever that word means.

AI shops rely on scale automation, metrics, and LLM judges, which heavily distorts their visibility into what they're doing. This often results in being out of touch. That's likely how Claude lost its writing ability (they stopped reading its outputs).