chilmers

Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.

> If that's true, how do you explain the observable pattern of many consumer applications degrading in UI functionality and coherence after a certain point?

Most consumer software in developed under the assumption that features only have a cost when being built. Their maintenance cost, and the combinatorial complexity each new feature adds, is barely considered. As a result, software teams add complexity at a speed that exceeds the rate at which the team's size, and the volume of work it can handle, grows. Eventually this reaches a critical threshold and the quality of the software starts to decline. But this doesn't mean the software couldn't still be improved, just that there isn't the resource available to do so.

Unless you're Sam Bankman-Fried, Bernie Madoff, Elizabeth Holmes, Allen Stanford, Raj Rajaratnam, Michael Milken...

This is not a "concern", this is literally what happens. Even prior to AI vulnerability scanning, release notes and diffs for any normal release or security patch of popular software were analysed and used to develop exploits, which could then be applied against non-upgraded instances. Nowadays, AI is doing this en masse and automatically producing exploits. And, as has been demonstrated, nowadays even the rumour of a major security bug is enough for people to find it, as people will immediately burn huge volumes of tokens looking for it.

Nor can you just keep it quiet and smuggle it out in a regular release as you suggest. For one thing, changes in important software are be linked to commits and have pull requests, reviews, etc. Changes without these will immediately raise suspicion, as either a malicious change snuck into a release, or as a "quiet" fix to a major vulnerability, completely undermining the purpose of keeping it quiet in the first place. Moreover, major vulnerability fixes cannot be "quiet", because the whole point is to get as many people as possible to upgrade as quickly as possible. If PHP (or whoever) push out a release containing a major security fix, but don't make this clear, then many people will be in no rush to upgrade, and will get pwned.

You could say it got a bit footloose.

This article is proof you don't need to use or like AI to be a victim of deep AI psychosis.

I have subscriptions to both, one for personal and one for work use, and I haven't really come close to hitting the limits since the release of Opus 5.5 and GPT-Sol 6.1, and I'm no longer hitting any of the capability limits I used to experience when using the "mid" tier models, so I don't miss using Astra. It feels like limits at pretty much a solved problem right now, for my level of use.

How is tracking a women’s world record discrimination or “bending the tournament”? It’s not as if women are being prevented from competing with men. Speedcubing competitions are open, single-category.

A company selling combined human + AI financial advice finds that AI advice alone is unreliable? Color me surprised.

I feel like any service that is free for open source is going to struggle if it gets significant adoption and becomes a default target for hosting LLM generated code. If even MS/GitHub cannot scale to the current demand, what hope do smaller and less well-funded alternatives have?