Diogenesian

  Thariq wrote "personal software was a bit early in 2020 but in 2026, it really can be as personal as a home cooked meal, or a handwritten letter."

Excuse me??? Vibe-coding personal software is not a homecooked meal or a handwritten letter! It is [at best!] more akin to tastefully combining microwaved ramen with microwaved veggies and toaster oven chicken, or using ChatGPT to write a letter then editing it afterwards. It's fine, maybe even very good. But it's not "homecooked."

Ugh. The most depressing thing about the AI boom is watching tech people devalue human experience.

Even in early 2023 people were talking about this as a use case for ChatGPT copy-paste coding. Arvind Naranyan (AI As Normal Technology) used it to build edutainment applets for his kids.

What an asshole:

  On the other side of the AI-reasoning fence, the disdain seems to be mutual. “These ‘scientific’ papers from last summer — I would put this in big, big air quotes,” said Sébastien Bubeck, a member of OpenAI’s technical staff (and a prominent evangelist for the company’s reasoning models among scientists and mathematicians). He called earlier Apple results critiquing AI reasoning “wrong,” claiming that they were due to a training quirk in models that are now obsolete. “Modern models starting with GPT-5.5 do not suffer from this issue,” he said. “It would be interesting to revisit those results.” (Apple did not make its researchers available for interviews.)

Then, later:

  The “think” part is what OpenAI, for one, is doubling down on. When I asked Bubeck if the splashy unit distance proof was produced with methods outside the LRM’s own chain of thought — perhaps with Lean verifying its results — he seemed to find the question almost nonsensical.

  “It’s not like we’re making a mystery of it,” he said. “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Technically, OpenAI released a “rewritten summary” of the model’s chain of thought produced by two human experts using Codex, another OpenAI model. Since 2024, the company has not publicly revealed “raw” chains of thought from its reasoning models, a policy also adopted by Google DeepMind and Anthropic.

That "training quirk" thing is obvious (yet unfalsifiable) BS, and who the hell is he to sneer about "science" when his company won't release the raw data for independent scientists to look at?

Hmm, fair enough - I checked Wikipedia. My mind completely erased the "sophon" bit or whatever it was called, it just got erased again. Not a fan of that particular fairy dust.

I thought they were referring to the inability of the aliens to make an accurate calendar.

But I think this has to be judged against an adversarial influx of LLM-assisted pseudomathematics slop vendors. A sorry-free proof in Lean (or Rocq!) whose top-level types check out is not good enough. You gotta check for compiler chicanery in all the private methods.

Or, alternatively: refuse to accept a Lean program as a valid proof. I assume LLMs are pretty good at Lean -> mathematical English in LaTeX.

I believe the alien scientists were depressed by the guaranteed collapse of their civilization, not the difficulty of certain differential equations. So it's more like a doctor getting weird results while working on a cure for a pandemic.

Edit: oops, see the reply below... and my grumpy response. I seriously forgot about that entire plot point.

I don't think open-weight models are what motivated these specific comments. It's a way of deflecting OpenAI's failure with HuggingFace/etc as a natural consequence of powerful AI instead of a specific problem with OpenAI's recklessness.

The problem is that this doesn't distinguish LLMs from other forms of software! The whole point of computers is to offload tasks requiring reasoning, and they've always helped us understand complex nonlinear systems is that computers work through the hideous details.

More significantly: many major advances in scientific or mathematical programming were heralded as a thinking computer! 50's numerical scientific programming, 60's Lisp+ symbolic programming, 70's logic and symbolic algebraic programming, 80's unifying all this with encyclopedias and NLP like Mathematica - all of this is clearly useful and cool, all designed to offload intelligence... and, with modern eyes, we see plainly that not a single shred of intelligence is required to execute any of it. The AI researchers of yesteryear consistently made cool computer programs and consistently overestimated their progress at cracking human intelligence. It is plain as day to me that ANN researchers are making precisely the same scientific and philosophical mistakes as yesteryear's symbolics researchers, and even Alan Turing. This does not diminish from the coolness of the computer programs. But pretending these systems are intelligent, even "verbally" intelligent, is bad for society.

It's bad for computer science, too. None of the progress in LLMs or ANNs gets us any closer to making a robot as smart as a cockroach. I doubt any of us will live to see that milestone.

It's plausible quite a few smaller datacenters have modern engineered wood frames. Wood is pretty high-tech these days. Engineered wood is cheap, lightweight, preserves as well as reinforced concrete, and increasingly for new residential buildings under 5 stories. It would be competitive in a bidding process considering the investors clearly have an appetite for risk.

But even for a steel and concrete datacenter you would want a few carpenters for the interior where humans (at least occasionally) work, more to help with scaffolding and other temporary buildings (e..g housing) during construction, and building framing for drainage ditches and other landscaping. I imagine wood-and-nails carpenters are quite in-demand for most datacenter projects.

But bugs like this aren't because natural language is ambiguous, it's because the LLM/etc has inadequate safeguards against unambiguously malicious text. If LLMs were capable of understanding human language and only subject to natural linguistic ambiguities like any other college-educated humans, bugs like this wouldn't be reliably reproducible across different models. People in this thread are trying very hard to argue that humans are subject to this via social engineering but it is not the same. GPT-5.6 is subject to this bug for the same reason it sometimes rm-rfs stuff it "knows" it shouldn't: these machines are still stochastic parrots. It is borderline magical how powerful stochastic parroting is as a means of computation, but in the same way that a minimal Lisp system can magically be extended to a powerful theorem-proving algebra-cruncher. But parroting is simply not how humans actually understand language, and it is clearly an inadequate way of implementing language on a computer.

Philosophically no, but that shouldn't be a distraction from the issue with LLMs. This really is closer to "Outlook runs an untrusted VBA macro" than "intelligent entity gets confused by inherent ambiguity in human language."

No, it's an honest thing to say. If you're not able to read a graduate-level textbook and teach yourself category theory then you don't have the mathematical maturity to understand category theory. You need to start smaller: ideally abstract algebra and point-set topology, or set theory and mathematical logic if you just want the CS applications. Likewise with Lean. Of course there is a place for a good human instructor. But mathematical maturity must be developed the hard way.