EigenLord

I've always felt that you can't think in anything besides thought. Words, images, symbols, etc, are all side-effects. They absolutely bend back and influence the thought process, but they are always secondary and indirect. Thought itself is ineffable.

Interesting work but this strikes me as a somewhat quixotic fight against inevitable tendencies of statistical models. Reinforcement learning has a single goal, an agreeable mean. Reinforcement learning stops when the LLM produces agreeable responses more often than not, the only way you can achieve absolute certainty here is if you tune it for an infinite amount of time. I also don't see how this method couldn't be subsumed by a simpler method like dynamic temperature adjustment. Transformers are fully capable of generating unpredictable yet semantic text based on a single hyperparameter. Maybe it would make more sense to simply experiment with different temperature settings. Usually it's a fixed value.

I think there's a critical flaw with Anthropic's approach to memory which is that they seem to hide it behind a tool call. This creates a circularity issue: the agent needs to "remember to remember." Think how screwed you would be if you were consciously responsible for knowing when you had to remember something. It's almost a contradiction in terms. Recollection is unconscious and automatic, there's a constant auto-associative loop running in the background at all times. I get the idea of wanting to make LLMs more instrumental and leave it to the user to invoke or decide certain events: that's definitely the right idea in 90% of cases. But for memory it's not the right fit. In contrast OpenAI's approach, which seems to resemble more generic semantic search, leaves things wanting for other reasons. It's too lossy.

I wish more comp sci curricula would sprinkle in more general courses in logic and especially 20th century analytic philosophy. Analytic philosophy is insanely relevant to many computer science topics especially AI.

I've been thinking lately about how AGI runs up against the No Free Lunch Theorem. This is what irritates me: science is not determining the narrative. Money is. I highly recommend mathematician David Wolpert's work on the topic. I think he inadvertently proved that ASI is physically impossible. Certainly he proved that AOI (artificial omniscient intelligence) is impossible.

One thing he showed is that you can't have a universe with two omniscient intelligences (as it would be intractable for them to predict the other's behavior.)

It's also very questionable whether "humanlike" intelligence is truly general in the first place. I think cognitive neurobiologists would agree that we have a specific "cognitive niche", and while this symbolic niche seems sufficiently general for a lot of problems, there are animals that make us look stupid in other respects. This whole idea that there is some secret sauce special algorithm for universal intelligence is extremely suspect. We flatter ourselves and have committed to a fundamental anthropomorphic fallacy that seems almost cartoonishly elementary for all the money behind it.

Diffusion is just the logically most optimally behavior for searching massively parallel spaces without informed priors. We need to think beyond language modeling however and start to view this in terms of drug discovery etc. A good diffusion model + the laws of chemistry could be god-tier. I think language modeling has the AI community's in its grips right now and they aren't seeing the applications of the same techniques to real world problems elsewhere.

Years ago, in my writings I talked about the dangers of "oracularizing AI". From the perspective of those who don't know better, the breadth of what these models have memorized begins to approximate omniscience. They don't realize that LLMs don't actually truly know anything, there is no subject of knowledge that experiences knowing on their end. ChatGPT can speak however many languages, write however many programming languages, give lessons on virtually any topic that is part of humanity's general knowledge. If you attribute a deeper understanding to that memorization capability I can see how it would throw someone through a loop.

At the same time, there is quite a demand for a (somewhat) neutral, objective observer to look at our lives outside the morass of human stakes. AI's status as a nonparticipant, as a deathless, sleepless observer, makes it uniquely appealing and special from an epistemological standpoint. There are times when I genuinely do value AI's opinion. Issues with sycophancy and bias obviously warrant skepticism. But the desire for an observer outside of time and space persists. It reminds me of a quote attributed to Voltaire: "If God didn't exist it would be necessary to invent him."

I think the answer to the professor's dismay is quite simple. Many people are in university to survive a brutal social darwinist economic system, not to learn and cultivate their minds. Only a very small handful of them were ever there to study Euler angles earnestly. The rest view it as a hoop they have to jump through to hopefully get a job that might as well be automated away by AI anyway. Also viewed from a conditional reinforcement perspective, all the professor has to do is start docking grade points from students who are obviously cheating. Theory predicts they will either stop doing it, or get so good at it that it becomes undetectable-possibly an in-demand skill for the future.

I would love it if LLMs told me I'm wrong more often and said "actually no I have a better idea." Provided, of course, that it actually follows up with a better idea.

The author makes good general points but seems to be overloading MCP's responsibilities imo. My understanding of MCP is that it just provides a ready-made "doorway" for LLMs to enter and interact with externally managed resources. It's a bridge or gateway. So is it really MCP's fault that it:

>makes it easier to accidentally expose sensitive data.

So does the "forward" button on emails. Maybe be more careful about how your system handles sensitive data. How about:

>MCP allows for more powerful prompt injections.

This just touches on wider topic of only working with trusted service providers that developers should abide by generally. As for:

>MCP has no concept or controls for costs.

Rate limit and monitor your own usage. You should anyway. It's not the road's job to make you follow the speed limit.

Finally, many of the other issues seem to be more about coming to terms with delegating to AI agents generally. In any case it's the developer's responsibility to manage all these problems within the boundaries they control. No API should have that many responsibilities.

I can see why some fields would have an overwhelmingly negative reaction to AI but I simply can't grasp why some software devs are. The entire point of the field is to get computers to do stuff for you. I've been doing this s*it for 10 years, there's too many little details and commands to remember and too much brutally dull work to not automate it.

I also have come to realize that in software development, coding is secondary to logical thinking. Logical thinking is the primary medium of every program, the language is just a means to express it. I may have not memorized as many languages as AI, but I can think better than it logically. It helps me execute my tasks better.

Also, I've been able to do all kinds of crazy and fun experiments thanks to genAI. Knowing myself I know realistically I will never learn LISP, and will always retain just an academic interest in it. But with AI I can explore these languages and other areas of programming beyond my expertise and experience much more effectively than ever before. Something about the interactive chat interface keeps my attention and allows me to go way deeper than textbooks or other static resources.

I do think in many ways it's a skill issue. People conceptualize genAI as a negation of skills, an offloading of skill to the AI, but in actuality grokking these things and learning how to work with them is its own skill. Of course managers just forcing it on people will elicit a bad reaction.

Research = revenue if you figure out the right idea.

Is it really a culture divide or is it an economic incentives divide? Many AI researchers are mathematicians. Any theoretical AI research paper will typically be filled with eye-wateringly dense math. AI dissolves into math the closer you inspect it. It's math all the way down. What differs are the incentives. Math rewards openness because there's no real concept of a "competitive edge", you're incentivized to freely publish and share your results as that is how you get recognition and hopefully a chance to climb the academic ladder. (Maybe there might be a competitive spirit between individual mathematicians working on the same problems, but this is different than systemic market competition.) AI is split between being a scientific and capitalist pursuit; sharing advances can mean the difference between making a fortune or being outmaneuvered by competitors. It contaminates the motives. This is where the AI researcher's typical desire for "novel results" comes from as well, they are inheriting the values of industry to produce economic innovations. It's a tidier explanation to tie the culture differences to material motive.

I think the author has a point. LLMs struggle with what you might call epistemically constructive novelty. It's the ability not just to synthesize existing knowledge, but to identify what's missing and conjecture something to fill the gap and demonstrate it to satisfaction. Out-of-distribution knowledge gaps are typically where LLMs "hallucinate." Unlike highly skilled human researchers, they don't pause and construct the bridge that will get them from known to unknown, they just immediately rush to fill in the blank with whatever sounds most plausible. They need to ask questions that haven't been asked before, or answer ones that haven't been answered. Is this just some missing subroutine that we'll eventually figure out? Or is this conjecture-proving process much more elaborate than whatever existing models, no matter how scaled, can manage? I'm not sure. But the answer starts with a question.

You've got to do piecemeal validation steps yourself, especially for models like Sonnet 3.7 that tend to over-generate code and bury themselves in complexity. Windsurf seems to be onto something. Running Sonnet 3.7 in thinking mode will sometimes reveal bits and pieces about the prompts they're injecting when it mentions "ephemeral messages" reminding it about what files it recently visited. That's all external scaffolding and context built around the model to keep it on track.