EarthBlues

I like AI coding. I was talking about this with a writer friend: what I get paid for is my judgement. Which goals are worth pursuing and which aren't? We always knew tech is all tradeoffs all the way down, so how are they to be evaluated? My writer friend faces similar challenges. AI doesn't have an order of values within which to weigh judgements, except for whichever one is implicitly ir explicitly imposed upon it. I also don't think that LLM context works the same way my reflective process works, as far as I can tell. It's much better at the activity of coding than I am and that's great, I'm happy to let it do that. I've never found it capable of independently making the kinds of judgements that are the meat of my writer friend's and my professions.

You might be right. I dont think its certain though. Humans seem to be able to communicate across language gaps in ways we can't with animals. Things really could be that different. For example, all human languages that we know about have some idea of a who doing something to a whom, even if they express it in drastically different syntax. For all we know (I'm just making this up, don't take it too seriously), whales might experience the world as a stream of objects. Maybe that's perfectly fine for whale existence. But we can't even really conceptualize that! And how would we know, anyway?

I have some problems with Wittgenstein's philosophy when applied to human rationality (I think cross-game invariant structures do exist) but here I think it's an apt observation and you could very well be right. Nice!

I don't want to poopoo this research because I think it's really cool. I just want to urge some caution. There are three distinct risk categories with this research that I think should trigger a halt if certain pre-specified conditions are breached.

(1) Animal welfare. The researchers are aware of the animal welfare issues at stake, but a collection of metrics around pod culture, stress, and reproduction should be tracked across experiments to make sure we're not hurting the animals. If thresholds are exceeded across multiple experimebts with this tech, the research should stop.

(2) Acoustic -> semantic gap. There's a green lumber problem with this research. We may be able to predict animal responses in playback tests, but we don't have a way to map that back to semantic meaning. This is probably fine for now, but if we don't have adequate semantic theory or if we have multiple competing semantic theories that all explain the patterns equally well, the we're just manipulating the animals, not communicating with them. (eg, It's not even really clear to me that "whale concepts" and "human concepts" would be commensurable.)

(3) Bad incentives. Between media sensationalism, the normal problems and biases that come with funding science today, and potentially terrible usages of "whale tech" (eg, clearing out shipping lanes, warfare), it seems to me there are some extra ethical considerations with this research such that there may be a point where the outcome space is overwhelmingly negative. It would be nice if mechanisms existed such that research could be stopped if these bad incentives activate.

My point is that by your criteria, the enlightenment isn't uniquely pro-science either.

I think Nietzsche is broadly right, that Christianity and enlightenment are brother-enemies. The belief that Christianity was Anti-Science and then the Enlightenment came and saved the day is a plainly falsifiable myth. From The Gay Science:

"It is still a metaphysical faith upon which our faith in knowledge rests—that even we knowers today, we godless anti-metaphysicians still take our fire from the flame lit by a faith that is thousands of years old, that Christian faith which was also the faith of Plato, that God is truth, that truth is divine."

You may want to look up what Hume, Kant, Mill, Rousseau, Voltaire thought about non-whites and "backwards" peoples.

Or what constituted "enlightened, scientific, progressive" opinion on eugenics until WWII.

History is not a superhero comic.

For me, I just can't falsify my experience that things do matter, that I make choices. The people in my life that I encounter seem to have this same experience.

well, its a matter of definitions.

the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.

llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.

the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.

llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!

forgive typos, i am on mobile.

I agree that "maintainability" is the wrong place to stake one's flag. It's a technical problem and will probably be overcome. The "lack of wisdom" in AI coded projects which does not strike me as a soluble technical problem concerns questions like: which software is worth writing and which isn't? How are tradeoffs evaluated? I don't think AIs are structurally capable of answering these questions in the same way humans are. But they make those decisions anyway!

Good essay with lots of interesting points. I especially liked how the author tried throughout to appeal beyond a narrow left-wing audience.

Something that I noticed that might help clarify things: I think the author mixes two phenomena together: (1) Rents and (2) subjective vs objective property rights.

Rents were pervasive in the middle ages. There are lots of black legends about medieval Europe, but we don't have to tell golden legends about it either. The whole economy if medieval Europe relied on people controlling land, then demanding goods and service in exchange for its use. That is Rent! It was often exceedingly exploitative and harsh!

What changed in the modern era was a the decline of objective property rights. The way property was treated in law and fact went from "this is my farm, which is mine for farming, and it comes with these rights and obligations. That's the commons. I have these rights to it and these obligations to my lord and his other subjects in how I use it." This changed to "this is my property with which I can do what I see fit. I have no objective obligations to anyone else." Subjective property rights also existed before capitalism, but they completely eclipsed objective right in the early modern period. This increased economic productivity and growth on a scale that was not thought possible, but also, as the author notes, had serious negative consequences.

The reason that this distinction matters is that you can absolutely abolish subjective property rights and still have rents. You can actually make them worse. If people are being extorted for access to some good, they will not care if it's done by a private company or a public administrator. I don't see how the author's framework of "personal property" overcomes the issue.

A book that has shaped my thinking on AI, and much else besides, has been Horkheimer's Eclipse of Reason.

This book gives a conceptual groundwork for what I believe to be both the promise, and crisis of AI. AI excels at, and either will soon or already has surpassed humans at a kind of reasoning that Horkheimer calls "instrumental reason." This is reason as a tool for achieving ends. AI will, I think, surpass humans at writing maintainable code, as this falls within that domain. The writer is thoughtful, but the obstacles listed in the article are technical, and they will likely fall.

A second kind of reasoning, which he calls "objective reason," is reasoning about which ends are worth pursuing. Our contemporary pragmatic, positivist culture in the West has a hard time reasoning about ends. We tend to just see different perspectives, competing power networks, etc. We are skeptical of capital-J Justice, for example. We tend to see it as historically and culturally contingent, or else as a mask for powerful interests. Horkheimer believed this learned relativism was responsible for new forms of domination and control that emerged in the 20th century in both totalitarian and putatively free societies.

I think the author's thesis would be correct if it had focused on AI's inability to discern ends. But that then raises the question: can we discern ends? Horkheimer never settled on a way to rehabilitate objective reason. He focused on critique, and his critics suggest he ended up with a tangle of negations that couldn't hold up any objective ends. I think rehabilitating objective reason is the task that faces us whether we like it or not, given our moment in history with the advent of AI. It's something practitioners need to think about. Either we recover the philosophical tools of objective reason in light of the critique which caused them to be discarded in the first place, or ends will be imposed arbitrarily by any and everyone with the means to do so.

The problem, as ever, is figuring what's worth doing and what's not. LLMs are not as useful there.

I wonder to what extent this effect comes from the fact that robotaxis don't get drunk.

I seem to recall a paper showing that human crash frequencies surge up to 6x between midnight and 4am on weekends, despite that traffic accounting for only a miniscule percentage of vehicle miles traveled.

This is a good example of a positive, necessary conversation that was almost thwarted by an engagement-optimized article. It only happened (three quarters of the way down the HN comment section) because a few users were mature enough to not take the bait and actually discuss the issues at hand. Good stuff from both you and the parent poster.

I think people get hung up on wanting to collapse developer productivity into a single dimension, usually for stack-ranking purposes. This, I think, is always going to punish good engineers and reward bad ones to some degree.

Measuring developer productivity should, in my opinion, have one dimension for speed, one for quality, and one for user impact. LOC can be fine as a measurement for speed, you just don’t want to look at it in isolation. You would want to also measure, for example, escape rate and usage for the features the developer worked on, and be willing to change or refine these if circumstances require it.

You also need to look for different profiles based on the developer’s level of seniority. A senior dev probably shouldn’t be writing as much code as a contributor, but their user impact should be high, and escape rate low. Analyzing differences between teams is important, as well. A team that has a lot of escapes or little user impact probably has issues that need management attention, and may not have anything at all to do with individual developer productivity or ability.

In brief, the numbers are there to help you make better management decisions, not to relieve you of having to make them.