didibus

You lost me.

If we're discussing the article, than we should be using the definition put forward in the article.

I also think it means you have to operate very differently around it, it's about guardrails, quality assurance, funneling context, keeping output in check, emphasizing what matters and doesn't, etc.

It's less about code structure, maintainability, readability, algos, class/function abstractions, etc.

And to Rich's point, you're not getting to experience the process of coding anymore, what you'd have learned by doing it, about the domain, about logic, about structure, you're missing on that. You're likely learning something else though, at a higher plane, and maybe about managing autonomous agents, and so on.

Ya, and in a comment he mentions he sees a spectrum from tool to automation.

I think a power tool would exist somewhere in the middle of that spectrum, closer to automation than a hand tool, but further away than a CNC router.

To be honest, I think he could have tried to find a better word for the left side of the spectrum than tool. Because in dictionary sense, all these things are tools. Anything a human uses for some purpose is a tool. I think he's trying to simply say, yes all these can be used to do things, AI can be used to make software, so can your IDE, but please think through the difference in how the experience and the process of making software differs vastly if using an IDE for it versus using an autonomous AI agent. And if you don't think it changes anything, well that might be at your own peril.

You're too hung up on the social agreement of what a tool is. Yes, if I move into the sun to warm up, the sun is a tool. That's proper dictionary "tool definition". I believe Rich's point is, if you see it that way, then think AI is just a tool, you're missing on the world of differences between things.

So he purposely narrows the definition of a tool, and exclude autonomous things from it. He also excludes passive things like the drive-through window, which isn't actively being manipulated by anyone, but just serves as a part of allowing you to get food through the wall while remaining in your car. Yes, dictionary definition would say a drive-through window is a tool as well, but saying that is not very useful to reason about how a drive-through window and a hammer actually relate or differ.

If you start to instead think of autonomous things as automation, and things that are more extension of the human expression or movement as tools. You can start to better understand those things and that using one over the other changes your experience and what you gain or lose in the process even if your use of them results in a similar outcome.

I think that's the next layer of conversation. The first is that Rich is saying, if we think of everything that a human uses for any purpose as a tool, it's not very useful to reason about.

When a definition explains everything, it explains nothing. So he goes on to setup some boundaries and describe some narrower qualities to try and help us reason about the nature of those things.

The only similarity between your keyboard and an AI agent is that they both have utility. You might say: "I used AI to write my essay" or "I used my keyboard to write my essay". It's very deceptive and easy to get trapped in thinking, these are the same, in both cases you just used a tool to write an essay with? AI is just a tool, like a keyboard. What Rich is trying to bring about is that there's actually a world of difference here. And if you keep thinking of a AI as "just a tool", you're missing that entire world of difference. It becomes much better to think of AI as automation and leave the tool definition to the keyboard. Now instead of saying "I used AI to write my essay" say "I automated the writing of my essay, I did not write the essay, AI did". That latter framing is so much more accurate, and changing that framing and perspective is going to give you a much deeper understanding of the impact, trade-offs, consequences, etc.

Pretty sure your phone assistant isn't using their best model. It's probably using one of the old flash models no?

> Buying Siri and having it be useless for anything beyond setting timers for 5 years, is kind of unforgivable

I don't follow, that was just the state of the art at the time, Alexa, Cortana, Hey Google, were all similarly shit. And none has yet offered a scaled up assistant at the free-tier/cheap bundled sub yet.

> I suppose he's right that it's not a tool for writing code.

I think that's exactly what he means.

> But it very much feels like a tool I'm wielding to do software engineering at a far more powerful level than I could have done without it.

Ya, I'm not sure if he heeds a warning that AI is slowly moving to automate all that as well. Only 2 years ago it couldn't automate writing all of the code for example, it's possible in 2 years it has automated most of that higher level as well, and therefore software making becomes same as ordering a sandwich, just tell it the app you want and you get it. Or if he's just trying to make the reader realize they've given up coding and are losing that skill (or if new to coding are not learning it by using AI), and for better or worse is on the reader to decide.

> Would you bet your life, and everyone else's, to test that assumption?

Haha, I guess it depends what I'd win.

But I wasn't challenging that to say it's a good idea to build this. Simply an irk with the logical reasoning that was used.

Rich mentions in another comment that he's talking about its embodiment in conversational systems purporting to be 'tools' for doing homework, research, coding, writing, etc.

What that means is that these are tools for ordering a sandwich, but not for making one. The making of the sandwich was automated, and you're not making a sandwich using a tool to do so, you're ordering one and someone else is making the sandwich for you.

Back to coding, if you use AI agents to generate your code, it means you're not coding anymore, you've automated that, you're ordering code and maybe sending it back to ask for a better one if you didn't like it. Meaning you've lost the fine grained control, the tactile feedback, the learning opportunity, the experience, etc. You have gained the benefit of automation instead, unlike those of a good tool, which make you better at doing the work, automation likely makes you worse at it.

I don't think he's implying automation is bad, just that you should understand that the dynamic of automating something is very different from that of doing the thing using a tool.

Yes, Rich says so explicitly as his reply to a comment:

> Rich Hickey: Toolness is a spectrum. The more using a thing is like ordering a sandwich, the less it is a tool. And yes, cars are decreasingly tools.

I don't think he's trying to make a semantic discussion.

He's trying to unpack various qualities: learning potential, reliability, predictability, tactile feedback, simple construction, etc. Then goes to define tools as those things that have these set of qualities, and automation as those that have these other set of qualities: it does something for you, takes the practice, learning and details away from you, harder to control and can get derailed, etc.

In some of his comment replies, he even says there's a spectrum from tool to automation, because he's focusing on those qualities more so than on the semantics.

Don't think that's at all what the article in question is discussing, but this was interesting.

The assumption I need to challenge is that "an ultraintelligent machine could design even better machines".

That assumes that better machines are even possible, that there's no point of bottleneck that cannot be broken, or that they are so much more intelligent than us that they too won't hit a limit pretty quickly to what they can figure out.

It also says it surpasses any man however clever, but doesn't say it can surpass the totality of man interacting and building on each other's. If you take the most clever of all man, they are not so clever as to surpass humanity's total ingenuity and invention. And it's not a given that this super intelligent machine, more clever than any one man, is able to replicate itself and work in tandem with copies of itself in a way that is as effective as how humans somehow form a dynamic system between themselves.

Lastly, and I know the premise assumes it's better at everything, including at building intelligent machines, but I think an ultra-intelligent machine could exist that struggles at building better machines. Depending on your faith, we might be the construction of lesser intelligent organisms eventually discovering and evolving into us, and yet we do not yet understand how to make a more intelligent human.

I think Rich's comment reply here explains the post:

> Commenter: Well, this was a twitter-like controversial clickbait but I will say it anyway. LLM is a tool. To be more specific, it's a lossy knowledge base with a human language for querying and built-in summarization.

> Rich Hickey: Agreed. I'm talking about its embodiment in conversational systems purporting to be 'tools' for doing homework, research, coding, writing etc.

With that framing in mind, the article makes a lot of sense. AI agents are not tools for coding, they automate coding, which means you lose control, learning, predictability, etc.

In and of itself, an AI agent is a tool, can be used for things, like say making software, but if thinking of it as a tool for coding, that appears wrong, as using one changes the modality of the task, you are not coding using a tool, the coding is no longer done by you and has been automated.

As he also says, reading an AI summary of a paper is a different modality from using it to read the entirety of a paper, therefore it is not a tool for reading the way a Kindle, glasses, reader-mode are. And similar to with coding, you've lost things from automating the reading, that reading just a summary does not provide.

That's just another P(doom), or does making an ASI somehow negate the risk of nuclear war? Cause I'd assume it actually increases it.