dagss

    But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.

Sure about that? I thought that what LLMs know about themselves is from humans writing about LLMs?

Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.

You are both talking about this claim?

    LLMs do fantastic on problems they can copy out of their training set, but they fall apart on actually novel tasks.

I mean, I give Claude what I consider novel problem solving work all the time.

You can make the statement true if you raise the bar for "novel task" to a point where most humans also can't do them. In these threads the goal posts for words like "understanding", "creative", "novel" etc gets moved for LLMs to way past their previous definitions, in fact way past a large % of the worlds human software developers.

To just take one example: Ask for a security review and you get back a list of all the things you would have found yourself after some days of pouring over the code (if you are senior developer well versed in security issues), plus the ones you would have overlooked, but in 15 minutes instead of in several days.

Before LLMs came along this would obviously have been "novel work". A shining example of a human using their intellect! But now that LLMs can do it it is just "copy and paste work"...

When I look at the history of human culture I really see a lot of remixing and "trends" and the leaps are rather small.

What sticks out is really the lack of people doing something totally new. Look back on history and it seems like humans can't do anything except as in response to some input that we re-mix. (Although it is across fields -- a change in economic conditions or scientific outlook may affect music and painting; a small improvement in factory production methods may ripple down to experiments validating general relativity, etc)

Why didn't Mozart create a piece of jazz or rock to mix things up? Why are there no medieval cubistic painters? Why didn't someone understand $scientific_topic before?

Most such questions can be answered by some kind of "the social environment and culture at that time didn't prepare people to do it". I.e., humans couldn't remix it yet from what was already there.

Yes there are leaps but they seem to always be within some small "box" that prior knowledge creates for us. We can jump one inch at the time but no further..

Yes that is what I say so we agree.

"Writing code is probably something LLMs have excelled at the most. But how much of their ability to write code is due to their understanding of it vs. just them recalling code snippets which were present in their training dataset or their context window?"

This makes me think mostly about how humans work.

Programming is to humans like speaking a language. You have ideas for what to achieve and express, and if you are a proficient programmer then outputting those thoughts in a programming language is second nature.

Programming (like most things humans learn) takes practice. It is NOT enough to simply read a specification and then you know how to code (except for transferring skills between similar languages). You repeat, make mistakes, and learn from them.

I would be tempted to say that humans definitely "repeat snippets from their training dataset". We mainly recall and express from our training/experiences. Creativity is mostly recombination of things already seen.

I have used -- for decades.

Yes I have often caught stuff like "not using DB transactions in the right way" or "this is not going to be race safe" for instance from juniors just by glancing on a PR.

I find recent LLMs don't do such mistakes though, in my experience. And also LLMs will find such mistakes if asked to review, if not perfect then more predictable than humans.

I meant it seriously though.

Characterizing the difference between humans and AI is going to be important.

Pointing out that AI is software isn't helpful in answering that question.

What is the defining feature of human minds making them "not software"?

Running agents in MicroVMs and skipping permissions is the most impactful change I have done in my habits for a while.

Have a look into Docker sbx for instance.

...and cars also changed very rapidly during early years, not to mention how often they would break down and how they basically required the user to be a mechanic to repair them on the spot for many years.

So the analogy isn't all that good is it?

You are comparing immature tech with very mature tech.

What technology emerged into the market fully finished?

If your point is "don't use technology until it is mature" then you are of course free to not use it for now.

HN is probably not the best forum for theological debate...

But e.g. modern American evangelical churches are to me quite clearly quite far from the original meaning of the texts with some theology outright invented in 18th and 19th centuries.

It really is the case that what you bring to a text colors what you get out of it, and the more people insist that they are reading a text "as it is clearly written" the blinder they seem to be to this.

Sorry, I didn't mean to make any statements about you personally, at all (and I do feel you are interpreting my comment uncharitably).

It has nothing with knowing you or not to do at all, I was just commenting on the contents of your post, not about you.

I meant that the act that you described (comparing religion with a virus) is in itself using the name-calling-aspect-of "virus". One is borrowing the negative connotation for a purpose, because one has (for other reasons) decided that the religion or ideology is bad.

The descriptive property of virus that you then mention (as in being "viral", packets of information in cell etc) equally applies to liberty, democracy and other things that (at least I) consider good.

The point was that when you do this you participate in name-calling/slurs, not making an argument.

Many things that you personally think are good (I don't know you but say for instance ideas around democracy or liberty) can also be described as viruses in the same way.