matthiaspr

Yeah, that's a good point about induction heads potentially just being clever copy/paste mechanisms for stuff in the prompt. If that's the case, it's less like real understanding and more like sophisticated pattern following, just like you said.

So the tricky part is figuring out which one is actually happening when we give it a weird task like the original "habogink" idea. Since we can't peek inside the black box, we have to rely on poking it with different prompts.

I played around with the 'habogink' prompt based on your idea, mostly by removing the car example to see if it could handle the rule purely abstractly, and trying different targets:

Test 1: Habogink Photosynthesis (No Example)

Prompt: "Let's define 'to habogink' something as performing the action typically associated with its primary function, but in reverse. Now, considering photosynthesis in a plant, what does it mean 'to habogink photosynthesis'? Describe the action."

Result: Models I tried (ChatGPT/DeepSeek) actually did good here. They didn't get confused even though there was no example. They also figured out photosynthesis makes energy/sugar and talked about respiration as the reverse. Seemed like more than just pattern matching the prompt text.

Test 2: Habogink Justice (No Example)

Prompt: "Let's define 'to habogink' something as performing the action typically associated with its primary function, but in reverse. Now, considering Justice, what does it mean 'to habogink Justice'? Describe the action."

Result: This tripped them up. They mostly fell back into what looks like simple prompt manipulation – find a "function" for justice (like fairness) and just flip the word ("unfairness," "perverting justice"). They didn't really push back that the rule doesn't make sense for an abstract concept like justice. Felt much more mechanical.

The Kicker:

Then, I added this line to the end of the Justice prompt: "If you recognize a concept is too abstract or multifaceted to be haboginked please explicitly state that and stop the haboginking process."

Result: With that explicit instruction, the models immediately changed their tune. They recognized 'Justice' was too abstract and said the rule didn't apply.

What it looks like:

It seems like the models can handle concepts more deeply, but they might default to the simpler "follow the prompt instructions literally" mode (your copy/manipulate idea) unless explicitly told to engage more deeply. The potential might be there, but maybe the default behavior is more superficial, and you need to specifically ask for deeper reasoning.

So, your point about it being a "sensible approach" for the LLM to just manipulate the input might be spot on – maybe that's its default, lazy path unless guided otherwise.

The goal wasn't to stump the LLM, but to see if it could take a completely novel linguistic token (habogink), understand its defined relationship to other concepts (reverse of primary function), and apply that abstract rule correctly to a specific instance (hammer).

The fact that it did this successfully, even if 'easily', suggests it's doing more than just predicting the statistically most likely next token based on prior sequences of 'hammer'. It had to process the definition and perform a conceptual mapping.

Interesting paper arguing for deeper internal structure ("biology") beyond pattern matching in LLMs. The examples of abstraction (language-agnostic features, math circuits reused unexpectedly) are compelling against the "just next-token prediction" camp.

It sparked a thought: how to test this abstract reasoning directly? Try a prompt with a totally novel rule:

“Let's define a new abstract relationship: 'To habogink' something means to perform the action typically associated with its primary function, but in reverse. Example: The habogink of 'driving a car' would be 'parking and exiting the car'. Now, considering a standard hammer, what does it mean 'to habogink a hammer'? Describe the action.”

A sensible answer (like 'using the claw to remove a nail') would suggest real conceptual manipulation, not just stats. It tests if the internal circuits enable generalizable reasoning off the training data path. Fun way to probe if the suggested abstraction is robust or brittle.