gabiteodoru

Do you know which method is this? Euler's by any chance? And do you have an idea how one would prove that it creates a magic square? It's actually one of my inspirations for writing this, the relationship between the code that does something and the proof that the code actually does what it claims. I'd argue an LLM would find the proof helpful if it were asked to generalize an existing function in some way

Oh nice solution, thanks for sharing!

Perhaps if we also trained them on natural language ASTs at the same time when asking the questions? :)

It did go back on itself 3 times, no? "Actually, let’s trace for x=3:" (it had just computed for x=3 the first time); then "Better to check actual q output:" -- did it actually run it in a q session, or just pretended? And another one "That doesn’t seem to align. Let’s do it step by step:"

Author here. Majromax challenged me to test `i = 1 + i`, which broke my theoretical framework. While setting up that experiment, I realized I hadn't used chat templates in my original measurements (rookie mistake with an Instruct model!).

Re-running with proper methodology completely flips the results - the terse version actually wins. I'll add a correction note to the article once AWS/Medium comes back online and will write a follow-up with the corrected experiments.

This is open science working as intended - community scrutiny improves the work. Thank you all for the engagement, and especially to Majromax for the challenge that led to discovering this!

Thanks so much for this challenge! I just ran the experiment with i = 1 + i and you're absolutely right - it breaks my theoretical framework (same semantic information, but much higher perplexity).

While setting this up, I realized I hadn't used chat templates in my original measurements (rookie mistake with an Instruct model!). Re-running with proper methodology completely flips the results - the terse version actually wins.

I'll add a correction note to the article once AWS/Medium comes back online, and will write a proper follow-up with all the corrected experiments. Your comment literally made the research better - thank you!

Yep, that's exactly what my MCP server is doing -- write in a Python-like language (I called in Qython), parse into AST tree, write the q code.

Or, even better, also from the cookbook: {(2#x)#1,x#0} But this really borders on obfuscation :P

Those github links are so cool, thanks for sharing! :)

I think a translation layer to a lower-density language might be a good solution; e.g. Iverson's divisible-by-11 check, 0=11|-/d, can be verbosely done in Python with

import numpy as np

def flippedSubtract(a, b): return b - a

flipSubUfunc = np.frompyfunc(flippedSubtract, 2, 1)

def isDivBy11(number): digits = list(map(int, str(number))) discriminant = flipSubUfunc.reduce(digits) return (discriminant % 11) == 0

Though Claude already understands (has already seen?) 0=11|-/d so it's hard to tell for this example

As for the cat attack, my gut feeling is that it has to do with the LLM having been trained/instructed to be kind

:D Well I'm still building Qython, but if your colleague has some example code snippets they think particularly difficult to translate, I'd love to take on the challenge!