Author here. Majromax challenged me to test `i = 1 + i`, which broke my
theoretical framework. While setting up that experiment, I realized I hadn't
used chat templates in my original measurements (rookie mistake with an
Instruct model!).
Re-running with proper methodology completely flips the results - the terse
version actually wins. I'll add a correction note to the article once AWS/Medium
comes back online and will write a follow-up with the corrected experiments.
This is open science working as intended - community scrutiny improves the work.
Thank you all for the engagement, and especially to Majromax for the challenge
that led to discovering this!
Re-running with proper methodology completely flips the results - the terse version actually wins. I'll add a correction note to the article once AWS/Medium comes back online and will write a follow-up with the corrected experiments.
This is open science working as intended - community scrutiny improves the work. Thank you all for the engagement, and especially to Majromax for the challenge that led to discovering this!