It really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.