As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.