Comment by MattCruikshank

Do you want to measure the ability of the box, or measure the ability of the box with one hand tied behind its back?

More to my point, I think it's stupid to have LLMs do work that should be done by programs... programs potentially written by LLMs.

I'm advising people that they should think about this distinction, themselves, when they have data and want answers.