Yeah I agree - if you're very clear and consistent with how you ask it score something it will do well. Not fundamentally different than giving it clear instructions when you code.
One thing that has helped me make the LLM more consistent on scoring is deduction based rubrics. Start everything at perfect and deduct points for specific things (ie style, missed facts, etc). It gives much tighter final scores under these conditions (for me) than giving rubrics + samples.
One thing that has helped me make the LLM more consistent on scoring is deduction based rubrics. Start everything at perfect and deduct points for specific things (ie style, missed facts, etc). It gives much tighter final scores under these conditions (for me) than giving rubrics + samples.