This is a really interesting application of LLMs. The lack of "repeatable, traceable results" is indeed a huge issue for any serious use case (we see this constantly in enterprise adoption).
Have you found that forcing the LLM into a structured scoring framework reduces its tendency to hallucinate specs? Or does it just hallucinate the scores with more confidence?
Also, curious if you've tried different models for the "scoring" vs "reasoning" steps. We've found Claude is much better at adhering to complex constraints than GPT-4o for tasks like this.
Have you found that forcing the LLM into a structured scoring framework reduces its tendency to hallucinate specs? Or does it just hallucinate the scores with more confidence?
Also, curious if you've tried different models for the "scoring" vs "reasoning" steps. We've found Claude is much better at adhering to complex constraints than GPT-4o for tasks like this.