Thanks for reading! We'll definitely include our Sonnet results in the next revision. It's worth pointing out that we're comparing accuracy on text responses and not log probability based scoring, which I think is the number you're referring to (based on Section E of this paper https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...). But if I'm mistaken and you have a direct pointer, that'd be super helpful! In general, we've been basing our comparisons against the models in the Open Medical LLM leaderboard here: https://huggingface.co/spaces/openlifescienceai/open_medical...
Also definitely a good idea on the ablation study. We had some results internally based on a production-tuned version of our model that includes a much higher weighting of records-data. It's an imperfect ablation, but it supports the story -- so I think it's there, but you're right that it would be more complete to develop and include the data directly.
Also definitely a good idea on the ablation study. We had some results internally based on a production-tuned version of our model that includes a much higher weighting of records-data. It's an imperfect ablation, but it supports the story -- so I think it's there, but you're right that it would be more complete to develop and include the data directly.