I doubt that an A/B test would really do much. System prompts are kind of a superficial kludge on top of the model. They have some effect but it generally doesn't do too much beyond what is already latent in the model. Consider the following alternatives:
1.) A model with a system prompt: "you are a specialist in USDA dairy regulations".
2.) A model fine tuned to know a lot about USDA regulations related to dairy production.
The fine tuned model is going to be a lot more effective at dealing with milk related topics. In general the system prompt gets diluted quickly as context grows, but the fine tuning is baked into the model.
1.) A model with a system prompt: "you are a specialist in USDA dairy regulations". 2.) A model fine tuned to know a lot about USDA regulations related to dairy production.
The fine tuned model is going to be a lot more effective at dealing with milk related topics. In general the system prompt gets diluted quickly as context grows, but the fine tuning is baked into the model.