I used to work at a musical instruments startup and we tried blind tests of prototypes to assess which combination of materials recipes/sensor design/firmware algorithms etc were 'the best'.
Ultimately these tests only reveal the biases of the players. They will prefer – and actually perform better with – that which they believe to be configured in a certain way, regardless of whether it is in reality or not. We had one prototype that us engineers hated to death because the only difference between it and the rest was that it was a different colour, but ALL the players hailed it as the golden standard.
The same goes for the famous 'Does Fuck All' button on recording mixing desks, which you can use to tell players that you've made the change that they wanted to satisfy them. And also when you're buying speakers it's very common to be shown the same set of speakers three times and have three opinions.
I just recently tour managed a friend and would set up her mic sound every night, and each time I would need to play some sort of subtle trick with her so that she felt things were just right. No amount of 'rational tweaking' could achieve the same.
Ultimately these tests only reveal the biases of the players. They will prefer – and actually perform better with – that which they believe to be configured in a certain way, regardless of whether it is in reality or not. We had one prototype that us engineers hated to death because the only difference between it and the rest was that it was a different colour, but ALL the players hailed it as the golden standard.
The same goes for the famous 'Does Fuck All' button on recording mixing desks, which you can use to tell players that you've made the change that they wanted to satisfy them. And also when you're buying speakers it's very common to be shown the same set of speakers three times and have three opinions.
I just recently tour managed a friend and would set up her mic sound every night, and each time I would need to play some sort of subtle trick with her so that she felt things were just right. No amount of 'rational tweaking' could achieve the same.