nernst
Joined 16 years ago
Joined 16 years ago
Your point is still valid, but perhaps we approach a time when over-reliance is better than all but the best human pilots (Sully, perhaps).
My other concern was the answer to #1, namely, "4m + 5b" vs "5m + 4b". I feel like if you know it is in that form (sum rather than product) the two answer choices are just intended to trip people up who are moving through the test at speed.
In fact I would say an important "mistake reviewers make" is ... not actually doing much work. I've seen some appalling 2-3 line comments like "seems fine", even from senior academics. That's not even to talk about problems with misunderstood p-values, not reading the algorithm closely, not walking through the proof manually, and so on.
I did not explicitly look at size as this was not one of the original research questions, but a good point. I suspect older systems will tend to be larger (in the domains we studied, anyway). And your point about arch choices being great "early on" can, I think, be captured in the "system age" variable. I guess I'm trying to think of a system that might be young and yet quite large in LOC.. would be an interesting outlier to look at.