From:
"If we see that 1000 of the 10,000 random iterations had a difference of more than $1.50, we can say that there is a 10% chance that our $1.50 observed difference was due to randomness."
I'm wondering if maybe you're doing way too many simulations in the calibration, do you really need more than a few hundred to a thousand or so? The 0.1 quantile is reasonably well separated from 0, and I would have expected you'd get "good enough" convergence pretty quickly.
Also: "we can compare our computed p-value of 11% to our simulated 10% result to determine whether or not the model is accurate enough." you're getting a full pdf out of the simulations, are you also comparing to the full distribution of your test statistics?
I'm wondering if maybe you're doing way too many simulations in the calibration, do you really need more than a few hundred to a thousand or so? The 0.1 quantile is reasonably well separated from 0, and I would have expected you'd get "good enough" convergence pretty quickly.
Also: "we can compare our computed p-value of 11% to our simulated 10% result to determine whether or not the model is accurate enough." you're getting a full pdf out of the simulations, are you also comparing to the full distribution of your test statistics?