Comment by Trusteando

To meaningfully test a forecasting model, use a data-generating process with a high signal-to-noise ratio where a classical method (empirical conditional distributions, i.e., counting) can capture most of the signal. Then check whether the model captures that signal too. If you only test on high-noise processes, any model looks comparable to SOTA because there's little signal to differentiate them. Also testing on a range of signals with different signal-to-noise ratio would allow us to know how much of the signal the model captures in different scenarios.