Comment

Comment by traK6Dcm

What people are complaining about is not the overfitting, but the unrealistic assumptions in the backtest. In the real world there is slippage, latencies/jitter, special market open regimes, hidden orders, market impact, front-running, variable fees, and all kinds of other complexities. Their transaction costs are apparently also an unreasonable assumption. Sophisticated simulators used in professional trading firms can account for such things to some extent, but most academic papers conveniently ignore these complexities and just assume they can trade at whatever price the data tells them. It's completely unrealistic.

To answer your original question about overfitting, they can still overfit to test data by running a lot of experiments with different hyperparameters, architectures and parts of the data, and only report what has worked. There are also more complex ways that test data can leak into training data (see the book Advances in financial ML for a good overview). You can already see this is likely the case just from the variance in their results and trades. They also don't compare to baselines. It's not unlikely that the results are just random and they fail to report those experiments that didn't work. Of course, you cannot prove this without having an exact log of all things they ever did to the data. But again, that's not the main issue here.

parent traK6Dcm 6 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton