Why Good Backtests Often Fail Live

The short answer

A backtest is a search, not an experiment. If you try enough variations of a strategy, one of them will look excellent on your sample purely because you tried a lot of variations — and that one is the one you will pick, because it looks excellent. Then the simulator that produced the result also simplified several things about real trading, in ways its own manual lists.

So a strategy that performed beautifully on historical data and then lost money live has not necessarily been unlucky. More often it was never measured in the first place. It was selected — and selecting a strategy on past data and then reporting how it did on that same past data is a circular exercise with a known failure mode.

Two separate things go wrong. The first is statistical and applies to every backtest anywhere. The second is mundane and specific to the software you ran it in. Both are documented, and neither requires taking anyone's word for it.

The statistical problem: you found it, you did not discover it

In 2014, David Bailey, Jonathan Borwein, Marcos Lopez de Prado and Qiji Jim Zhu published a paper in the Notices of the American Mathematical Society on exactly this. Their finding, in their words:

"High simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations, a practice we denote 'backtest overfitting'."

Note "relatively small". This is not a warning about industrial-scale data mining. Changing a lookback period, nudging a stop, trying three indicators instead of two — ordinary tinkering is enough. And the relationship is monotonic:

"The higher the number of configurations tried, the greater is the probability that the backtest is overfit."

Here is the part that makes published backtests nearly impossible to assess from outside:

"Most financial analysts and academics rarely report the number of configurations tried for a given backtest."

Without that number you cannot judge the result, because the result's credibility depends almost entirely on it. The same equity curve means something very different if it was the first thing tried versus the best of four hundred. The authors are blunt about the consequence:

"Investors can be easily misled into allocating capital to strategies that appear to be mathematically sound and empirically supported by an outstanding backtest."

And the outcome is not simply that the edge disappears:

"Backtest overfitting leads to negative expected returns out-of-sample, rather than zero performance."

That is worth sitting with. An overfit strategy is not a coin flip once it meets new data. It can be systematically worse than doing nothing, because the configuration was tuned to features of the sample that do not recur.

The same pattern shows up in machine-learning research

This is not confined to hand-built rules. A 2023 benchmark study examined fifteen state-of-the-art deep-learning models for stock price trend prediction on limit-order-book data, using an open-source framework the authors built so that every model was evaluated the same way. Their result:

"Our extensive experiments reveal that all models exhibit a significant performance drop when exposed to new data, thereby raising questions about their real-world market applicability."

All fifteen. Keep the scope in mind — this is short-horizon prediction from order-book data, not a verdict on every model and every market, and we are not going to stretch it into one. But it is the same shape of finding: performance that exists under the conditions it was developed in, and degrades on data the model had not seen.

The mundane problem: what the simulator left out

The second failure is not statistical at all. The tester simplified things, and MetaQuotes documents which ones. On spread:

"During testing, the spread is not modeled but is taken from historical data."

Spread is a cost you actually pay, and in a test it is a historical record rather than something reconstructed for the conditions your strategy would have created. On clocks:

"During testing, the local time is always equal to the server time."

Any logic that depends on the gap between your clock and the broker's is untested. The documentation also notes that graphical objects are not plotted during testing, that global variables are "emulated during testing, but they are not related to real global variables of the platform", and that in the pips profit-calculation mode, "margin control is not performed".

Network delay deserves particular attention, because it is easy to assume it is handled. MetaQuotes describes it as something the tester "enabled the emulation of ... in order to provide close-to-real conditions for testing", offered with selectable modes. In other words it is a setting. A default run is not a realistic-latency run unless you made it one.

One honest limitation of this section: MetaQuotes does not publish a statement that slippage, requotes, commission or tick-data quality go unmodelled. We looked. So we are not going to tell you the tester ignores execution costs — only that these specific simplifications are the vendor's own documented words, which is a stronger claim than a general complaint.

What to do with this

The useful response is not to abandon backtesting. It is to treat a backtest as evidence about a search process rather than a measurement of a strategy. This applies whichever of the kinds of tool on sale produced the result — a hand-written rule set and a fitted model are both selected against history.

Consider a hypothetical, with invented figures purely to make the shape clear: a vendor tries 4 configurations, keeps the best, and shows you its curve. Now suppose instead they tried 400 and showed you that same curve. Nothing about the picture changes, but the two situations are not remotely equivalent — and the picture alone cannot tell you which one you are looking at. That is the whole problem in one image, and it is why the number of attempts matters more than the result.

Ask how many configurations were tried. If a vendor cannot or will not say, that is the answer to the question of how much the number should move you. Bailey and colleagues make this the central point, and it applies equally to your own work.

Hold back data you never look at. Not as a formality, and not data you peek at and then retune against — once you have tuned on it, it is sample.

Read the tester's manual for what it did not do. For MetaTrader 5 that list is above, and it is short enough to check against your own strategy's assumptions.

Be most suspicious of the best result. It is the one most likely to owe its appearance to the number of attempts, by exactly the mechanism the paper describes.

A backtest that survives honest scrutiny is still not a promise. But a backtest nobody can tell you the search history of is not information at all. How we source claims like these is set out in our editorial policy; the risk disclosure covers what this publication does and does not offer.