A backtest simulates how a rule would have behaved historically. It is an experiment: given these rules, this data and these assumptions about costs and fills, what would have happened?

Reading a backtest

  • Return and volatility over the full period and by year
  • Maximum drawdown and time to recovery
  • Number of trades — too few means weak evidence
  • Turnover and estimated costs

In-sample and out-of-sample

Data used to design and tune a strategy is in-sample. Data kept aside and used only once, at the end, is out-of-sample. Results on out-of-sample data are far more informative, because the strategy was not shaped to fit them.

The asymmetry of evidence

A poor backtest is strong evidence that an idea does not work. A good backtest is weak evidence that it does, because many choices could have produced it by chance. Treat good results as a reason for further testing, not for confidence.

Continue with Seven Backtesting Pitfalls.

Check your understanding

What is out-of-sample data?

Data held back during development and used only to evaluate the final strategy.

Why is a good backtest weak evidence?

Many design choices and variants could have produced good results by chance (overfitting).