A backtest simulates how a rule would have behaved historically. It is an experiment: given these rules, this data and these assumptions about costs and fills, what would have happened?
Reading a backtest
- Return and volatility over the full period and by year
- Maximum drawdown and time to recovery
- Number of trades — too few means weak evidence
- Turnover and estimated costs
In-sample and out-of-sample
Data used to design and tune a strategy is in-sample. Data kept aside and used only once, at the end, is out-of-sample. Results on out-of-sample data are far more informative, because the strategy was not shaped to fit them.
The asymmetry of evidence
A poor backtest is strong evidence that an idea does not work. A good backtest is weak evidence that it does, because many choices could have produced it by chance. Treat good results as a reason for further testing, not for confidence.
Continue with Seven Backtesting Pitfalls.
Check your understanding
What is out-of-sample data?
Data held back during development and used only to evaluate the final strategy.
Why is a good backtest weak evidence?
Many design choices and variants could have produced good results by chance (overfitting).