Manual vs Automated Backtesting: Which to Use First
Manual and automated backtesting compared: what each one is good at, where each misleads you, which to start with, and a simple way to use both on one strategy.
Automated backtesting runs a written rule across past data with no human in the loop. Manual backtesting means you step through the past yourself and make each decision. They answer different questions, so most traders end up using both: the script asks whether the rule has an edge, and the replay asks whether you can actually trade it.
The comparison
| Manual | Automated | |
|---|---|---|
| Speed | Slow, tens of trades an hour | Years of data in a minute |
| Needs code | No | Sometimes, or a ready strategy |
| Fuzzy rules | Works fine | The rule has to be exact |
| Your discipline | Tested | Not tested |
| Hindsight risk | High unless the date is hidden | None in the decisions |
| Overfitting risk | Lower | High, because tweaking is so easy |
Where each one misleads you
Manual testing suffers from hindsight. If you can see where the day went, or even recognize the date, your eye finds the trade that worked. Hide the date and only move forward.
Automated testing suffers from overfitting. Change enough settings and something will look great on past data, then fall apart on new days. Keep some days out of the test and check that a small change in a setting doesn't destroy the result. It can also ignore real costs, so check that commission and slippage are counted.
Which to start with
Start with manual if you are new or your idea is still vague. A few dozen trades will teach you what you really mean by your own rule. Then write the rule exactly and run it as a script over far more days than you could ever replay.
A simple way to use both
- Replay 30 trades by hand with the date hidden, to shape the rule.
- Write the rule down so that nothing depends on judgment.
- Run it automatically over as much history as you can and read the sample size, the profit factor and the result by hour.
- Compare with doing nothing. A strategy on a rising market may have made less than simply holding.
- Replay it again by hand until your results are close to the script's. The gap between them is your execution.
ChartTrainer does both in one place. Chart replay steps random days candle by candle with the date hidden and saves every trade to a journal. The backtester runs a script or a ready strategy across many sessions, shows each trade on a chart, and puts buy and hold beside the result. Backtest and practice results don't promise real results.
Questions
- Is manual backtesting better than automated?
- They answer different questions. Automated testing shows whether the rule made money. Manual testing shows whether you can follow it, and it works for ideas too fuzzy to code.
- Can I backtest without coding?
- Yes. Manual backtesting needs only a replay tool and a journal. Many backtesters also come with ready strategies you can change one setting at a time.
- Why do my manual results differ from the script?
- Usually because you skipped trades, entered late or moved stops, or because the rule wasn't as exact as you thought. The difference is useful: it shows what you do under pressure.
- How do I avoid hindsight in manual backtesting?
- Hide the date and the right side of the chart, and step forward one candle at a time. Write your decision before you press next, and don't pick the days yourself.
- Which one is faster?
- A script is far faster once it exists, and can run years of data in a minute. Manual replay is slower, but you can start right away and you learn the market while doing it.