Backtesting Without Fooling Yourself

A backtest answers one question: would these rules have made money over this period? That is worth knowing and it is much easier to get wrong than right, because every common mistake makes the results look better. None of them make the results look worse, which should tell you how they survive.

Lookahead bias

Lookahead bias is using information in a decision that was not available when the decision would have been made. It is the most damaging error here because it is invisible in the output — the equity curve just looks excellent.

Row 5 is the one that catches manual backtesters, and it is close to unavoidable when the future of the chart is on screen. Knowing how it turned out changes what looks like a valid setup, and no amount of intending to be fair prevents it.

Defences

1. Cover the right-hand side of the chart, literally. Use the bar-replay mode most platforms now ship, which reveals candles one at a time.
2. Assume the worst available fill. Entry at the next candle's open, not at the close you observed.
3. Confirm on close, not intrabar, unless your live rules genuinely act intrabar — in which case say so in the rules.
4. Write the decision before advancing, so the record is made without the outcome.

Costs and slippage in the test

A test run on raw price with no costs is measuring a market nobody trades in. Costs are the difference between a strategy and a losing strategy far more often than entries are.

• Spread, on both sides, at a realistic level for the hours you trade — not the broker's advertised best case. Measure it yourself, as Track 2 described.
• Commission, if the account charges it.
• Slippage on market orders and especially on stops. One pip on entries and two to five on stops is a reasonable starting assumption for a major.
• Swap, for anything held overnight, at your broker's actual rate in your direction.

The test worth running

Run the backtest twice: once with your best estimate of costs, and once with costs doubled.

A strategy that survives the doubled version is robust. One that turns unprofitable is a strategy whose entire result is an assumption about execution — and since your execution will be worse than your assumption, that finding is the test working.

Holding data back

Before you look at anything, split the data and put part of it away. This is the single most effective defence against every bias in this lesson, and it only works if it happens first.

1. Split before looking. Choose the boundary by date, not by where the results are convenient.
2. Develop only on in-sample. Iterate freely — that is what it is for.
3. When you believe it is finished, run out-of-sample once.
4. Accept the result. If it fails, the strategy failed. Going back to adjust and re-testing turns the held-out data into more in-sample data, and you no longer have a test at all.

That last point is where most people lose the benefit. The held-out set is usable exactly once per strategy; after that it has informed your choices and cannot be a clean check again. If you must iterate further, you need a third period you have still never seen.

How much data is enough

• At least 100 trades in-sample, and more for a low win rate — Track 5's expectancy lesson set that threshold and it applies here unchanged.
• Several market conditions. Two years of a single trend measures the trend. Include a range, a crisis, a quiet stretch.
• Enough out-of-sample trades to mean something. Thirty is the bare minimum for the result to carry any weight.

Logging every change

During development you will change the rules many times. Each change is a decision made with the data in front of you, and the cumulative effect is what lesson 6 calls curve fitting.

The log from the previous lesson, applied to testing, adds one field:

• Count the changes. Thirty edits to a strategy tested on 150 trades means the rules have been fitted to those specific trades, whatever the reasons said.
• Watch for changes with no reason beyond the number improving. That is the signature of fitting rather than developing.
• Note how many trades each change touched. A change affecting 3 of 200 trades that moves the headline number has not improved the strategy; it has removed three losses.

The log is also what makes the out-of-sample result interpretable. Two strategies can both pass with the same figure, and the one that got there in four changes is far more likely to keep working than the one that took forty — which is the argument the next lesson picks up, once the strategy has to survive contact with a live market.