Over-Optimisation and Curve Fitting

Here is the uncomfortable finding that closes this track: among strategies you develop, the one with the best backtest is usually not the one that performs best live. Often it is among the worst. The reason is not bad luck — it is that the process of making a backtest better is, past a point, the process of fitting it to noise.

Degrees of freedom

Every parameter you are free to choose is a degree of freedom, and each one lets the strategy bend closer to the specific history you tested on.

Count them honestly, because they hide:

With ten parameters and 150 trades you can fit almost any history. The strategy will show an excellent backtest, and what it has learned is the particular sequence of trades in your sample rather than anything about markets.

A rough guide

• Under 5 parameters is defensible.
• 5 to 8 demands out-of-sample evidence.
• Above 8 is very likely fitted, whatever the test says.
• At least 20 to 30 trades per parameter in-sample, as a minimum standard.

Peaks versus plateaus

The single most useful diagnostic here costs nothing: after finding a parameter value, test its neighbours.

P looks better and is worthless. Its result exists at one setting and vanishes on either side, which means the market would have to repeat its past behaviour to within one parameter step. Q is flat across the whole neighbourhood — the edge does not depend on the exact number, which is what a real edge looks like.

1. Prefer the middle of a plateau to the top of a peak, even when the peak tests higher.
2. Test at least two steps either side of anything you choose.
3. Treat a sharp peak as evidence of fitting, not as a discovery.
4. Round your parameters. 14, 20, 50 rather than 13, 21, 47. Numbers that look tuned usually are.

Out-of-sample as a check, not a tuning set

The previous lesson set aside 30% of the data. Its value is entirely in being unseen, and it is spent the moment you look at it.

The pattern that destroys it, which nearly everyone runs at least once:

1. Develop on in-sample. Looks good.
2. Run out-of-sample. Disappointing.
3. Adjust the rules. Re-run out-of-sample. Better.
4. Adjust again. Better still.

By step 4 the held-out data has informed four decisions, which makes it in-sample data with extra steps. The result now looks like an honest out-of-sample test and is nothing of the kind — and that is worse than having no test, because it carries false confidence into a live account.

• One run per strategy. If it fails, the strategy failed.
• If you must iterate, you need a third period that has still never been seen, and the same rule then applies to it.
• Walk-forward analysis is the disciplined version: develop on a window, test on the next, roll forward, repeat. It is more work and it is the only approach that survives repeated iteration.
• Expect out-of-sample to be worse than in-sample. Somewhat worse is normal. Dramatically worse means fitting.

Knowing when to stop

The hardest part is that improving a backtest always feels like progress. The output goes up; the work seems to be paying off. Fitting and developing are indistinguishable from the inside, which is why you need external stopping rules.

1. Decide the parameter budget before you start, and do not exceed it.
2. Cap the number of iterations — twenty changes on one strategy is a reasonable ceiling, and the log from lesson 3 is how you count.
3. Every rule needs a mechanism. If you cannot say why it should work, it does not go in, whatever it does to the result.
4. Accept a worse backtest for a simpler strategy. Between +0.25R with four parameters and +0.40R with twelve, take the four.
5. Stop when it is good enough, which means positive expectancy, an acceptable drawdown, and rules you will actually follow. A better backtest past that point buys nothing you can spend.

The honest summary of this track

Most of a strategy's real performance comes from things this track spent little time on — position sizing, costs, and whether you execute the rules as written. Parameter tuning is the most engaging part of strategy development and close to the least productive, which is precisely why it absorbs so much of the time people give it.

A simple strategy with four parameters, positive expectancy, a drawdown you can sit through and a deviation rate under 5% will outperform a beautifully optimised one you abandon in month three. That is the whole lesson, and it is the only defensible reason to stop optimising while the number is still going up.