Overfitting

Here is the entire problem in one sentence: if you test enough strategies against one dataset, some will look excellent by chance alone, and you will select exactly those.

Not "might." Will. It's arithmetic.

The maths of best-of-N

Suppose you test 1,000 parameter combinations of a strategy with no edge whatsoever. Each produces a Sharpe ratio drawn from a distribution centred on zero with standard deviation around 0.5 (typical for a few hundred trades).

The maximum of 1,000 such draws sits roughly 3.2 standard deviations above the mean - a Sharpe near 1.6. That's a number most people would trade.

You did not find a strategy. You performed a maximum operation over noise and reported the result as a discovery. The reported metric is a biased estimator of the true one, and the bias grows with the number of trials.

This is the same phenomenon as p-hacking in the sciences, and it has the same solution: account for how many things you tried.

Degrees of freedom are a budget

Every tunable parameter is a withdrawal from a limited account of credibility.

A strategy with 3 parameters showing a profit factor of 1.3 is more believable than one with 12 parameters showing 2.1. This is counterintuitive if you're used to systems where more capability is better, and it is not negotiable here.

Your working prior should be: the simplest system that shows an edge is the one most likely to keep showing it.

The hidden degrees of freedom

Here's what almost nobody counts, and it's usually the larger number.

Your budget isn't just the parameters you swept. It includes every choice you made before you started sweeping:

The instrument you picked (out of how many you could have picked?)
The timeframe you settled on (after trying a few?)
The date range in your backtest
The indicator you chose, out of the hundreds available
Every strategy idea you abandoned along the way because it didn't work

That last one is the killer. If you tried eight strategy concepts and are now optimising the one that showed promise, you have already performed a selection over eight candidates before running a single sweep. The promising one may be promising because of noise.

Keep the honest count. It's the number from Chapter 5's lab notebook, and it is the number that determines how much to discount your final result.

Plateaus, not peaks

The single most practical diagnostic. Sweep a parameter across its whole range and look at the shape, not the maximum:

   PLATEAU - structural                 PEAK - coincidence

   PF                                   PF
   1.4 |    ___________                 2.0 |        /\
   1.3 |   /           \                1.5 |       /  \
   1.2 |  /             \___            1.0 |______/    \______
   1.1 | /                              0.5 |
       +--------------------                +--------------------
        1.0   2.0   3.0   4.0                1.0   2.0   3.0   4.0
              atrMultSL                            atrMultSL

A plateau means neighbouring values also work, which means the effect doesn't depend on a specific number. When conditions shift, you slide along the plateau and survive.

A peak means 2.3 works and 2.2 doesn't. No market mechanism makes 2.3 special. You've fitted noise, and live trading will place you on a neighbour.

Always take the centre of the widest acceptable region, never the maximum - particularly when the maximum looks better. For two parameters, build a heatmap; a grid of stop multiplier against target multiplier with expectancy as colour will teach you more in twenty minutes than a month of single runs.

Warning signs

Profit factor above ~2.5 on anything but a small, specific sample.
An unusually smooth equity curve. Real edges are lumpy.
Removing the single best trade collapses the result.
Optimal parameters that are oddly precise (13.7, 2.83) or sit at the edge of the tested range.
Neighbouring parameter values performing much worse.
Optimal values that differ wildly between time periods.
It works on exactly one instrument and one timeframe.
A filter that improved your metrics and also cut trade count by 60%.

That last one generalises into a rule worth memorising: any filter you add will improve in-sample results, because it removes trades and hindsight lets you find rules that remove the losing ones. A filter is only credible if you can state its mechanism before testing it and it survives out-of-sample.

Formal treatments

If you want the rigorous version, two references are worth your time. Bailey and López de Prado's work on the probability of backtest overfitting formalises exactly the best-of-N problem above and provides a way to estimate how likely your selected strategy is to underperform out-of-sample. Their deflated Sharpe ratio adjusts an observed Sharpe for the number of trials that produced it - it turns "I tested 500 configurations" from a vague worry into a specific haircut on your reported metric.

The practical takeaway even if you never implement them: the number of trials must be reported alongside the result, or the result is uninterpretable.