Validation

Chapter 8 described the disease. This chapter is the treatment: a set of procedures for getting an honest estimate of whether an edge is real.

In-sample / out-of-sample

The basic discipline, which almost nobody follows properly:

1. Split your history. Roughly 70% in-sample, 30% out-of-sample, with OOS being the most recent data.
2. Do all development and optimisation on IS. Do not look at OOS. Do not plot it. Do not glance at it.
3. Choose parameters. Lock them.
4. Run once on OOS.
5. If it fails, the strategy failed.

Step 5 is where everyone falls. The temptation to go back, adjust, and re-run is enormous, and yielding to it converts your OOS data into IS data. Do it three times and you've performed full-sample optimisation with extra ceremony.

The useful framing for an engineer: out-of-sample data is a consumable resource. You get to spend it once per strategy. Spend it deliberately.

Walk-forward analysis

The stronger version. Roll the split forward:

|==== IS 1 ====|OOS 1|
       |==== IS 2 ====|OOS 2|
              |==== IS 3 ====|OOS 3|
                     |==== IS 4 ====|OOS 4|

Optimise on each IS window, apply the resulting parameters to the following OOS window, then concatenate the OOS segments. That stitched curve is the closest honest approximation of live performance available from historical data, because every point in it was generated by parameters chosen without seeing it.

Two things it reveals that a single split cannot:

Parameter stability. If your optimal stop multiplier goes 1.2, then 4.5, then 2.0, then 3.8 across windows, the parameter is meaningless and you are fitting noise in every window. Stable parameters across windows are a strong signal; unstable ones are close to conclusive evidence against.

Walk-forward efficiency - OOS performance divided by IS performance. Above roughly 0.5 is respectable. Near zero or negative means your optimisation procedure is actively harmful, which is a genuinely useful thing to learn.

Purging and embargo

A subtlety that catches people doing cross-validation on time series.

Standard k-fold cross-validation assumes samples are independent. Financial data violates this badly: trades overlap in time, features are autocorrelated, and a trade opened just before your split boundary is influenced by data on both sides. Information leaks across the boundary and inflates your validation score.

Two fixes, both from López de Prado's Advances in Financial Machine Learning:

• Purging - remove training samples whose outcome period overlaps the test set.
• Embargo - additionally drop a buffer of samples immediately after the test set, since serial correlation means adjacent data still carries information.

If you're doing anything ML-flavoured on price data, this matters enormously. Naive cross-validation on financial time series produces results that are essentially meaningless.

The permutation test

The cleanest significance test available, and one that should appeal to you immediately because it makes no distributional assumptions at all.

The logic: build a null distribution empirically by destroying the structure your strategy claims to exploit, then see where your real result falls.

def permutation_test(strategy, prices, n_permutations=1000):
    """
    If the strategy exploits real temporal structure, it should beat
    versions of history where that structure has been destroyed.
    """
    real_result = run_backtest(strategy, prices).expectancy

    returns = np.diff(np.log(prices))
    null_results = []

    for _ in range(n_permutations):
        shuffled = np.random.permutation(returns)
        synthetic = prices[0] * np.exp(np.cumsum(shuffled))
        null_results.append(run_backtest(strategy, synthetic).expectancy)

    p_value = np.mean(np.array(null_results) >= real_result)
    return real_result, p_value

Shuffling returns preserves the distribution - same volatility, same fat tails, same drift - while destroying the sequence. Any strategy that depends on temporal structure (which is all of them) should perform worse on shuffled data.

If your real result sits comfortably inside the null distribution, your strategy is not exploiting structure. It's harvesting the return distribution, which you could do more cheaply by holding the asset.

Two cautions. First, this tests one strategy; if you've tested 500 configurations, you need a p-value 500 times more stringent, roughly. Second, the shuffle destroys volatility clustering, which is real and which some strategies legitimately exploit - a block bootstrap that preserves short-range dependence is a more conservative null.

Monte Carlo on trades

Take your trade list, shuffle the order, rebuild the equity curve, repeat a thousand times. The distribution of maximum drawdowns tells you what your realistic worst case looks like.

Your historical maximum drawdown is one draw from that distribution, and it is usually not close to the worst. People calibrate their risk tolerance to the drawdown they saw in the backtest and are then surprised by a larger one, which was always likely.

The limitation: shuffling assumes trades are independent. For trend strategies they aren't - losses cluster in choppy regimes, which is precisely what makes drawdowns painful. Shuffling therefore understates clustering and gives you an optimistic drawdown estimate. Treat it as a floor.

Pre-registration

Borrowed from clinical trials, and the highest-leverage habit in this book.

Before running a test, write down: the hypothesis, the parameters you'll test, the metric you'll judge on, and what result would make you reject the idea. Then run it.

This costs five minutes and prevents the most common failure in quantitative research - running an experiment, seeing an interesting result somewhere you weren't looking, and retroactively deciding that's what you were testing. Your experiment log from Chapter 5 becomes the pre-registration record.

If you can't specify in advance what would falsify your idea, you're not testing it.