Backtesting as a Test Harness

A backtest is an integration test against a mock of the market. That mock is written by an optimist, and the errors are not randomly distributed - they are almost all in your favour.

This chapter is about building one that lies less, and about the thing nobody does: testing the test harness.

Two architectures

Vectorised. Compute signals across the whole series with array operations, then compute returns. Fast - you can sweep thousands of parameter sets in minutes. And structurally prone to lookahead, because every row of the array can see every other row. A single misaligned .shift() and you're trading on tomorrow's close.

Event-driven. Iterate through time. At each step the strategy receives only what was available then, emits orders, and a simulated broker fills them on subsequent data. Slower by orders of magnitude. Correct by construction, if you enforce the interface.

Use vectorised for coarse exploration, event-driven for anything you'd risk money on. And when the two disagree - they will - the discrepancy is a bug, and it's usually in the vectorised one.

Make lookahead structurally impossible

The critical design decision: do not rely on discipline to avoid lookahead. Make it unrepresentable.

The strategy should never receive the full DataFrame. It receives a view that cannot return future data:

class MarketView:
    """A window onto history that physically cannot see the future."""

    def __init__(self, data, cursor: int):
        self._data = data
        self._cursor = cursor  # index of the most recent CLOSED bar

    def history(self, field: str, n: int = 1):
        start = max(0, self._cursor - n + 1)
        return self._data[field][start : self._cursor + 1]

    def latest(self, field: str):
        return self._data[field][self._cursor]

    # There is deliberately no method that returns data beyond _cursor.


class Strategy:
    def on_bar(self, view: MarketView) -> list[Order]:
        closes = view.history("close", 21)
        ...

The strategy author now cannot commit lookahead without deliberately reaching around the interface. This is the same principle as making illegal states unrepresentable in a type system, and it's worth the extra structure - lookahead bias is the single most common cause of a backtest that looks too good.

Fill models

How you fill an order is where most of the optimism lives. Build three and run all of them:

Optimistic - filled at the signal bar's close, no costs. Useless as a result, useful as an upper bound.

Realistic - filled at the next bar's open, plus commission, plus a slippage estimate. Your working model.

Pessimistic - filled at the worst price in the next bar's range, with doubled cost assumptions.

If your edge only exists under the optimistic model, you don't have an edge - you have an artefact of the fill assumption. The gap between optimistic and pessimistic is a direct measure of how much of your P&L is assumption rather than measurement. On short timeframes with wide stops, that gap is frequently larger than the entire result.

The intrabar ambiguity

When a bar's range contains both your stop and your target, the backtester must guess the order of events. As established in Chapter 2, that information is not recoverable from the bar.

Three options, in ascending order of goodness:

1. Assume the worst - the stop hit first. Conservative, biased against you, safe.
2. Drop to a finer timeframe to resolve the sequence. TradingView calls this Bar Magnifier; in Python you simulate on 1-minute data while signalling on hourly.
3. Avoid the situation - use exits that can't be ambiguous within a bar.

What you must not do is accept the default assumption without knowing what it is. Most backtesters resolve ties in a direction that flatters the strategy, and with wide ATR stops this affects a substantial fraction of your exits.

Test the backtester itself

This is the chapter's real point, and the thing that separates an engineer from a trader with a Python book.

You would never trust an untested piece of financial software. Your backtester is an untested piece of financial software. Write tests for it.

Known-answer tests. Feed synthetic data with an arithmetically knowable outcome and assert exact equality:

def test_straight_line_long():
    """Price rises 1.0 per bar. A long held 10 bars must make exactly 10.0."""
    prices = np.arange(100.0, 120.0, 1.0)
    result = run_backtest(BuyAndHoldFor(10), prices, commission=0, slippage=0)
    assert result.net_pnl == pytest.approx(10.0)


def test_costs_are_actually_charged():
    """Flat price, enter and exit immediately. P&L must equal minus the costs."""
    prices = np.full(50, 100.0)
    result = run_backtest(EnterExitEveryBar(), prices, commission_pct=0.1, slippage=0)
    expected = -result.trade_count * 2 * 100.0 * 0.001
    assert result.net_pnl == pytest.approx(expected)


def test_pnl_conservation():
    """Sum of individual trade P&Ls must equal the change in equity. Always."""
    result = run_backtest(SomeStrategy(), realistic_data)
    assert sum(t.pnl for t in result.trades) == pytest.approx(
        result.final_equity - result.initial_equity
    )

That second test catches an entire family of bugs. Costs that are silently dropped are common, and they produce a backtest that's wrong by exactly the amount that matters.

The null strategy test. Run random entries with your exit logic and your cost model over many seeds:

def test_random_entries_lose_money():
    results = [
        run_backtest(RandomEntry(seed=s), data, realistic_costs)
        for s in range(200)
    ]
    mean_pnl = np.mean([r.net_pnl for r in results])
    assert mean_pnl < 0, "Random entries are profitable - the harness is broken"

Random entries with realistic costs must lose approximately the cost of trading. If they don't, one of three things is true, and all of them are worth knowing:

1. Your cost model is broken or not being applied.
2. You have lookahead somewhere.
3. Your exit logic is the entire edge, and your entry signal contributes nothing.

That third possibility is not a bug, and it's one of the more useful discoveries you can make about a strategy.

Determinism. Seed every random number generator. Same code plus same data plus same config must produce byte-identical results, or you cannot attribute a change to a cause.

What a backtest cannot tell you

Even a well-built one is silent on:

• Whether the market regime that generated your edge still exists.
• Whether you will actually follow the system through a 20% drawdown.
• Whether your broker will be functional during the volatility that triggers your best trades.
• Whether the liquidity you assumed will be there when everyone wants out.

A backtest is a necessary filter, not evidence of profitability. Its highest use is rejecting strategies cheaply. It never promotes one to "works" - only to "not yet eliminated."