Making It Honest

You have a backtester. It produces numbers. The numbers are almost certainly too good, and this final chapter is about proving that to yourself.

Test the harness, not just the strategy

From the other book's Chapter 7. Three tests, in order of importance:

func TestStraightLine(t *testing.T) {
	// Price rises exactly 1.00 per bar. A long held 10 bars,
	// no costs, must make exactly 10.00 per unit.
	bars := linearBars(100.0, 1.0, 20)
	engine := &Engine{Costs: NoCosts{}, Sizer: OneUnit{}, Equity: 10000}

	result, err := engine.Run(BuyAndHoldFor(10), bars)
	if err != nil {
		t.Fatal(err)
	}
	got := result.FinalEquity - result.InitialEquity
	if math.Abs(got-10.0) > 1e-9 {
		t.Errorf("got %f, want 10.0", got)
	}
}

func TestCostsCharged(t *testing.T) {
	// Flat price, enter and exit repeatedly.
	// P&L must equal exactly minus the total commission.
	bars := flatBars(100.0, 50)
	costs := PercentCosts{Commission: 0.001}
	engine := &Engine{Costs: costs, Sizer: OneUnit{}, Equity: 10000}

	result, _ := engine.Run(EnterExitEveryBar{}, bars)

	want := -float64(len(result.Trades)) * 2 * 100.0 * 0.001
	got := result.FinalEquity - result.InitialEquity
	if math.Abs(got-want) > 1e-6 {
		t.Errorf("got %f, want %f", got, want)
	}
}

func TestPnLConservation(t *testing.T) {
	// The sum of trade P&Ls must equal the change in equity. Always.
	result, _ := defaultEngine().Run(someStrategy(), randomBars(1000))

	var sum float64
	for _, tr := range result.Trades {
		sum += tr.PnL()
	}
	// (net of commissions, which the engine also subtracts)
	if math.Abs(sum-expectedGross(result)) > 1e-6 {
		t.Errorf("P&L does not reconcile")
	}
}

The second test catches silently-dropped costs, which is a bug that makes every result you've ever produced wrong by exactly the amount that matters.

The null test

The most important test in the book:

func TestRandomEntriesLoseMoney(t *testing.T) {
	bars := loadTestBars()
	costs := PercentCosts{Commission: 0.001, Slippage: 0.0002}

	var total float64
	const runs = 200

	for seed := 0; seed < runs; seed++ {
		engine := &Engine{Costs: costs, Sizer: RiskBased{Pct: 1.0}, Equity: 10000}
		result, err := engine.Run(NewRandomEntry(int64(seed)), bars)
		if err != nil {
			t.Fatal(err)
		}
		total += result.Summary().Expectancy
	}

	if avg := total / runs; avg >= 0 {
		t.Errorf("random entries have expectancy %.4fR - the harness is broken", avg)
	}
}

Random entries with realistic costs must lose approximately the cost of trading. If they don't, one of three things is true:

Your costs aren't being applied.
You have lookahead somewhere.
Your exit logic is the entire edge, and the entry signal contributes nothing.

The third isn't a bug, and it's one of the more useful things you can discover about a strategy.

The three fill models

type OptimisticCosts struct{}    // fill at the signal price, no costs
type RealisticCosts struct{}     // next open, commission, slippage
type PessimisticCosts struct{}   // worst price in the bar, doubled costs

Run all three. If your edge only survives the optimistic model, you don't have an edge - you have an artefact of the fill assumption. The gap between optimistic and pessimistic measures how much of your P&L is assumption rather than measurement.

What you have, and what you don't

You have a correct simulator. That is a real achievement and it is not a profitable strategy.

Your backtester still cannot tell you whether the market regime that produced your result still exists, whether you'd actually follow the system through a 20% drawdown, whether the liquidity you assumed will be there, or whether you've fitted noise by trying two hundred parameter combinations and reporting the best.

That last one is the big one, and it's arithmetic rather than opinion: test 200 configurations of a strategy with no edge, and the best will look excellent purely because you took the maximum of 200 noisy draws.

Where to go next

Everything from here is in the other book:

Chapter 4 - expectancy, and why a +0.1R edge needs 400 trades to detect
Chapter 6 - real data, and every way it lies
Chapter 8 - overfitting, plateaus versus peaks
Chapter 9 - out-of-sample discipline, walk-forward, permutation tests
Chapter 10 - position sizing derived from a drawdown budget
Chapters 11-15 - where edges plausibly come from, and their payoff distributions
Chapter 17 - running this in production without losing money to your own infrastructure

What you actually learned

If you worked the exercises, you now know: types, control flow, functions, errors as values, slices and their sharing semantics, ring buffers, hash tables from the inside, structs and memory layout, pointers, sorting and binary search, heaps, interfaces and implicit satisfaction, packages, file and format parsing, table-driven tests, fuzzing, coverage, timezones and DST, goroutines, channels, worker pools, mutexes, and the race detector.

That is a genuine working knowledge of Go and a solid chunk of practical computer science. It happens to have arrived through market data, and the same material would carry you through a web service, a compiler, or a container runtime.

The trading was the excuse. The programming was the point.

And one thing worth carrying forward from the domain: the instinct, when handed a number, to ask how many things were tried before this one was reported. That question is Chapter 8 of the other book, and it applies to benchmarks, A/B tests, model evaluations, and performance claims - everywhere an engineer is given a result and asked to believe it.

End of The Machine and the Market.