The Data Layer

Every conclusion you reach is downstream of your data. This is the least interesting chapter in the book and the one most likely to be the reason your results are wrong.

The reason it's dangerous is that data problems don't announce themselves. A bug in your strategy logic usually produces something obviously broken. A bug in your data produces a plausible equity curve that is quietly a work of fiction.

What data actually is

Working from finest to coarsest:

Trade data (tick data) - every execution: timestamp, price, size, and sometimes the aggressor side. The ground truth of what happened.

Quote data (L1) - best bid and ask over time. Necessary to know what you could actually have transacted at. Trade data alone tells you where trades happened, not what was available to you.

Order book data (L2/L3) - full depth. Large, expensive, and necessary only if you're modelling execution seriously or doing microstructure work.

Bars (OHLCV) - an aggregation of trades over an interval. What most retail strategies run on, and what most retail traders never think about critically.

That last point deserves unpacking, because a bar is the result of a series of choices someone made on your behalf:

What's the interval boundary? Midnight in which timezone? Exchange local time, UTC, or your vendor's server time?
Is the timestamp the bar's open or close? This is a coin flip across vendors and it's a one-bar shift in your entire dataset. A one-bar shift in the wrong direction is lookahead bias.
Does volume include auctions, block trades, off-exchange prints? Vendors differ. Your volume filter differs accordingly.
How are gaps handled? A period with no trades - is it a missing bar, or a zero-volume bar carrying the previous close?

Two vendors will give you different candles for the same instrument and interval. Neither is wrong. They made different choices. Pick one source and stay on it, because otherwise a "strategy improvement" may just be a data change.

Adjustments: where equity backtests die

Corporate actions mean the raw historical price of a stock is not comparable to today's price.

Splits. A 4:1 split quarters the price without changing anyone's wealth. Unadjusted data shows a 75% crash that never happened. Every momentum strategy will short it.

Dividends. On the ex-dividend date the price drops by roughly the dividend. Unadjusted price series therefore show a systematic downward drift for high-dividend stocks that shareholders never experienced.

The standard fix is back-adjustment: scale historical prices so the series is continuous. This solves the discontinuity and introduces a subtler problem - the adjusted historical prices are not the prices that traded. A stock showing €2.50 in 2015 on your adjusted chart may have actually traded at €10.

That breaks anything referencing absolute price:

Penny-stock filters ("exclude anything under €5") select the wrong universe.
Round-number logic is meaningless.
Tick-size and minimum-increment assumptions are wrong.
Sufficiently aggressive back-adjustment of futures can produce negative historical prices, which will do interesting things to any percentage-return calculation.

And note that adjusted series are revised: today's adjusted history differs from the one you downloaded last year, because dividends have been paid since. Your backtest is not reproducible unless you snapshot the data.

The rule: use adjusted data for return calculations, unadjusted data for anything involving price levels, and know which one you're holding.

Continuous futures

Futures expire, so a multi-year chart is a construction - contracts stitched together. The stitching method is a choice:

No adjustment - leaves a price jump at each roll, which your strategy will read as a real move.
Ratio-adjusted - multiplicative, preserves percentage returns, distorts levels.
Panama / difference-adjusted - additive, preserves absolute moves, can drive prices negative over long histories.

And the roll date itself is a choice: on expiry, on volume crossover, N days before expiry. Different choices produce measurably different backtests of the same strategy. Your vendor made one. Find out which.

Survivorship and point-in-time

Survivorship bias is the most famous data trap and still catches people. If you backtest on "current S&P 500 constituents," you are testing on a list selected for having survived and thrived. The companies that went bankrupt, got delisted, or were acquired after collapsing are simply absent. Your backtest cannot lose money on them because they aren't there.

The general form of the problem is broader: you need to know what the world looked like at each historical moment, not what it looks like now. This is called point-in-time data:

Index membership as of that date, not today.
Fundamentals as first reported, not as later restated. A company's Q3 earnings were reported in November and revised in March; using the revised figure in a November backtest is lookahead.
The report date, not the period end. Q3 data was not available on 30 September.

Point-in-time data is expensive, which is why free-data backtests of fundamental strategies are almost always wrong in the same direction.

Timestamps: the boring one that will get you

Store everything in UTC. Convert only for display. No exceptions.
Know your bar labelling convention and assert it in code.
Daylight saving moves exchange sessions relative to UTC twice a year. A session filter written as fixed UTC hours silently misaligns for half the year.
Distinguish exchange time, vendor receipt time, and your receipt time. They differ, and the difference is what you can't act inside.

Treat your data like input you don't trust

Here's where your instincts are worth more than a trader's. You already validate untrusted input. Do it here:

python

import pandas as pd

def validate_ohlcv(df: pd.DataFrame, expected_freq: str = "1h") -> list[str]:
    """Returns a list of problems. Empty list means the data passed."""
    problems = []

    if not df.index.is_monotonic_increasing:
        problems.append("timestamps not monotonic")
    if df.index.has_duplicates:
        problems.append(f"{df.index.duplicated().sum()} duplicate timestamps")
    if df.index.tz is None:
        problems.append("timestamps are timezone-naive")

    # OHLC internal consistency
    bad_high = (df["high"] < df[["open", "close"]].max(axis=1)).sum()
    bad_low = (df["low"] > df[["open", "close"]].min(axis=1)).sum()
    if bad_high:
        problems.append(f"{bad_high} bars where high < max(open, close)")
    if bad_low:
        problems.append(f"{bad_low} bars where low > min(open, close)")

    if (df[["open", "high", "low", "close"]] <= 0).any().any():
        problems.append("non-positive prices present")
    if (df["volume"] < 0).any():
        problems.append("negative volume present")

    # Gaps against the expected grid
    expected = pd.date_range(df.index[0], df.index[-1], freq=expected_freq)
    missing = expected.difference(df.index)
    if len(missing):
        problems.append(f"{len(missing)} missing bars")

    # Implausible single-bar moves - usually bad ticks, sometimes real
    returns = df["close"].pct_change()
    sigma = returns.std()
    outliers = (returns.abs() > 10 * sigma).sum()
    if outliers:
        problems.append(f"{outliers} bars exceeding 10 sigma - inspect manually")

    return problems

Run this on every dataset before it touches a strategy. Fail loudly.

A caution on the last check: cleaning is itself a decision that can leak bias. If you remove outliers because they look wrong, you may be removing the crashes - precisely the events that determine whether your strategy survives. Flag anomalies for inspection; don't silently delete them. The 2010 flash crash and the 2020 negative oil settlement were real.

Storage and reproducibility

Raw data is immutable. Download it, store it, never modify it in place. All cleaning produces a new derived dataset.
Parquet over CSV - typed, compressed, fast.
Hash your datasets and record the hash with every backtest result. When a result changes and you don't know whether the strategy or the data changed, this is the only thing that saves you.
Snapshot rather than re-download. Vendors revise history. If your backtest re-fetches data each run, it is not reproducible.