Production Engineering

This is your home turf, and it's where careful work pays direct, uncomplicated dividends. It's also the part where trading systems differ from ordinary services in one crucial respect: a bug does not cause an error page, it causes a position.

The exchange is the source of truth

Everything else - your database, your in-memory state, your last known position - is a cache. Design accordingly:

Reconcile on every startup and periodically thereafter. Fetch positions and open orders from the exchange, compare against local state, and treat any divergence as a halt condition rather than something to auto-correct. Auto-correcting a divergence you don't understand is how a small bug becomes a large one.

async def reconcile(exchange, local_state, tolerance=1e-8) -> list[str]:
    """
    Compare local beliefs against exchange truth. Any divergence halts trading.
    Do NOT auto-correct: a divergence means one of your assumptions is wrong,
    and acting on wrong assumptions at speed is the failure mode to avoid.
    """
    discrepancies = []

    remote_positions = await exchange.fetch_positions()
    remote_by_symbol = {p.symbol: p.qty for p in remote_positions}

    all_symbols = set(remote_by_symbol) | set(local_state.positions)
    for symbol in all_symbols:
        remote_qty = remote_by_symbol.get(symbol, 0.0)
        local_qty = local_state.positions.get(symbol, 0.0)
        if abs(remote_qty - local_qty) > tolerance:
            discrepancies.append(
                f"{symbol}: local={local_qty} remote={remote_qty}"
            )

    remote_orders = {o.client_order_id for o in await exchange.fetch_open_orders()}
    local_orders = set(local_state.open_orders)
    for orphan in remote_orders - local_orders:
        discrepancies.append(f"orphan order on exchange: {orphan}")
    for ghost in local_orders - remote_orders:
        discrepancies.append(f"order we think is open but isn't: {ghost}")

    return discrepancies

Position state must be recoverable from the exchange, not from your local database. Your process will die mid-position. If recovery depends on local state that may be stale or corrupt, recovery is unreliable exactly when it matters.

Idempotency

Networks partition. Requests time out after the exchange received them. Every order carries a client-generated ID derived deterministically from the decision that produced it, so a retry is recognised as a duplicate rather than executed twice:

import hashlib

def client_order_id(strategy_id: str, symbol: str, bar_timestamp: int,
                    side: int, sequence: int) -> str:
    """
    Deterministic from the decision, not from wall-clock time. Retrying the
    same decision produces the same ID, so the exchange rejects the duplicate.
    """
    raw = f"{strategy_id}:{symbol}:{bar_timestamp}:{side}:{sequence}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

A timeout is not a failure. It is an unknown, and the only safe response is to query state rather than to retry blindly.

The silent killer: stale data

The failure that catches people is not a crash. A crash is loud and you'll notice. The dangerous failure is a data feed that stops updating while the connection stays open.

Your strategy sees a frozen price. Depending on its logic, it does nothing while the market moves 5%, or it acts on a stale value with full confidence. Neither produces an error.

Every data feed needs a staleness watchdog that halts trading when the last update exceeds a threshold. Treat data age as a first-class health metric alongside process liveness. The same applies to clock drift - sync via NTP and alert on drift, because signal timing and exchange timestamps depend on it.

Kill switch

One mechanism that flattens all positions, cancels all orders, and refuses further trading until manually re-enabled. Requirements:

• Reachable from your phone. Failures do not respect your schedule.
• Independent of the trading process. A kill switch inside the process that's misbehaving is not a kill switch.
• Tested. Run it in production, on purpose, on a small position. An untested kill switch is a comment.

Automatic triggers worth wiring in: daily loss limit breached, reconciliation discrepancy, data staleness beyond threshold, order rejection rate spike, unexpected exception in the strategy loop.

API key hygiene

Small section, large consequences. Withdrawal permissions off, always. Trading permissions only. IP allowlist where the venue supports it. Separate keys per environment so a leaked development key cannot touch production. Keys in a secret store, never in the repository, and rotate them on any suspicion.

Deployment

• Never deploy while holding a position. Flatten, deploy, resume.
• Shadow mode first - run the new version alongside the old, logging what it would have done, and diff the decisions. Any unexplained divergence is a bug.
• Config is versioned code, and every result is tagged with the config hash from Chapter 5.
• Roll back by flattening, not by reverting mid-flight. A half-deployed trading system holding positions from two different logic versions is a state you do not want to reason about.

Observability

Log every decision with its inputs, not just its output. When a trade looks wrong three weeks later, you need to reconstruct what the strategy saw - the indicator values, the position state, the account equity - not merely that it bought.

Alert on: process liveness, data staleness, reconciliation discrepancies, order rejection rate, drawdown thresholds, and unusual trade frequency in either direction. Trading far less than expected is as much a signal of breakage as trading far more, and it's the one nobody alerts on.