Getting Real Data, and Two Tools

Part I used numbers I made up. From here on you need real data and two ways of watching your code behave.

A data generator you can trust

Before real data, you want reproducible data - the same numbers every run, so an exercise that fails tells you something about your code rather than about the random seed.

Put this in generate.go:

package main

import (
	"encoding/csv"
	"fmt"
	"math"
	"math/rand"
	"os"
	"strconv"
	"time"
)

// generateOHLCV produces a deterministic random-walk price series.
// The same seed always gives the same data, which is what makes it
// useful for exercises.
func generateOHLCV(n int, startPrice float64, seed int64) [][]string {
	rng := rand.New(rand.NewSource(seed))
	rows := [][]string{{"timestamp", "open", "high", "low", "close", "volume"}}

	price := startPrice
	t := time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)

	for i := 0; i < n; i++ {
		open := price
		// Daily volatility of about 2%, with a slight upward drift.
		drift := 0.0002
		vol := 0.02
		change := drift + vol*rng.NormFloat64()
		closePrice := open * math.Exp(change)

		high := math.Max(open, closePrice) * (1 + math.Abs(rng.NormFloat64())*0.004)
		low := math.Min(open, closePrice) * (1 - math.Abs(rng.NormFloat64())*0.004)
		volume := int64(1_000_000 + rng.Intn(500_000))

		rows = append(rows, []string{
			t.Format(time.RFC3339),
			strconv.FormatFloat(open, 'f', 2, 64),
			strconv.FormatFloat(high, 'f', 2, 64),
			strconv.FormatFloat(low, 'f', 2, 64),
			strconv.FormatFloat(closePrice, 'f', 2, 64),
			strconv.FormatInt(volume, 10),
		})

		price = closePrice
		t = t.Add(24 * time.Hour)
	}
	return rows
}

func main() {
	f, err := os.Create("prices.csv")
	if err != nil {
		fmt.Println("could not create file:", err)
		return
	}
	defer f.Close()

	w := csv.NewWriter(f)
	defer w.Flush()

	if err := w.WriteAll(generateOHLCV(5000, 100.0, 42)); err != nil {
		fmt.Println("could not write:", err)
		return
	}
	fmt.Println("wrote prices.csv")
}

Run it with go run generate.go and you have 5,000 daily bars. Change the seed for a different market; keep it at 42 and your results match the ones in this book.

defer f.Close() means run this when the function exits, whatever happens. It's Go's answer to "don't forget to clean up," and you'll see it constantly.

Real data, when you want it

Synthetic data is honest about being synthetic. It has no fat tails, no volatility clustering, no gaps - so a strategy that works on it has proved nothing. Once your code runs, feed it something real:

• Binance publishes free historical CSV dumps of spot and futures data. No account needed.
• Stooq offers free daily CSVs for equities and indices.
• Your broker almost certainly lets you export history.

Any of these will hand you a CSV with roughly the columns above. Chapter 16 covers parsing them properly; until then, the generator is enough.

Worth knowing now: the moment you use real data, everything in Trading Systems for Software Engineers Chapter 6 applies - timezones, adjustments, gaps, vendors disagreeing. Real data is messy, and discovering that is part of the point.

Tool one: benchmarks

Go has a built-in benchmark runner, and it removes all the guesswork from "is this faster?"

Create bench_test.go - the _test suffix matters:

package main

import "testing"

func BenchmarkAppendNoPrealloc(b *testing.B) {
	for i := 0; i < b.N; i++ {
		s := []float64{}
		for j := 0; j < 10000; j++ {
			s = append(s, float64(j))
		}
	}
}

func BenchmarkAppendPrealloc(b *testing.B) {
	for i := 0; i < b.N; i++ {
		s := make([]float64, 0, 10000)
		for j := 0; j < 10000; j++ {
			s = append(s, float64(j))
		}
	}
}

Run it:

go test -bench=. -benchmem

BenchmarkAppendNoPrealloc-8    30000    45231 ns/op    386296 B/op    20 allocs/op
BenchmarkAppendPrealloc-8     150000     8104 ns/op     81920 B/op     1 allocs/op

Your numbers will differ; the ratio won't by much. Read it as: nanoseconds per run, bytes allocated, number of allocations. Go figures out b.N itself - how many repetitions it needs for a stable measurement.

This is the tool that turns "I think this is faster" into a fact, and I'll ask you to use it in most of the chapters that follow.

Tool two: printing what's really there

fmt.Printf has verbs that show you the machine rather than the value:

%p is the one people don't know about, and it makes invisible things visible. When you want to know whether two variables share memory or are separate copies, print both addresses and look.