{adamcoding}
← The Machine and the Market
Part III
16
Chapter 16

Reading the World

The problem

prices.csv exists on disk. You need []Bar in memory.

Opening a file

go
package market

import (
	"encoding/csv"
	"fmt"
	"os"
	"strconv"
	"time"
)

func LoadCSV(path string) ([]Bar, error) {
	f, err := os.Open(path)
	if err != nil {
		return nil, fmt.Errorf("opening %s: %w", path, err)
	}
	defer f.Close()

	r := csv.NewReader(f)
	rows, err := r.ReadAll()
	if err != nil {
		return nil, fmt.Errorf("reading %s: %w", path, err)
	}
	if len(rows) < 2 {
		return nil, fmt.Errorf("%s: no data rows", path)
	}

	bars := make([]Bar, 0, len(rows)-1)
	for i, row := range rows[1:] {          // skip the header
		bar, err := parseRow(row)
		if err != nil {
			return nil, fmt.Errorf("%s line %d: %w", path, i+2, err)
		}
		bars = append(bars, bar)
	}
	return bars, nil
}

Three things worth noticing.

`defer f.Close()` runs when the function returns, by any path - normal return, early error return, even a panic. Put it immediately after the successful open and you can never forget it.

`i+2` in the error. i starts at 0, we skipped a header, so line numbers in the file start at 2. When a file has 50,000 rows and one is malformed, the difference between "line 34,912" and "some line" is an afternoon.

`make([]Bar, 0, len(rows)-1)` - Chapter 7's preallocation, now that we know the count.

Parsing a row

go
func parseRow(row []string) (Bar, error) {
	if len(row) < 6 {
		return Bar{}, fmt.Errorf("expected 6 columns, got %d", len(row))
	}

	ts, err := time.Parse(time.RFC3339, row[0])
	if err != nil {
		return Bar{}, fmt.Errorf("timestamp %q: %w", row[0], err)
	}

	nums := make([]float64, 4)
	for i, name := range []string{"open", "high", "low", "close"} {
		v, err := strconv.ParseFloat(row[i+1], 64)
		if err != nil {
			return Bar{}, fmt.Errorf("%s %q: %w", name, row[i+1], err)
		}
		nums[i] = v
	}

	vol, err := strconv.ParseInt(row[5], 10, 64)
	if err != nil {
		return Bar{}, fmt.Errorf("volume %q: %w", row[5], err)
	}

	bar := Bar{
		Timestamp: ts.UTC(),
		Open: nums[0], High: nums[1], Low: nums[2], Close: nums[3],
		Volume: vol,
	}
	if err := bar.Validate(); err != nil {
		return Bar{}, err
	}
	return bar, nil
}

Every parse can fail, and every failure says what it was trying to parse. %q puts quotes round the value, which is how you spot a trailing space or an empty field in a log.

ts.UTC() converts on the way in. From here on, everything in the program is UTC - Chapter 18 explains why that rule is worth being absolute about.

Streaming, for files that don't fit

ReadAll loads everything. For a few hundred megabytes of tick data, read one row at a time:

go
func StreamCSV(path string, fn func(Bar) error) error {
	f, err := os.Open(path)
	if err != nil {
		return fmt.Errorf("opening %s: %w", path, err)
	}
	defer f.Close()

	r := csv.NewReader(f)
	r.ReuseRecord = true          // reuse the row slice; big win

	if _, err := r.Read(); err != nil {     // header
		return fmt.Errorf("reading header: %w", err)
	}

	line := 1
	for {
		row, err := r.Read()
		if err == io.EOF {
			return nil
		}
		if err != nil {
			return fmt.Errorf("line %d: %w", line+1, err)
		}
		line++

		bar, err := parseRow(row)
		if err != nil {
			return fmt.Errorf("line %d: %w", line, err)
		}
		if err := fn(bar); err != nil {
			return err
		}
	}
}

Passing a function in is a common Go pattern - the caller decides what to do with each bar, and memory use stays flat regardless of file size.

See It Work: measure the difference

go
func BenchmarkLoadAll(b *testing.B) {
	for i := 0; i < b.N; i++ {
		bars, err := LoadCSV("testdata/prices.csv")
		if err != nil { b.Fatal(err) }
		_ = bars
	}
}

func BenchmarkStream(b *testing.B) {
	for i := 0; i < b.N; i++ {
		count := 0
		err := StreamCSV("testdata/prices.csv", func(bar Bar) error {
			count++
			return nil
		})
		if err != nil { b.Fatal(err) }
	}
}

Run with -benchmem and look at the B/op column. The streaming version allocates a small constant amount; LoadCSV allocates proportionally to file size. On 5,000 rows it barely matters. Generate 5,000,000 rows and run it again - the difference becomes the difference between working and not.

This is the real lesson: the right choice depends on scale, and you can measure the crossover rather than guess it.

See It Work: break it on purpose

Copy prices.csv, corrupt a few rows, and run your loader:

101,not-a-number,99.5,100.2,1500000
2024-03-01T00:00:00Z,100.0,99.0,99.5,100.2,1500000
2024-03-02T00:00:00Z,100.0,102.0,99.5,100.2

Row one has a bad number. Row two has a high below its open - valid CSV, invalid market data, and only your Validate catches it. Row three is short.

You want three different, specific error messages naming the line. Running it gives roughly:

bad number -> prices.csv line 2: open "nope": strconv.ParseFloat: parsing "nope": invalid syntax
high<open  -> prices.csv line 2: high 99.00 below open 100.00 or close 100.00
short row  -> reading prices.csv: record on line 2: wrong number of fields

Note the third one: encoding/csv rejects it before parseRow ever sees it, because the reader enforces a consistent field count against the header by default. That's why the message looks different from the other two - it comes from the standard library, not your code. Your len(row) < 6 guard still earns its place, because it fires when you set r.FieldsPerRecord = -1 to accept ragged input, which you will eventually need for some vendor's export.

If you get a panic or a vague message instead, fix that now - real data has all three of these and worse, and the error message is the only thing standing between you and an hour of guessing.

JSON

Most exchange APIs speak JSON:

go
type APIBar struct {
	Time   int64   `json:"t"`
	Open   float64 `json:"o,string"`
	High   float64 `json:"h,string"`
	Low    float64 `json:"l,string"`
	Close  float64 `json:"c,string"`
	Volume float64 `json:"v,string"`
}

func parseAPIResponse(data []byte) ([]APIBar, error) {
	var bars []APIBar
	if err := json.Unmarshal(data, &bars); err != nil {
		return nil, fmt.Errorf("decoding response: %w", err)
	}
	return bars, nil
}

The backtick strings are struct tags - metadata telling the JSON decoder which field maps to which key. The ,string option handles the widespread exchange habit of sending numbers as quoted strings, which exists to avoid float precision loss in transit. Chapter 2, showing up in a wire protocol.

Note &bars - Unmarshal needs a pointer so it can fill in your variable. Chapter 11.

Exercises

16.1 Implement LoadCSV and load the generated prices.csv. Print the first and last bar and the count.

16.2 Implement StreamCSV and use it to compute the mean and standard deviation of daily returns without holding all bars in memory.

16.3 Corrupt three rows in three different ways. Confirm you get three distinct messages, each naming the line and field.

16.4 Benchmark both loaders with -benchmem at 5,000 and 500,000 rows. At what size does streaming start to matter?

16.5 Write SaveCSV(bars []Bar, path string) error, save a file, reload it, and confirm the round trip is lossless. (Watch the timestamp format and float precision.)

16.6 Write LoadAny(path string) ([]Bar, error) dispatching on the file extension to a CSV or JSON loader.

16.7 Harder. Handle CSVs whose columns are in a different order by reading the header and building a name-to-index map. This is Chapter 9 doing real work, and it's what every serious loader does.

Solutions

16.5 The trap is that writing strconv.FormatFloat(v, 'f', 2, 64) rounds to two decimals, so the round trip is lossy. Use 'f', -1, 64 to write the shortest representation that parses back to exactly the same float64.

16.7

go
func columnIndex(header []string) map[string]int {
	idx := make(map[string]int, len(header))
	for i, name := range header {
		idx[strings.ToLower(strings.TrimSpace(name))] = i
	}
	return idx
}

Then row[idx["close"]]. Check every required column is present before parsing a single row, and report all the missing ones at once.