{adamcoding}
← The Machine and the Market
Part IV
19
Chapter 19

Goroutines and Channels

The problem

You want to test 500 parameter combinations. Each backtest takes two seconds. Serially that's seventeen minutes, during which seven of your eight CPU cores sit idle.

This is what Go was built for, and it's the part where the language stops being "a simpler C" and becomes something with its own argument.

A goroutine is a function running alongside

go
package main

import (
	"fmt"
	"time"
)

func announce(name string) {
	for i := 1; i <= 3; i++ {
		fmt.Printf("%s: %d\n", name, i)
		time.Sleep(100 * time.Millisecond)
	}
}

func main() {
	go announce("first")
	go announce("second")

	time.Sleep(500 * time.Millisecond)
	fmt.Println("done")
}
first: 1
second: 1
second: 2
first: 2
first: 3
second: 3
done

go f() starts f and immediately carries on. Two functions are now running concurrently, and their output interleaves.

Run it several times - the order changes between runs. That unpredictability is the defining property of concurrent code, and it's what Chapter 21 is about.

Goroutines are cheap: a few kilobytes each, managed by Go's own scheduler rather than the operating system. Starting a hundred thousand is reasonable. Starting a hundred thousand OS threads is not.

The time.Sleep is a bug

That time.Sleep(500ms) is a guess. Too short and you cut the goroutines off mid-work; too long and you're wasting time. When `main` returns, the program exits and every goroutine dies instantly, finished or not.

The correct tool is a WaitGroup:

go
package main

import (
	"fmt"
	"sync"
)

func main() {
	var wg sync.WaitGroup

	for _, name := range []string{"first", "second", "third"} {
		wg.Add(1)
		go func() {
			defer wg.Done()
			fmt.Println("working:", name)
		}()
	}

	wg.Wait()
	fmt.Println("all finished")
}

Add(1) before starting, Done() when finished (via defer, so it runs even on an early return), Wait() blocks until the counter reaches zero.

The defer wg.Done() is not optional politeness. Forget it on one path and Wait() blocks forever.

See It Work: your go.mod changes the answer

That program captures name from the loop inside the goroutine. Whether that is correct depends on a line in your `go.mod`, and the difference is silent.

Run the program above, then edit go.mod:

go 1.22        →  working: first / second / third   (in some order)
go 1.21        →  working: third / third / third

Same source, same toolchain, different answers.

Chapter 11 mentioned that Go 1.22 gave each iteration its own loop variable. This is where it bites: under Go 1.21 semantics there is one name variable shared by all three goroutines, and by the time they run, the loop has finished and it holds the last value.

Two things follow:

Check your `go.mod` says `go 1.22` or later. Everything in this part assumes it.

`go vet` will tell you. Run go vet ./... on a pre-1.22 module and it reports loop variable name captured by func literal. If you're reading older code, or a project pinned to an older language version, that warning is real and the code is buggy.

If you need code that's correct on every version, pass the value as an argument instead - the goroutine then gets its own copy, exactly as in Chapter 11:

go
go func(name string) {
	defer wg.Done()
	fmt.Println("working:", name)
}(name)

Channels

Goroutines that can't communicate aren't much use. A channel is a typed pipe between them:

go
package main

import "fmt"

func main() {
	ch := make(chan int)

	go func() {
		ch <- 42          // send
	}()

	value := <-ch         // receive
	fmt.Println(value)    // 42
}

The arrow points the way the data moves. ch <- v sends, <-ch receives.

An unbuffered channel is a rendezvous. The sender blocks until a receiver is ready, and vice versa. It's not just a queue - it's a synchronisation point.

See It Work: watch it block

go
package main

import (
	"fmt"
	"time"
)

func main() {
	ch := make(chan string)
	start := time.Now()

	go func() {
		fmt.Printf("[%6.0fms] sender: about to send\n", ms(start))
		ch <- "hello"
		fmt.Printf("[%6.0fms] sender: send completed\n", ms(start))
	}()

	time.Sleep(500 * time.Millisecond)
	fmt.Printf("[%6.0fms] main: about to receive\n", ms(start))
	msg := <-ch
	fmt.Printf("[%6.0fms] main: got %q\n", ms(start), msg)
}

func ms(start time.Time) float64 {
	return float64(time.Since(start).Milliseconds())
}
[     0ms] sender: about to send
[   500ms] main: about to receive
[   500ms] sender: send completed
[   500ms] main: got "hello"

The sender sat blocked for 500ms waiting for someone to receive. The send didn't complete until the receive happened - they finished at the same instant. That's the rendezvous, made visible.

Now change make(chan string) to make(chan string, 1) and re-run. The send completes immediately, because a buffered channel accepts one value without a waiting receiver.

See It Work: the deadlock message

Delete the go keyword from that program and run it:

fatal error: all goroutines are asleep - deadlock!

Go's runtime noticed that every goroutine was blocked with no possibility of progress, and said so clearly rather than hanging forever. It's one of the friendlier error messages in any language, and you will see it. It almost always means: something is waiting to send with nobody receiving, or waiting to receive with nobody sending.

Ranging over a channel

go
package main

import (
	"fmt"
	"sync"
)

func main() {
	results := make(chan float64)

	var wg sync.WaitGroup
	for i := 1; i <= 5; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			results <- float64(i) * 1.5
		}()
	}

	// Close the channel once every sender is finished.
	go func() {
		wg.Wait()
		close(results)
	}()

	for r := range results {
		fmt.Printf("%.1f\n", r)
	}
	fmt.Println("channel closed, loop ended")
}

range over a channel receives until the channel is closed. That closing goroutine is the standard idiom, and the rules around it are worth memorising:

  • Only the sender closes. Closing from the receiving side causes a panic when a sender tries to send.
  • Closing twice panics.
  • Receiving from a closed channel is fine - it returns the zero value immediately. Use v, ok := <-ch to tell "real zero" from "channel closed."

select

Waiting on several channels at once:

go
select {
case price := <-priceFeed:
	handlePrice(price)
case order := <-orderUpdates:
	handleOrder(order)
case <-shutdown:
	return
case <-time.After(5 * time.Second):
	log.Println("no data for 5 seconds - feed may be stale")
}

Whichever is ready first wins; if several are ready, one is chosen at random.

That last case is the staleness watchdog from Trading Systems for Software Engineers Chapter 17 - the silent failure where a feed stops updating but the connection stays open. In Go it's four lines.

The proverb

Don't communicate by sharing memory; share memory by communicating.

Most languages do concurrency by putting a lock around shared data. Go's preference is to give each piece of data to one goroutine and pass messages instead. No shared state, no locks, no races.

You'll still need locks sometimes - Chapter 21 covers them - but reach for a channel first.

Exercises

19.1 Run the interleaving demo ten times and record the orders you get. How many distinct orderings appear?

19.2 Replace time.Sleep with a WaitGroup and confirm the program neither exits early nor waits too long.

19.3 Run the blocking demo with an unbuffered channel, then with buffer sizes 1 and 10. Record the timings and explain each.

19.4 Cause a deadlock deliberately in three different ways. Read each message.

19.5 Write a pipeline: one goroutine emits bars from a CSV, a second computes a moving average, a third prints crossovers. Connect them with channels.

19.6 Use select with time.After to build a feed watchdog that prints a warning if no bar arrives within one second. Test it by making your producer pause.

19.7 Harder. Use runtime.NumGoroutine() to print the live goroutine count before, during and after a run. Then write a program that leaks goroutines - blocked forever on a channel nobody sends to - and watch the number climb.

Solutions

19.3 Unbuffered: sender waits for the receiver. Buffer of 1: the first send returns immediately, a second would block. Buffer of 10: all sends return immediately until it fills. A buffer decouples the two sides up to its size, and no further.

19.7 Goroutine leaks are a real production problem: a goroutine blocked on a channel that will never receive is never garbage collected, and the count climbing steadily is the symptom. runtime.NumGoroutine() on a monitoring endpoint is the standard early warning.