Goroutines and the Go scheduler: two thousand bytes, preemption, and a GOMAXPROCS that limits the wrong thing

What is really being asked

The ladder is almost always this one:

1. "How does a goroutine differ from a thread?" — testing whether you name a number.
2. "What does it cost?" — a question about honesty: "little" without a number is not an answer.
3. "What is GMP?" — many know the three letters; few know what P means.
4. "What does GOMAXPROCS limit?" — most people answer "the number of goroutines", and that is wrong.
5. "Is the scheduler preemptive or cooperative?" — a question with a date: before Go 1.14 and after.
6. "When does a goroutine leave the CPU?" — about parking and switch points.

The lesson climbs that ladder. Its spine is one sentence: a goroutine is a runtime object, not an operating-system one.

Base: a goroutine is not an operating-system thread

The place to start is what a goroutine is not: nearly every wrong interview answer grows from the same transfer — from the habit of thinking about it as a thread.

An operating-system thread is a kernel entity, and it is expensive: its own record in the kernel, its own stack of megabytes, every switch going through the kernel. Threads are therefore created by the handful, and each one is thought about in advance.

A goroutine belongs to the Go program itself, and three of its properties are worth naming before any machinery:

1. It is not a thread. Underneath a thousand goroutines there are not a thousand threads but noticeably fewer — a handful of them.
2. Starting one costs a single word in the code. go goes in front of an ordinary call, the call starts running on its own, and the program moves on without waiting for it to finish.
3. There can be very many of them. A thousand goroutines is not a load but an ordinary state for a server, and what bounds their number is memory rather than a prohibition.

Hence the question this lesson is about: if there are thousands of goroutines and only a handful of places to run in, who hands those places out, and how? The answer: the Go program itself does. Inside it lives a scheduler that decides which goroutine is on a CPU right now and lays thousands of goroutines over a few threads — without asking the kernel on every switch.

That is already enough to answer the basic interview question, "how does a goroutine differ from a thread". Everything below is about how much cheaper it actually is, who hands out processor time and by what rules, what happens when a goroutine goes off to wait, and why one goroutine cannot hold a place forever.

Mechanism 1: what problem the scheduler solves

Before working through the letters it is worth saying why this machinery exists at all. The scheduler has exactly one problem, and it can be stated without a single Go term:

There are orders of magnitude more runnable goroutines than there are places to run. Places number as many as there are cores; goroutines, as many as the program created.

Everything else in the lesson answers questions that follow from that picture: who decides which G takes a free place; what to do when the goroutine holding a place goes off to wait; and how to stop one goroutine holding a place forever.

Go's answer consists of two decisions. First, make the unit of work cheap, so that thousands and millions are normal. Second, schedule them in user space, without going to the kernel on every switch.

Start with the first, because it is measurable.

A goroutine is a runtime structure, and it can be weighed

An OS thread is a kernel entity: its own table, a stack of megabytes, switching through the kernel. A goroutine is a structure on the Go process's heap with its own stack, and all the switching happens inside the process.

Hence the numbers. Measured over a thousand sleeping goroutines (bench/gosched/practice.go), divided by a thousand:

Two kilobytes is _StackMin from runtime/stack.go, the "minimum size of stack for new goroutines". And it is not a cap but a start: the stack grows as needed, copied to a new place, so there is no need to budget for the maximum in advance.

And the boundary of that number is named by the very place it comes from: _StackMin is a runtime constant, not a promise of the language. So the rule is written not from two thousand but from the order of magnitude: a goroutine costs kilobytes where an OS thread costs megabytes — and it is that gap, not the number, that survives a change of version.

These two numbers deserve a note. The first draft of the measurement counted only HeapAlloc and got ~500 bytes — a number that contradicts the common "a goroutine costs two kilobytes". There is no contradiction: a goroutine's stack is not in HeapAlloc at all, it is accounted separately, in StackInuse. One number without the other gives a wrong answer, and in an interview both are worth naming.

The practical point people want to hear: a million goroutines is gigabytes, not madness. A million OS threads is impossible; a million goroutines is a question of memory.

Mechanism 2: G, M and P — and the whole point is the third letter

The three letters are defined in the scheduler's own comment:

G and M need no explaining. The whole point is P, and it should be explained not as "a processor" but as a right: the right to execute Go code. There are exactly GOMAXPROCS of those rights, and a thread left without one executes no Go code.

Why the extra layer? Because it separates two different limits:

• how many goroutines may compute at once — the number of Ps;
• how many OS threads exist — the number of Ms, and it is larger: when a goroutine goes into a system call its M blocks along with it, hands its P to another thread, and the work carries on.

Which is exactly why the documentation says separately:

Queues and work stealing

The queues are not observable from a program, so no numbers here — but the construction is worth naming, because the scheduler's main property follows from it.

A freed P takes from its own local queue; if that is empty, from the global one; if that is empty too, it steals half of a random neighbour's.

Why local queues at all. A single shared queue would need a lock on every scheduling decision — that is, contention growing with the number of cores. A local queue needs no synchronisation in the common case, and the lock is only needed when stealing.

Work stealing is why load spreads without a dispatcher. Nobody hands goroutines out to processors; an idle P finds work for itself. That has an observable consequence: a goroutine may start running on one core and continue on another, and counting on "its" core is not possible.

Mechanism 3: a system call and I/O are two different paths

The two ways of "going off to wait" are built differently, and the difference gets asked about.

A system call blocks the thread. The M goes into the kernel with the goroutine — the kernel knows nothing about goroutines. So that the stall does not halt the rest of the work, the scheduler takes the P away from that M and hands it to another thread.

Hence the documentation's caveat about GOMAXPROCS: threads blocked in system calls do not count towards the limit. The number of Ms is larger than the number of Ps, and that is how it grows.

Network I/O does not block the thread. Here the netpoller works: the socket is put into non-blocking mode, the goroutine is parked, and the M keeps its P and picks up the next piece of work. When the data arrives, the kernel notifies the runtime (epoll, kqueue, IOCP) and the goroutine goes back on the runnable queue.

That is why a Go server holds tens of thousands of connections on a handful of threads: waiting on the network does not cost a thread. Reading a file does, because there is no such mechanism for disk operations.

The full list of reasons to leave the CPU

The list is short and worth being able to recite:

• it blocked itself — a channel, a mutex, a WaitGroup, network I/O;
• a system call — the M goes into the kernel with the goroutine and hands over its P;
• waiting on the network — the goroutine is parked while the M keeps its P and works on;
• runtime.Gosched() — a voluntary yield;
• preemption by time — roughly every 10 ms, by signal, since Go 1.14;
• garbage collection — for short stop-the-world pauses.

And separately — main waits for nobody:

Returning from main ends the process, and every launched goroutine dies on the spot — with no deferred calls and no chance to finish writing anything. The wait must be explicit: sync.WaitGroup, a channel or errgroup.

Mechanism 4: the scheduler is preemptive — but only since Go 1.14

A question with a date, and both halves matter.

Before Go 1.14 the scheduler was cooperative: it could only switch where the goroutine gave it an opening — a function call, a channel operation, an allocation. A loop without a single call

for i := 0; i < 1e10; i++ {
    x = x*1664525 + 1013904223
}

gave no such point. At GOMAXPROCS=1 it hung the whole program — including garbage collection, which needs to stop every goroutine, and this one could not be stopped.

Since Go 1.14 goroutines are preempted asynchronously, by a signal:

Verified by running it: at GOMAXPROCS=1 the neighbouring goroutine did get the CPU while the compute loop was running — true in the run of bench/gosched/internals.go.

Why this is worth knowing rather than just memorising the date. Because a trace of that era survives in other people's code: runtime.Gosched() sprinkled through loops "so as not to hang the scheduler". Today it is unnecessary there: the scheduler takes the goroutine off without it.

And this is also the model case for how claims in this topic are bounded. The answer "the scheduler is cooperative" was true and stopped being true in a particular version: the scheduler's behaviour is part of the implementation, not a contract of the language. So any claim about it is named together with the version it was checked on.

Mechanism 5: GOMAXPROCS limits the simultaneous, not the existing

This is the most common wrong answer, and a measurement settles it.

The key word is simultaneously. There may be any number of goroutines; what is limited is how many are computing right now. Switch GOMAXPROCS — there are four goroutines in the figure the whole time, and only how many sit on CPUs changes:

The numbers under the figure come from a run of the same compute task:

With one P, four times the work takes four times the time: the goroutines compute in turn. With two, roughly twice. There are four goroutines throughout.

What to know about the default. It equals the number of available cores. In a container with a CPU limit that is often the host's core count rather than the container's limit — and then the runtime creates noticeably more Ps than it is allowed to compute on. The cure is either an explicit GOMAXPROCS or a library that reads the cgroup limits.

And here the boundary is worth naming, because its absence is what produces the surprise on a move. How many threads actually execute Go code is a property of the runtime version and the environment, not of a line in the code: the documentation states outright that this call will go away when the scheduler improves. So the rule is written from the boundary: the number of simultaneously executing threads is read off the running process on the Go version it runs on, rather than derived from the machine's core count.

Deeper: what costs is the parking, not the operation

Two measurements here, and together they give a rule.

First — spawning a goroutine. The same work, one atomic increment:

Eighty-two times looks frightening right up until you say against what: against work of six nanoseconds. That is the price of the spawn plus two switches — and as soon as there is real work inside the goroutine, the spawn's share falls to nothing.

Second — yielding the CPU. The same work, but the goroutine gives up the processor:

The third row matters more than the second: time.Sleep(0) is nearly free because it returns immediately and does not park the goroutine. So what costs is not "talking to the scheduler" but being taken off the CPU and put back.

Hence, too, the price of that runtime.Gosched() sprinkled through loops in old code: about a hundred nanoseconds per call spent on nothing, because since Go 1.14 the scheduler takes the goroutine off without being asked.

Hence a rule that carries over to channels, mutexes and WaitGroup: while a goroutine does not block, the scheduler plays no part in its cost. An uncontended mutex is an atomic operation; a channel with a waiting receiver is a copy. What costs is the waiting, not the operation.

Both measurements share one boundary: they were taken on one machine, two cores and one version of Go. What travels from here is the cause — that coming off a processor and going back on is what costs — rather than "82 times" or "106 ns": your numbers will be your own and the conclusion will be the same.

How to answer in an interview

Short answer: a goroutine is not an operating-system thread but a unit of work belonging to the Go program itself. One word, go, starts it; it costs kilobytes against a thread's megabytes; and which one is on a CPU right now is decided by a scheduler inside the process. That is why a program has thousands of goroutines with a handful of threads, and why GOMAXPROCS limits not how many goroutines exist but how many run at once.

That is enough for a correct answer. What follows is what you add when the interviewer digs.

If the interviewer digs deeper

To "how does it differ from a thread" answer with a number. "A runtime structure with its own stack: two kilobytes of stack plus hundreds of bytes of structures against megabytes for an OS thread; switching inside the process rather than through the kernel."

On GMP, explain P as a right. "A P is the right to execute Go code, and there are exactly GOMAXPROCS of them; a thread without a P executes no Go code. Hence there can be more threads than Ps: one that goes into a system call hands its P to another."

On GOMAXPROCS say "simultaneously". "It limits not how many goroutines exist but how many run at once; with GOMAXPROCS=1 my four goroutines took four times as long as one." "How many goroutines you can create" is the most common wrong answer.

On preemption name the date and the consequence. "Preemptive since 1.14, cooperative before that; a loop without calls used to hang garbage collection."

On cost give both numbers and separate them. "A spawn with a wait is hundreds of nanoseconds, but that is against work of six; what costs is the parking, not the operation: Gosched adds a hundred nanoseconds while Sleep(0) adds almost nothing."

Next they ask

Common misconceptions

Practice

Two problems. Answer first, then check against the real output: in both, the correct answer is taken from a run of the script rather than assigned.

fmt.Println(runtime.NumGoroutine())

before := runtime.NumGoroutine()
go func() {
defer wg.Done()
}()
fmt.Println(runtime.NumGoroutine() - before)

fmt.Println((m2.StackInuse - m1.StackInuse) / n / 100 * 100)
fmt.Println((m2.HeapAlloc - m1.HeapAlloc) / n / 100 * 100)

1
1
2000
500

0
1
8000
2000

1
1
500
2000

2
1
2000
0

Knowledge check