Profiling in Go: what pprof shows and what it does not show by default

What is really being asked

The ladder is almost always this one:

1. "How do you find what is slow?" — the warm-up; "pprof" is correct, but you need to start with which profile.
2. "How do you take a profile?" — three ways, and each has its place.
3. "How does flat differ from cum?" — this checks whether the candidate has read a profile themselves.
4. "How do you find a memory leak?" — about the heap profile's two value types.
5. "Why is the block profile empty?" — the question that trips up even people who have read profiles.
6. "Can you profile in production?" — about the cost, and about who can reach /debug/pprof.

The ladder has one spine: a profile is a sample of a particular event, and each profile has its own event, its own rate and its own default. Almost every misreading comes from looking at the wrong profile.

Base: a profile is a sample, not a full count

The profiler does not record every call: that would be far too expensive. Almost every profile is a sample (the exception is the goroutine profile, more on it below): at some rate it takes a sample — a snapshot of the stack at the moment of an event — and adds up identical stacks. What counts as an event depends on the profile:

A profile is taken in three ways, and each has its place:

• from a test or benchmark — go test -bench . -cpuprofile cpu.out, and next to it -memprofile, -blockprofile, -mutexprofile. This is the place for a reproducible load;
• from a running service — import _ "net/http/pprof", and every profile is served under /debug/pprof/. This is the place for real load;
• from code — runtime/pprof: StartCPUProfile, Lookup("heap").WriteTo. Both ways above are built on it; you write it by hand when you need a profile of a specific section.

A profile is read the same way in all cases: go tool pprof — with the -top, -list and -web modes.

And right away — which tool answers which question:

A Junior-level answer to "how do you find what is slow": take a CPU profile — from a benchmark with go test -cpuprofile or from a service with net/http/pprof — and open it in go tool pprof; at the top will be the functions where the program spends the most time.

Mechanism 1: the CPU profile — a sample, in go1.24.7 100 times a second

In runtime/pprof you do not choose the rate: StartCPUProfile does not take it, it is hard-coded. In the source the explanation sits right next to it:

So at the nominal 100 Hz one sample corresponds to roughly 10 ms of CPU time — in aggregate, statistically, not as a stopwatch on each function. Two seconds of one goroutine's work:

  duration of work               2.11 s
  samples in the profile          203
  samples per second               96

Two things follow from this number that are worth being able to say.

The profile may not see short functions. A function that ran for 5 ms over the whole run gets half a sample on average — that is, zero or one. A profile answers "where does the program spend a lot of time", not "was the function called".

"Total samples" in the output is time, not a count. In another run — section 1 of internals.go, a different workload of the same length — go tool pprof -top prints:

  Duration: 2.12s, Total samples = 2070ms (97.61%)

2070 ms is 207 samples converted at the nominal 10 ms. The percentage is CPU time as a share of wall time: 97.61%, because one goroutine uses at most one core. With several busy goroutines the percentage can exceed 100 — the run of bench/goprof/cores.go:

  one busy goroutine:   Duration: 2.11s, Total samples = 1.97s (93.39%)
  two busy goroutines:  Duration: 2.15s, Total samples = 3.89s (181.35%)

On two cores, two goroutines accumulated almost twice as much CPU time as passed on the clock.

Mechanism 2: flat and cum

Here is the table you open a profile for. The handle function does nothing itself — it only calls hashAll (heavy work) and parse (light work). The rows are picked from the full -top listing, so do not read the sum% column here — it is the running flat% in the original order:

2) FLAT VERSUS CUM ON THE SAME CALL CHAIN
─────────────────────────────────────────
        flat  flat%   sum%        cum   cum%
           0     0% 94.20%     1750ms 84.54%  main.hashAll
        50ms  2.42% 77.29%       50ms  2.42%  main.parse
           0     0% 94.20%     1800ms 86.96%  main.handle

• flat — samples in which the function was at the top of the stack: time in its own code.
• cum — samples in which the function was anywhere in the stack: itself plus everything it called.

handle has flat 0 and cum 86.96%: it spends nothing itself, but almost all of the program's time is spent under it. Sorting by flat would not show it; sorting by cum would put it at the top.

Note hashAll too: its flat is 0 as well. The heavy work sits deeper — in crypto/sha256 functions, and those are what stand at the top of -top:

      1310ms 63.29% 63.29%     1310ms 63.29%  crypto/internal/fips140/sha256.blockAVX2

Hence the reading order worth naming in an interview:

• flat — where the processor was executing directly;
• cum — which call paths are responsible for that time.

You need both. Flat shows the hot code — here blockAVX2; cum shows whose decision it is to call it: handle via hashAll. Often the win is not in speeding up the top of flat but in calling less often the path that cum reveals.

Mechanism 3: the heap profile — two different value types

Every heap profile record carries four value types — allocated and in use, in objects and in bytes — and -sample_index picks which one to show. The two main ones answer different questions. The documentation defines the moment the profile reflects:

Two functions: churn allocates a lot and holds nothing, retain allocates a little and holds all of it:

3) HEAP PROFILE: alloc_space VERSUS inuse_space
───────────────────────────────────────────────
  alloc_space — how much was allocated over all time:
    596.83MB 76.83% 76.83%   596.83MB 76.83%  main.churn
     16.19MB  2.08% 99.58%    16.19MB  2.08%  main.retain
  inuse_space — how much is held right now:
  there is no main.churn line in the output
     16.19MB 97.00% 97.00%    16.19MB 97.00%  main.retain

The function that allocated almost 600 MB is entirely absent from inuse_space. That is not a profile error but the point of the two value types:

• inuse_space (the default for heap) — how many bytes of the Go heap are held now. This is where a Go heap leak shows: what grows from capture to capture and does not go away.
• alloc_space (the default for allocs) — everything that was allocated. This is where you hunt pressure on the collector: who creates work for garbage collection even while holding nothing.

But inuse_space is not a universal leak detector:

• inuse_objects — the same view in counts rather than bytes: a leak of many small objects can get lost in bytes;
• a goroutine leak is the goroutine profile (Mechanism 5): in the heap profile it shows at most indirectly, through what those goroutines hold;
• memory outside the Go heap — allocated via cgo, by the kernel, mapped into memory — the Go heap profile does not show at all. If the process grows and inuse_space does not, that is where to look.

The heap profile is a sample too: on average one allocation per runtime.MemProfileRate = 524288 bytes. A small allocation made once may not make it into the profile, while a frequent one gets in in proportion to its size. And a note on units: "MB" in pprof's output is 2²⁰ bytes, that is, MiB.

Mechanism 4: the profiles that are off by default

Four goroutines fight over one mutex — the contention is real. The block and mutex profiles:

4) BLOCK AND MUTEX PROFILES BY DEFAULT
──────────────────────────────────────
  unconfigured:   block 0 records, mutex 0 records
  after SetBlockProfileRate(1) and SetMutexProfileFraction(1):
                  block 3 records, mutex 2 records

Unconfigured, both are empty. This is the most common trap: a service with net/http/pprof, you open /debug/pprof/block — empty, and you conclude "there is no blocking".

They answer different questions:

• block — where and for how long goroutines waited on synchronisation primitives: channels, mutexes, WaitGroup. The stack is of the one waiting;
• mutex — where there was contention on a mutex, and the stack of the one holding it, at Unlock: where to shorten the critical section.

They are turned on like this:

A value of 1 — "record everything" — is convenient for debugging, but not free. On extreme workloads where almost all the work is blocking or locking:

1) BLOCK PROFILE: SetBlockProfileRate(1)
────────────────────────────────────────
  series                                 off          "all"    ratio
  1      ping-pong, 200,000 exchanges      88.7 ms    145.1 ms   x1.64
  2      ping-pong, 200,000 exchanges      82.5 ms    146.6 ms   x1.78
  3      ping-pong, 200,000 exchanges      75.1 ms    142.8 ms   x1.90

2) MUTEX PROFILE: SetMutexProfileFraction(1)
────────────────────────────────────────────
  series                                 off          "all"    ratio
  1      4 goroutines × 200,000 locks      16.5 ms     25.1 ms   x1.52
  2      4 goroutines × 200,000 locks      33.7 ms     54.1 ms   x1.61
  3      4 goroutines × 200,000 locks      29.2 ms     39.3 ms   x1.35

The run of bench/goprof/interfere.go: ×1.64–1.90 for blocking and ×1.35–1.61 for mutexes. On a service where blocking is a small part of the work the share will be smaller, but "every event" mode in production is a deliberate decision, not a default. Usually you set a larger value and get a sample.

And a distinction worth knowing. In go test the -blockprofile flag turns recording on by itself: if -blockprofilerate is not given, every blocking event is recorded. A service has nothing like that — the call to runtime.SetBlockProfileRate must be in your code.

Mechanism 5: the goroutine profile — and why one snapshot is not enough

A hundred goroutines wait on a channel nobody will ever write to:

5) THE GOROUTINE PROFILE FINDS A LEAK
─────────────────────────────────────
  goroutines before: 1, after: 101
  in the profile: 100 goroutines with the same stack:
    main.main.func2              internals.go:246

The goroutine profile is not a sample but a snapshot of every goroutine at once, grouped by stack. One snapshot shows a suspicious accumulation: a hundred goroutines on the same operation. But a hundred goroutines may be honestly waiting for work — a pool of handlers looks the same. What proves a leak is the dynamics:

snapshot A → wait → snapshot B
the same stack has grown and has not come back down

The line internals.go:246 is where they wait; that is where you look for the cause.

Why goroutines leak, and why -race does not catch such a leak, is covered in the lesson on channels. Here is the tool: /debug/pprof/goroutine?debug=1 in a running service, and a comparison of two captures a few minutes apart.

Deeper: what profiling costs and who can reach /debug/pprof

The same workload — 400 calls of handle — with the CPU profile and without, three series:

6) THE CPU PROFILE'S COST TO THE PROGRAM ITSELF
───────────────────────────────────────────────
  series  400 calls without profile   with CPU profile   overhead
  1                     228.6 ms           219.4 ms      -4.0%
  2                     223.3 ms           223.5 ms       0.1%
  3                     222.1 ms           221.0 ms      -0.5%

The overhead changes sign from series to series: it is smaller than the measurement noise. The reason is the same rate: a hundred stack snapshots a second are a small share of the work next to the 800,000 sha256 calls that 400 calls of handle make in ≈222 ms.

Hence the practical distinction people ask about: the CPU profile is usually lightweight enough for a limited capture in production — on request, via /debug/pprof/profile (30 seconds by default; ?seconds=N sets the duration). And -race, which on the measured workload cost ×4.8–5.0 in memory and ×7.6–12 in time, is not kept on across the fleet (that is covered in the lesson on data races).

And these numbers have a boundary: the measurement was taken with GOMAXPROCS=2 on one go1.24.7 workload. No overhead above noise was found on it — that is a measurement result, not a universal number: the cost depends on the workload.

The tool is not neutral. Profiles affect both the program and each other. A memory profile that records every allocation itself becomes a line in the CPU profile:

3) MEMORY PROFILE: MemProfileRate = 1
─────────────────────────────────────
  series                                 off          "all"    ratio
  1      2,000,000 allocations of 64 B     67.0 ms   3488.9 ms   x52.03
  2      2,000,000 allocations of 64 B     59.8 ms   3734.4 ms   x62.43
  3      2,000,000 allocations of 64 B     71.4 ms   3599.8 ms   x50.43

  share of runtime.mProf_Malloc in the CPU profile of the same workload (cum%):
    MemProfileRate by default: not in the profile
    MemProfileRate = 1:        76.64%

The workload is deliberately extreme — nothing but small allocations — but the conclusion is general: a CPU profile taken with MemProfileRate = 1 is 76.64% the work of the memory profiler itself. That is why detailed modes are not all turned on at once, and each profile is read with an allowance for what else was on.

Who can reach /debug/pprof. This is the mandatory second half of the advice "capture in production". Importing net/http/pprof registers handlers on http.DefaultServeMux, and a service that serves it on its working port serves the profiles too — with no access check at all:

  the service registered only /hello and serves DefaultServeMux

  GET /debug/pprof/cmdline           -> 200, arguments: ["-db-password=demo-secret"]
  GET /debug/pprof/goroutine?debug=2 -> 200, "file:line" lines: 53
  GET /debug/pprof/heap              -> 200

The run of bench/goprof/exposure.go; the password in the arguments is made up. What leaks out is the launch arguments, stacks with source paths and memory profiles. That is why /debug/pprof is an internal surface: a separate listener (the example in the package documentation listens on localhost:6060), access only from the internal network or behind authentication, no exposure to the public Internet.

How to answer in an interview

Short answer: take a CPU profile — from a benchmark with go test -cpuprofile or from a service with net/http/pprof — and open it in go tool pprof -top. The profiler samples, so the functions where the program spends the most time end up at the top, while short calls may not get in at all. The rate — 100 times a second in go1.24.7 — is named as an implementation detail, not a property of the language.

That is enough for a correct answer. What follows is what you add if the interviewer digs.

If the interviewer digs deeper

On "how does flat differ from cum" — the dispatcher example. "flat is where the processor was executing directly, cum is which call paths are responsible. A function that only hands out work has flat zero and cum almost the whole profile — I got 86.96%. You look at both: often it is cheaper to call a path less than to speed up the hot function."

On "how do you hunt a memory leak" — the two value types. "inuse_space is what is held as of the last collection; a leak is hunted there by comparing two captures. alloc_space is everything allocated; that is where you look for pressure on the collector. A function that allocated six hundred megabytes and held nothing is not visible in inuse_space at all. A goroutine leak is hunted in the goroutine profile, and memory outside the Go heap the heap profile will not show."

On "why is the block profile empty" — it is off. "Block and mutex are not recorded by default: under real contention I had 0 records. You need to call runtime.SetBlockProfileRate and SetMutexProfileFraction. In go test -blockprofile it is turned on by itself, in a service it is not."

On a goroutine leak — the goroutine profile, two captures. "One snapshot shows an accumulation on one stack; a leak is when that stack grows from capture to capture and does not come back. The stack shows what they are blocked on."

On "can you do it in production" — the cost and access. "The CPU profile — as a rule, yes, as a limited capture on request: on my workload the overhead did not exceed noise, but that is not a universal number. /debug/pprof — only on an internal listener, not outside. The whole fleet under -race — no."

Next they ask

Common misconceptions

Practice

Two problems. Answer first, then check against the real output: in both, the correct answer is taken from a run of the script rather than assigned.

fmt.Println(pprof.Lookup("block").Count())

before := pprof.Lookup("goroutine").Count()
never := make(chan int)
for i := 0; i < 100; i++ {
go func() { <-never }()
}
time.Sleep(50 * time.Millisecond)
fmt.Println(pprof.Lookup("goroutine").Count() - before)

churn(20000)
runtime.GC()
fmt.Println(strings.Contains(string(out), "main.churn"))

0
100
false

4
100
false

0
100
true

4
101
true

Knowledge check