io.Reader and io.Writer: a contract of two methods
Interview questions on io climb a ladder: what a Reader is — how to write a read loop — why Read returned less than asked — how io.Copy copies and how much memory it takes — why bufio and what happens without Flush — what is wrong with defer f.Close(). A run answers each step: a loop that looks at the error before the bytes loses the last chunk; io.Copy sends 100,000 bytes in a single Write if the source has WriterTo; and a function with bufio and defer f.Close() returns nil and leaves an empty file on disk.
Full technical treatment
TL;DR
io.Reader and io.Writer are interfaces with one method each: Read(p)
puts up to len(p) bytes into p, Write(p) writes p and, if it wrote
less than len(p), must return an error. Both return
(n int, err error) — how many bytes went through and what happened. Everything
rests on these two signatures: a file, a network connection, gzip, bufio,
bytes.Buffer, sha256, and io.Copy, which connects any source to any
destination.
Read may return data and an error in the same call — and they must be
handled in that order: first the n bytes, then err. The documentation
allows two endings of a stream: the last chunk with nil and then
(0, EOF) — or the last chunk right together with EOF. A loop with
if err != nil { break } before handling n collects the whole string
"hello, world" on a strings.Reader, and "hello, wor" on a reader of the
second kind: two bytes vanished without a single error.
Beyond that comes what separates real understanding from having read about
it. Read does not have to fill the buffer: with a buffer of 8 the reader
returned 4 bytes, and filling the buffer completely is guaranteed by
io.ReadFull — and on short input it answers io.ErrUnexpectedEOF. io.Copy takes a 32 KiB buffer, but first
looks for WriterTo on the source and ReaderFrom on the destination: from a
*bytes.Reader 100,000 bytes went out in one Write, without a single
memory allocation. bufio.Writer turned 1000 writes into 3, but the third went
down only on Flush, and the write error also arrived only on Flush. And a
function with defer f.Close() and bufio without Flush returned nil and
left a 0-byte file on disk. The lesson measured the price of the buffer in one
configuration: 10,000 writes of 16 bytes to a file in a temporary folder,
go1.24.7, linux/amd64 — 4248 µs without bufio and 65 µs with it. The
mechanism explains why there is a difference; how large it is for you has to be
measured on your own machine.
- how to declare a type and a method on it — that is the lesson on structs;
- what a byte slice
[]byteis, its length and capacity; the "how to use it" level is enough, the details are in the lesson on slices; - that a type implements an interface if it has all of the interface's methods — there is nothing to declare; the details are in the lesson on interfaces, which comes later in the track;
- that an error in Go is an ordinary
errorreturn value compared withnil— the details are in the lesson on errors; - that
deferpostpones a call until the function returns — the details are in the lesson on defer, panic and recover.
- what exactly
Readpromises and why the error comes together with data; io.ReadFull,io.Copy,WriterToandReaderFrom, the wrappersMultiWriter,TeeReader,LimitReader,Pipe;bufio.Writer,Flushand the error that gets lost indefer f.Close().
What is really being asked
The ladder is usually this one:
- "What is
io.Reader?" — the warm-up; everyone remembers the signature. - "Write a loop that reads a stream to the end" — this checks in which
order the candidate looks at
nand aterr. - "Why did
Readreturn less than the buffer size?" — the short read andio.ReadFull. - "How does
io.Copycopy, and how much memory does it need?" — a 32 KiB buffer and two interfaces that let it do without one. - "Why
bufio, and what happens if you forgetFlush?" - "What is wrong with
defer f.Close()?" — about the error returned by a write that happens on close.
The ladder has one spine: Read and Write promise only one thing — how
many bytes went through in this call and what happened. Neither "the whole
buffer", nor "the end separately from the data", nor "written to disk" is part
of the contract. Everything you get caught on in an interview is an attempt to
read more into the contract than is written there.
Base: two methods and what they promise
Fragment of io/io.go — the interface definitions, comments omitted:
type Reader interface {
Read(p []byte) (n int, err error)
}
type Writer interface {
Write(p []byte) (n int, err error)
}The caller brings the buffer: Read writes into p, Write reads from
p. That is why neither method has to allocate memory for the data itself, and also
why the documentation forbids both of them to retain p after returning. The
main sentence of the Read contract:
Read reads up to len(p) bytes into p. It returns the number of bytes read (0
<= n <= len(p)) and any error encountered.
"Up to len(p)" are the key words. Write promises something else: if it wrote
less than the whole slice, it must explain why — "Write must return a non-nil error if it
returns n < len(p)". The contracts are asymmetric, and this table is worth
memorizing right away — all the mechanisms below rest on it:
Read(p) | Write(p) | |
|---|---|---|
| how many bytes | 0 to len(p), whatever is there | len(p); fewer only with an error |
n > 0 together with err != nil | happens: handle n first | happens, including with n == len(p); if fewer than len(p) bytes were written, the error is mandatory |
| end of data | io.EOF — not a failure, a signal | — |
buffer p | must not be retained | must not be retained or modified |
That is enough for a Junior-level answer to the first question: io.Reader
is an interface with one method Read(p []byte) (n int, err error): it puts up
to len(p) bytes into p and returns how many it put. io.Writer is the same
for writing: if it wrote less than len(p), it must return an error. Any type with such a
method — a file, a connection, a buffer — fits wherever the interface is
expected. Why an interface needs no "implements" declaration and what lives
in an interface variable is covered in the lesson on
interfaces.
The Write rule works in one direction: a short write without an error breaks
the contract, while a full write with an error does not. A writer may
accept all the bytes and still report a problem, and the caller checks err
even when n == len(p). The run checks both sides.
Output of bench/goio/internals.go, section 9; labels translated:
9) WRITE: A FULL WRITE TOGETHER WITH AN ERROR
─────────────────────────────────────────────
the writer returns len(p) and errQuota — the Write contract allows this:
io.Copy (generic loop) -> 12, err=quota exceeded
io.Copy (*strings.Reader) -> 12, err=quota exceeded
bufio: Flush -> quota exceeded
the writer returns len(p)/2 and nil — the contract is broken:
io.Copy (generic loop) -> 6, err=short write
io.Copy and bufio pass the error of a full write up: all 12 bytes are
counted, and still the copy returned quota exceeded. A short write without an
error was turned into short write by the io.Copy loop — more on that in
Mechanism 3.
What this was checked on: the runs — go1.24.7 and go1.25.1, linux/amd64; the
documentation quotes were checked against go doc on 14.10.2026. By that date
Go 1.26 and 1.27 had already been released; the lesson was not run on them.
Mechanism 1: data arrives together with an error
The first question people get wrong is the order. The Read documentation
allows a reader two ways to end a stream:
When Read encounters an error or end-of-file condition after successfully
reading n > 0 bytes, it returns the number of bytes read. It may return the
(non-nil) error from the same call or return the error (and n == 0) from a
subsequent call.
The run reads the stream "hello, world" with a 5-byte buffer using four
readers. strings.Reader, *os.File and io.Pipe return the last chunk with
nil and EOF on the next call with n=0; iotest.DataErrReader from the
standard library returns EOF in the same call as the last chunk: it was made
precisely to test code against that behavior.
Output of bench/goio/internals.go, section 1, one block per reader; labels translated:
stream "hello, world", 12 bytes, 5-byte buffer
strings.NewReader
Read #1 -> n=5 "hello" err=nil
Read #2 -> n=5 ", wor" err=nil
Read #3 -> n=2 "ld" err=nil
Read #4 -> n=0 "" err=EOF
*os.File
Read #1 -> n=5 "hello" err=nil
Read #2 -> n=5 ", wor" err=nil
Read #3 -> n=2 "ld" err=nil
Read #4 -> n=0 "" err=EOF
io.Pipe, writer: Write("hel"), Write("lo, wor"), Write("ld")
Read #1 -> n=3 "hel" err=nil
Read #2 -> n=5 "lo, w" err=nil
Read #3 -> n=2 "or" err=nil
Read #4 -> n=2 "ld" err=nil
Read #5 -> n=0 "" err=EOF
iotest.DataErrReader(strings.NewReader)
Read #1 -> n=5 "hello" err=nil
Read #2 -> n=5 ", wor" err=nil
Read #3 -> n=2 "ld" err=EOF
Now the loop people write most often — "read, check the error, process".
Fragment — the function readBroken from bench/goio/internals.go, in essence:
for {
n, err := r.Read(buf)
if err != nil {
break
}
out = append(out, buf[:n]...)
}Output, section 2; labels translated:
2) A LOOP THAT LOOKS AT THE ERROR FIRST
───────────────────────────────────────
if err != nil { break } before processing n bytes:
strings.NewReader -> "hello, world"
*os.File -> "hello, world"
io.Pipe ("hel", "lo, wor", "ld") -> "hello, world"
DataErrReader -> "hello, wor" (2 bytes lost)
On the first three readers the loop works; on the fourth it loses "ld" — the last
chunk arrived together with EOF, and break threw it away unprocessed. And
there is no error: the function returned the short string as if nothing had
happened.
That is why such a loop passes tests easily: they are usually run on
strings.NewReader or on a file — the kind of reader on which it is correct. It
breaks on another source that is entitled to hand over the tail together with
EOF, and it breaks silently. The documentation says how it should be done:
Callers should always process the n > 0 bytes returned before considering the
error err.
Fragment — the function readCorrect from bench/goio/internals.go, in essence:
for {
n, err := r.Read(buf)
out = append(out, buf[:n]...) // bytes first
if err == io.EOF {
return out, nil
}
if err != nil {
return out, err
}
}Output, section 2 (continued); labels translated:
n bytes first, then the error:
strings.NewReader -> "hello, world"
io.Pipe ("hel", "lo, wor", "ld") -> "hello, world"
DataErrReader -> "hello, world"
io.ReadAll(DataErrReader) -> "hello, world" err=nil
Two follow-up clarifications. io.EOF is compared with ==, and that is
not sloppiness: the documentation requires that "Read must return EOF itself,
not an error wrapping EOF, because callers will test for EOF using ==".
Wrapping and errors.Is are about errors propagating up the stack; those are
covered in the lesson on errors. And you
usually do not need to write your own loop: in the run io.ReadAll collected
the whole string from DataErrReader too, and the io.Copy loop in the source
first writes the nr > 0 bytes and only then looks at the error (io/io.go,
lines 430 and 448 on go1.24.7). A successful io.Copy returns nil, not
EOF.
The correct loop and a write through bufio with a checked Flush fit into a
short program. The full file bench/goio/cases/example/main.go (its comments
translated here) can be copied into an empty folder next to a go.mod
(module example, go 1.24) and run with go run .:
// The lesson's full short example: a read loop by the Read contract and
// a write through bufio with a checked Flush.
//
// Run in the folder with this file and go.mod: go run .
package main
import (
"bufio"
"fmt"
"io"
"os"
"strings"
"testing/iotest"
)
// readAll reads r to the end: the n bytes first, then the error.
func readAll(r io.Reader) (string, error) {
var out []byte
buf := make([]byte, 5)
for {
n, err := r.Read(buf)
out = append(out, buf[:n]...)
if err == io.EOF {
return string(out), nil
}
if err != nil {
return string(out), err
}
}
}
func main() {
// DataErrReader returns the last chunk together with io.EOF.
s, err := readAll(iotest.DataErrReader(strings.NewReader("hello, world")))
fmt.Printf("readAll: %q, err=%v\n", s, err)
w := bufio.NewWriter(os.Stdout)
n, err := io.Copy(w, strings.NewReader("line through bufio\n"))
if err == nil {
err = w.Flush() // without Flush the line would stay in the buffer
}
fmt.Printf("io.Copy: %d bytes, err=%v\n", n, err)
}Output of bench/goio/example.sh; the header translated:
go1.24.7 linux/amd64 | full example: go run .
readAll: "hello, world", err=<nil>
line through bufio
io.Copy: 19 bytes, err=<nil>
Mechanism 2: the short read — Read does not have to fill the buffer
The second half of the same "up to len(p)":
If some data is available but not len(p) bytes, Read conventionally returns
what is available instead of waiting for more.
In the run the buffer holds 8 bytes, and the stream is longer.
Output of bench/goio/internals.go, section 3; labels translated:
3) SHORT READ: READ DOES NOT HAVE TO FILL THE BUFFER
────────────────────────────────────────────────────
HalfReader.Read(buf[8]) -> n=4 "hell" err=nil
OneByteReader.Read(buf[8]) -> n=1 "h" err=nil
io.Pipe, the writer writes "hel", "lo, wor", "ld"; the reader uses a buffer of 8:
Read -> n=3 "hel" err=nil
Read -> n=7 "lo, wor" err=nil
Read -> n=2 "ld" err=nil
Read -> n=0 "" err=EOF
iotest.HalfReader and iotest.OneByteReader are test wrappers that return
less on purpose. The pipe shows the same thing with no intent at all: each
Read got exactly what one Write on the other side put in — 3, 7 and 2
bytes, although the buffer held 8. In Mechanism 1 the same stream, read with a
5-byte buffer, came in chunks of 3, 5, 2 and 2: one 7-byte Write was split
across two Reads. Chunk boundaries are set by the source, not by the buffer.
Hence the classic mistake with a fixed-length header.
An illustrative fragment, not part of the run:
hdr := make([]byte, 8)
n, err := r.Read(hdr) // n may turn out to be 4Code that goes on to parse all 8 bytes will read half garbage. For "exactly
this many bytes" there is io.ReadFull:
ReadFull reads exactly len(buf) bytes from r into buf.
Output, section 3 (continued); labels translated:
io.ReadFull(HalfReader, buf[8]) -> n=8 "hello, w" err=nil
io.ReadFull(5 bytes, buf[8]) -> n=5 "hello" err=unexpected EOF
io.ReadFull(0 bytes, buf[8]) -> n=0 err=EOF
errors.Is(ErrUnexpectedEOF, EOF) -> false
HalfReader returns half at a time, and ReadFull keeps calling it until it
has 8. The two lower cases are more interesting: EOF and
ErrUnexpectedEOF are different answers. EOF means "there was no data at
all" — the stream ended exactly on a record boundary, and that is a normal end.
ErrUnexpectedEOF means "a record started and was cut off" — the input is
truncated. errors.Is does not confuse them, and a loop reading fixed-length
records is written exactly that way: io.EOF — exit, io.ErrUnexpectedEOF —
report broken input.
Mechanism 3: io.Copy — a 32 KiB buffer or someone else's method
io.Copy(dst, src) is the very loop from Mechanism 1, written once. But
before running it, Copy asks whether the two sides can copy by themselves:
If src implements WriterTo, the copy is implemented by calling src.WriteTo(dst).
Otherwise, if dst implements ReaderFrom, the copy is implemented by calling
dst.ReadFrom(src).
What it looks like in code — io/io.go, lines 407–419:
func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
// If the reader has a WriteTo method, use it to do the copy.
// Avoids an allocation and a copy.
if wt, ok := src.(WriterTo); ok {
return wt.WriteTo(dst)
}
// Similarly, if the writer has a ReadFrom method, use it to do the copy.
if rf, ok := dst.(ReaderFrom); ok {
return rf.ReadFrom(src)
}
if buf == nil {
size := 32 * 1024
if l, ok := src.(*LimitedReader); ok && int64(size) > l.N {The run checks every branch with counters: a wrapper around the source counts
Read calls and records len(p), a wrapper destination counts Write calls.
The wrappers deliberately hide every method except their own — otherwise
Copy would find WriterTo on the *bytes.Reader inside.
Output of bench/goio/internals.go, section 4; labels translated:
4) IO.COPY: A 32 KB BUFFER OR SOMEONE ELSE'S METHOD
───────────────────────────────────────────────────
copying 100000 bytes
Read only -> Write only: Read ×5, len(p)=32768; Write ×4 [32768 32768 32768 1696]; copied 100000
*bytes.Reader (WriterTo): Read ×0; Write ×1 [100000]; copied 100000
-> *bytes.Buffer (ReaderFrom): Read ×10, first len(p)=[512 512 1024]; copied 100000
io.LimitReader(src, 100): Read ×1, len(p)=100; Write ×1; copied 100
total heap allocated (TotalAlloc) per io.Copy, average over 1000:
generic path (io.Copy buffer) allocations: 1, 32.0 KiB
via WriterTo allocations: 0, < 1 KiB
Four lines — four branches of the source. The generic path: a buffer of
32,768 bytes, one allocation per Copy call, four Writes and a fifth Read
that returned EOF. WriterTo: *bytes.Reader handed the destination
the whole slice in one Write, Read was never called, no memory was
allocated. ReaderFrom: now *bytes.Buffer does the reading — in its own
chunks, starting at 512 bytes. LimitReader: the buffer shrank to 100
bytes, because there is nothing more to read anyway.
Who in the standard library can bypass the buffer this way.
Output, section 5:
5) WHO IN THE STANDARD LIBRARY IMPLEMENTS WRITERTO AND READERFROM
─────────────────────────────────────────────────────────────────
*strings.Reader WriterTo: true ReaderFrom: false
*bytes.Reader WriterTo: true ReaderFrom: false
*bytes.Buffer WriterTo: true ReaderFrom: true
*bufio.Reader WriterTo: true ReaderFrom: false
*bufio.Writer WriterTo: false ReaderFrom: true
*os.File WriterTo: true ReaderFrom: true
*net.TCPConn WriterTo: true ReaderFrom: true
*io.PipeReader WriterTo: false ReaderFrom: false
*io.LimitedReader WriterTo: false ReaderFrom: false
io.Discard WriterTo: false ReaderFrom: true
The practical takeaway — a wrapper hides the methods of what it wraps. The
counter from the run, io.LimitReader, any type of yours with a single Read
switch off the source's WriterTo: Copy sees only the methods of the outer
type. The destination's ReaderFrom still works — in the run onlyReader →
*bytes.Buffer still went through ReadFrom. That follows from how a type assertion to an interface works, not from anything
special about io.
The io wrappers do not accumulate data. MultiWriter hands every write
to all its writers in turn, TeeReader writes what it read into a writer right
inside Read, LimitReader cuts the stream off, Pipe connects code that
writes with code that reads. About Pipe the documentation says it plainly:
"The data is copied directly from the Write to the corresponding Read (or
Reads); there is no internal buffering".
Output, section 6; labels translated:
6) COMPOSITION: WRAPPERS DO NOT ACCUMULATE DATA
───────────────────────────────────────────────
stream of 1048576 bytes
MultiWriter(a, b): a got 1048576 bytes in 32 Writes, b — 1048576 in 32
TeeReader(r, sha256): wrote 1048576 bytes, hash matches sha256.Sum256: true
LimitReader(r, 1000): wrote 1000 bytes
Pipe, one Write of 1048576 bytes: the reader got 1048576 bytes in 32 Writes of 32768
total heap allocated (TotalAlloc) per copy of 1 MiB, average over 200:
io.Copy(w, *bytes.Reader) allocations: 0, < 1 KiB
io.Copy(MultiWriter(w1, w2)) allocations: 2, < 1 KiB
io.Copy(w, TeeReader) allocations: 2, 32.0 KiB
io.Copy(w, io.Pipe) allocations: 6, 32.5 KiB
io.ReadAll(*bytes.Reader) allocations: 27, 5118.3 KiB
writer returned n < len(p) and nil: io.Copy -> 6, err=short write
A megabyte through TeeReader or a pipe costs roughly one io.Copy buffer —
32.0 and 32.5 KiB, and TeeReader computed the SHA-256 of the same stream in
the same pass. Through MultiWriter from a *bytes.Reader — under a
kilobyte: the source's WriterTo passed the whole slice in one write, and
MultiWriter handed it to two writers. For comparison, the bottom line:
io.ReadAll of the same megabyte allocated more than five megabytes — it
collects everything into one growing slice (more in "Deeper"). A stream that
can be processed on the way is better processed on the way.
The last line is about the Write contract. The writer returned half and
nil, breaking the rule n < len(p) → error, and the generic io.Copy loop
turned it into io.ErrShortWrite by itself (io/io.go, lines 443–446 on
go1.24.7). The source in the run is wrapped so that the loop is reached: a bare
*strings.Reader has WriterTo, and it would have made the same check.
Without that check, half the data would have disappeared silently.
Mechanism 4: bufio.Writer — writes go down in chunks, and the tail only after Flush
bufio.Writer is a writer on top of a writer: it accumulates writes in a
buffer and hands them down in large chunks. A counter underneath shows how many
Writes reach the underlying writer.
Output of bench/goio/internals.go, section 7; labels translated:
7) BUFFERING: HOW MANY WRITES REACH THE UNDERLYING WRITER
─────────────────────────────────────────────────────────
1000 writes of 10 bytes = 10000 bytes
without bufio: Write below ×1000 bytes arrived: 10000
bufio.NewWriter, before Flush: Write below ×2 bytes arrived: 8192, in the buffer: 1808
after Flush: Write below ×3 bytes arrived: 10000 [4096 4096 1808]
default buffer size: 4096 bytes
one Write of 10000 bytes into an empty buffer: Write below ×1 [10000]
A thousand writes became three. But the third went down only on Flush:
before it the underlying writer had received 8192 bytes, and 1808 sat in the
buffer. If the program ends or the function returns without Flush, those
bytes will be nowhere — bufio.Writer does not know it has been abandoned. The
documentation says so directly:
After all data has been written, the client should call the Writer.Flush
method to guarantee all data has been forwarded to the underlying io.Writer.
The last line of the block is an implementation detail: a write larger than
the buffer into an empty buffer is not copied by bufio but handed down
directly (bufio/bufio.go, lines 680–683 on go1.24.7, the comment "Large
write, empty buffer").
The other side of the buffer — the error is deferred too. /dev/full on
Linux is a device on which every write ends with "no space left on device".
Output, section 7 (continued); labels translated:
bufio.Writer on top of /dev/full (every write is ENOSPC):
100 x Write(10 bytes) without error: 100 of 100
Flush -> write /dev/full: no space left on device
Write after the error -> write /dev/full: no space left on device
without bufio, the very first f.Write -> write /dev/full: no space left on device
A hundred writes "succeeded": they landed in the buffer and never reached the
device. Only Flush returned the error — and after it bufio.Writer answers
with the same error on any call:
If an error occurs writing to a Writer, no more data will be accepted and all
subsequent writes, and Writer.Flush, will return the error.
Hence the rule that is actually being tested: with bufio, checking the
error of every Write is not enough — you have to check Flush. The
stickiness of the error is even convenient: you can write without checks and
check once — on Flush.
Mechanism 5: Close writes too — and its error gets lost in defer
The last step of the same idea: for many writers the last write happens not in
Write but on close. The most vivid example is gzip.Writer:
Close closes the Writer by flushing any unwritten data to the underlying
io.Writer and writing the GZIP footer.
Output of bench/goio/internals.go, section 8; labels translated:
8) CLOSE WRITES TOO — AND ITS ERROR GETS LOST
─────────────────────────────────────────────
defer f.Close(), bufio without Flush, 100 lines: returned nil, file size 0 bytes
gzip without Close: bytes in the underlying writer: 10
reading back -> unexpected EOF
gzip after Flush, without Close: bytes in the underlying writer: 62
reading back -> unexpected EOF
gzip after Close: bytes in the underlying writer: 75
reading back -> 2600 bytes, err=nil
The first line is a function people write all the time.
Fragment — the function writeReportDeferOnly from bench/goio/internals.go, in essence:
func writeReport(path string, lines int) error {
f, err := os.Create(path)
if err != nil {
return err
}
defer f.Close()
w := bufio.NewWriter(f)
for i := 0; i < lines; i++ {
fmt.Fprintf(w, "line %d\n", i)
}
return nil
}The hundred lines of the run take 1590 bytes, less than the default bufio
buffer (section 10 of the run: "100 report lines from section 8: 1590 bytes;
default bufio buffer: 4096 bytes"). All of them stayed in the buffer, the
file was closed empty, and the function returned nil. Not a single error
check fired, because there was no error — there was a lost write. With gzip
it is the same on another level: without Close the underlying writer holds
only the 10-byte header, even after Flush the archive cannot be read — there
is no footer, and the reader answers unexpected EOF.
Now about defer f.Close() itself. Close returns an error, and defer
throws that result away. The run shows that there is somewhere for an error to
come from.
Output, section 8 (continued); labels translated:
first f.Close() -> nil
second f.Close() -> close <folder>/twice.txt: file already closed
f.Write after Close -> write <folder>/twice.txt: file already closed
writing to /dev/full:
defer w.Flush(); defer f.Close() -> the function returned nil
Flush checked, Close in defer with err -> the function returned write /dev/full: no space left on device
The function that deferred both Flush and Close returned nil when
writing to a device with no space. The one that checked Flush and captures
the Close error into a named result returned the real cause. Here the error
came from Flush: capturing the Close error is insurance for the case when
Close writes by itself, as with gzip.Writer; this run does not show it in
action.
Fragment — the function writeToChecked from bench/goio/internals.go under a shorter name:
func writeTo(path string) (err error) {
f, err := os.OpenFile(path, os.O_WRONLY, 0)
if err != nil {
return err
}
defer func() {
if cerr := f.Close(); err == nil {
err = cerr
}
}()
w := bufio.NewWriter(f)
if _, err = w.WriteString("report\n"); err != nil {
return err
}
return w.Flush()
}Why defer can change the return value and why that needs a named result is
covered in the lesson on defer, panic and
recover. What matters here is the
rule: for a writer, the last write is Flush or Close, and their errors are
checked just like the error of Write. For a file opened for reading only,
defer f.Close() without a check is common practice: there is nothing left to
write. Whether Close on a regular file can return a write error was not tested by
this run: on a local disk the first Close returned nil. And the guarantee
that the data reached the disk comes not from Close but from f.Sync() —
according to the documentation it "commits the current contents of the file to
stable storage", and its error is checked too.
Deeper: why a small Write is expensive — and what memory costs
The counter from Mechanism 4 tells how many calls reach the bottom; the
measurement tells what that means for a file. 10,000 writes of 16 bytes
straight into an *os.File and through a bufio.Writer.
Output of bench/goio/practice.go, problem 2; labels translated:
10000 x f.Write(16 bytes) to *os.File 4248 µs, 10000 Writes reached the file
the same writes via bufio.Writer + Flush 65 µs, 40 Writes reached the file
ratio 65
per write: 425 ns against 7 ns
The counter in the same run shows that all 10,000 Write calls reached the
file, and each f.Write on an *os.File goes into the write system call
(internal/poll/fd_unix.go, FD.Write, line 381 on go1.24.7:
syscall.Write): 425 ns per write. How the price depends on the size of one
write was not tested — the measurement has a single size, 16 bytes. bufio
turns 10,000 calls into 40; its 65 µs include both those 40 calls and copying
10,000 records into the buffer — 7 ns per write on average. Hence the rule: many small writes to a file or a socket go through a
buffer. Where Write writes to memory anyway, as with bytes.Buffer, the
buffer is superfluous; and bufio.NewWriter on top of a bufio.Writer with a
large enough buffer, according to the documentation, simply returns that same
writer.
The second price is memory, and here the trap is the opposite. io.Copy keeps
one 32 KiB buffer in memory, however much data flows through (with wrappers in
the run — no more than 32.5 KiB). io.ReadAll
keeps everything.
An output line of bench/goio/internals.go, section 6; labels translated:
io.ReadAll(*bytes.Reader) allocations: 27, 5118.3 KiB
This is the sum of all allocations per call (TotalAlloc), not memory held
at the same time. ReadAll starts with a 512-byte slice and grows it by
appending (io/io.go, lines 709–726 on go1.24.7); every growth is a new array,
and the previous one becomes garbage. At the end one slice remains.
Output of bench/goio/internals.go, section 10; labels translated:
io.ReadAll(1048576 bytes): len=1048576, cap=1114112 — the final slice
The final buffer is about 1.06 MiB; the remaining ~4 MiB of the 5118.3 KiB are
intermediate arrays that the garbage collector will reclaim. How many of them
are alive at the same time at the peak, the lesson did not measure; but they
had to be allocated. And when the
other side sets the size of the stream — a request body, another service's
response — ReadAll without a limit hands it the decision of how much memory
your process will take. If you really need all of it in memory, limit the size
(io.LimitReader); if not, process the stream on the way with io.Copy and
wrappers.
A summary table that is handy to keep in your head:
| task | tool | what to check |
|---|---|---|
| read a stream to the end | io.Copy, io.ReadAll, your own loop | n first, then err |
| read exactly N bytes | io.ReadFull | EOF versus ErrUnexpectedEOF |
| many small writes | bufio.Writer | the Flush error |
| writing with compression or a buffer | gzip.Writer, bufio.Writer | the Close / Flush error |
| a stream to two places | io.MultiWriter, io.TeeReader | the error of any of the writers |
| limit the input | io.LimitReader | that truncated input is not an error but EOF |
How to answer in an interview
Short answer: io.Reader is an interface with one method
Read(p []byte) (n int, err error): it puts up to len(p) bytes into p and
returns how many it put, plus an error. io.Writer is the same for writing,
but having written less than len(p) it must return an error, and the error is
checked even after a full write. When reading, you handle the n
bytes first and then look at err, and io.EOF is the normal end of a stream.
For ready-made tasks — io.Copy, io.ReadFull, bufio.
That is enough to answer correctly. What follows is what gets added if the interviewer digs.
If the interviewer digs deeper
On the read loop — the order of n and err. "The documentation allows
the last chunk to come together with EOF. A loop that checks err first
works on a strings.Reader, but on iotest.DataErrReader it returned
"hello, wor" instead of "hello, world" — with no error. So buf[:n] first,
then err."
On "why did it read less" — the short read. "Read promises up to
len(p), not len(p): with a buffer of 8 the reader returned 4. For a fixed
size — io.ReadFull; on truncated input it returns ErrUnexpectedEOF, on
empty input EOF, and those are different cases."
On io.Copy — the buffer and the interfaces. "First WriterTo on the
source, then ReaderFrom on the destination, and only then a 32 KiB buffer. From
a *bytes.Reader 100,000 bytes went out in one Write with no allocations; a
wrapper with a single Read switches that path off."
On bufio — Flush and the deferred error. "A thousand writes of 10 bytes
— three Writes below, and the last one only on Flush. The write error also
arrives only on Flush: on top of /dev/full all hundred Writes returned
nil."
On defer f.Close() — writing on close. "For gzip.Writer and bufio the
last write is in Close and Flush. A function with defer f.Close() and
bufio without Flush returned nil and left an empty file. So I check
Flush explicitly, and for a writer I capture the Close error into a named
result."
Next they ask
What is wrong with io.ReadAll for an HTTP request body?
It keeps the whole body in memory and allocates more than it reads: in the run
a megabyte cost 5118.3 KiB of allocations in total over 27 growth steps (the
final slice has a capacity of 1114112 bytes, the rest are intermediate arrays). And the body
size is set by the client. So the body is either limited — io.LimitReader or
http.MaxBytesReader in the handler — or read as a stream: a JSON decoder,
io.Copy into a file, bufio.Scanner line by line.
io.LimitReader has a quirk worth mentioning: it ends truncated input with an
ordinary EOF, and the code will not find out that the data was longer. If that
matters, read one byte more than the limit and check whether you reached it.
How this works in a server is in the lesson on the HTTP
server.
When do you need io.Pipe?
When one library can only write to an io.Writer and another can only
read from an io.Reader. The classic example is compressing data on the
fly and handing it to an HTTP client as a request body: gzip.Writer writes to
a PipeWriter in a separate goroutine, and the request reads from the
PipeReader.
There is no buffer in a pipe: each Write waits until the data is taken. In the
run the reader received one 1-megabyte Write in 32 chunks of 32768 bytes —
exactly the size of the io.Copy buffer that was reading from the pipe. Two
things are mandatory: the writer must close the pipe (Close or
CloseWithError), or the reader will wait forever, and writing and reading must
happen in different goroutines — otherwise the very first Write will block the
program.
How do you test that code reads a stream correctly?
With the testing/iotest package — that is what it is for. DataErrReader
returns the last chunk together with EOF and catches the loop from
Mechanism 1; HalfReader and OneByteReader return less than asked and catch
code that expects a full buffer; ErrReader returns a given error on every
Read; TimeoutReader returns iotest.ErrTimeout on the second read. Wrapping
a test's input in them costs one line.
There is also the reverse check — iotest.TestReader(r, content): it tests
your reader — reads it in chunks of different sizes until EOF and compares
what it read with the expected content.
Common misconceptions
If Read returned an error, there is no data in that call
There is, and it must be handled first: the documentation allows returning n > 0 together with an error, including io.EOF. A loop that checked err before n collected "hello, wor" instead of "hello, world" on iotest.DataErrReader — with no error.
Read fills the whole buffer if there is enough data
It promises "up to len(p)", not len(p): with a buffer of 8 the reader returned 4 bytes, the pipe 3, 7 and 2, exactly one Write each. Exactly N bytes are read by io.ReadFull.
io.Copy always copies through its own 32 KiB buffer
Only if the source has no WriterTo and the destination no ReaderFrom. From a *bytes.Reader 100,000 bytes went out in one Write, Read was never called, zero allocations. And conversely: a wrapper with a single Read method hides the source's WriterTo.
If no Write returned an error, the data is written
Not with bufio.Writer: on top of /dev/full all hundred writes returned nil, and no space left on device came only from Flush. Without Flush the last 1808 bytes of 10,000 never went down at all.
defer f.Close() is all a file needs
Not for a file being written. A function with defer f.Close() and bufio without Flush returned nil and left a 0-byte file; gzip without Close cannot be read — unexpected EOF. A writer's Flush and Close errors are checked, and defer throws them away.
io.ReadAll is the simplest and cheapest way to read a stream
Simplest — yes, cheapest — no: a megabyte cost 5118.3 KiB of allocations in total, because the slice grows by appending; the final slice is about 1.06 MiB, the rest are intermediate arrays for the garbage collector. io.Copy through wrappers in the run kept no more than a little over one 32 KiB buffer — 32.5 KiB with a pipe, under one with MultiWriter.
Practice
Two problems. Answer first, then check against the real output: in both, the correct answer is taken from a run of the script rather than assigned.
Practice · predict the output
r := iotest.DataErrReader(strings.NewReader("hello, world"))
buf := make([]byte, 5)
for {
n, err := r.Read(buf)
if err != nil {
break
}
fmt.Print(string(buf[:n]), "|")
}
fmt.Println()
n, err := io.ReadFull(strings.NewReader("hello"), make([]byte, 8))
fmt.Println(n, err)
var out bytes.Buffer
w := bufio.NewWriter(&out)
fmt.Fprint(w, "report")
fmt.Println(out.Len())Practice · estimate
Knowledge check
Read returned n = 2 and err = io.EOF. What do you do with those two bytes?
This is neither a retelling nor a separate text: everything below is taken from the article itself — its own summary, the section headings, the “actually” column and the version table. Which is why these theses cannot drift from the article.
The gist
io.Readerandio.Writerare interfaces with one method each:Read(p)puts up tolen(p)bytes intop,Write(p)writespand, if it wrote less thanlen(p), must return an error. Both return(n int, err error)— how many bytes went through and what happened. Everything rests on these two signatures: a file, a network connection,gzip,bufio,bytes.Buffer,sha256, andio.Copy, which connects any source to any destination.- Read may return data and an error in the same call — and they must be handled in that order: first the
nbytes, thenerr. The documentation allows two endings of a stream: the last chunk withniland then(0, EOF)— or the last chunk right together withEOF. A loop withif err != nil { break }before handlingncollects the whole string"hello, world"on astrings.Reader, and"hello, wor"on a reader of the second kind: two bytes vanished without a single error. - Beyond that comes what separates real understanding from having read about it.
Readdoes not have to fill the buffer: with a buffer of 8 the reader returned 4 bytes, and filling the buffer completely is guaranteed byio.ReadFull— and on short input it answersio.ErrUnexpectedEOF.io.Copytakes a 32 KiB buffer, but first looks forWriterToon the source andReaderFromon the destination: from a*bytes.Reader100,000 bytes went out in oneWrite, without a single memory allocation.bufio.Writerturned 1000 writes into 3, but the third went down only onFlush, and the write error also arrived only onFlush. And a function withdefer f.Close()andbufiowithoutFlushreturnedniland left a 0-byte file on disk. The lesson measured the price of the buffer in one configuration: 10,000 writes of 16 bytes to a file in a temporary folder, go1.24.7, linux/amd64 — 4248 µs withoutbufioand 65 µs with it. The mechanism explains why there is a difference; how large it is for you has to be measured on your own machine.
In fact
- There is, and it must be handled first: the documentation allows returning
n > 0together with an error, includingio.EOF. A loop that checkederrbeforencollected"hello, wor"instead of"hello, world"oniotest.DataErrReader— with no error. - It promises "up to
len(p)", notlen(p): with a buffer of 8 the reader returned 4 bytes, the pipe 3, 7 and 2, exactly oneWriteeach. Exactly N bytes are read byio.ReadFull. - Only if the source has no
WriterToand the destination noReaderFrom. From a*bytes.Reader100,000 bytes went out in oneWrite,Readwas never called, zero allocations. And conversely: a wrapper with a singleReadmethod hides the source'sWriterTo. - Not with
bufio.Writer: on top of/dev/fullall hundred writes returnednil, andno space left on devicecame only fromFlush. WithoutFlushthe last 1808 bytes of 10,000 never went down at all. - Not for a file being written. A function with
defer f.Close()andbufiowithoutFlushreturnedniland left a 0-byte file;gzipwithoutClosecannot be read —unexpected EOF. A writer'sFlushandCloseerrors are checked, anddeferthrows them away. - Simplest — yes, cheapest — no: a megabyte cost 5118.3 KiB of allocations in total, because the slice grows by appending; the final slice is about 1.06 MiB, the rest are intermediate arrays for the garbage collector.
io.Copythrough wrappers in the run kept no more than a little over one 32 KiB buffer — 32.5 KiB with a pipe, under one withMultiWriter.
What is covered
- What is really being asked
- Base: two methods and what they promise
- Mechanism 1: data arrives together with an error
- Mechanism 2: the short read — Read does not have to fill the buffer
- Mechanism 3: io.Copy — a 32 KiB buffer or someone else's method
- Mechanism 4: bufio.Writer — writes go down in chunks, and the tail only after Flush
- Mechanism 5: Close writes too — and its error gets lost in defer
- Deeper: why a small Write is expensive — and what memory costs
- How to answer in an interview
- Next they ask
- Common misconceptions
- Practice
- Knowledge check
Sources & further reading
6 SOURCES
- Package io — ReaderOfficial documentation. The Read contract. How many bytes: «Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered». The short read: «If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more». Data together with an error: «It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call». And the rule for the caller: «Callers should always process the n > 0 bytes returned before considering the error err». Checked against go doc io.Reader, go1.24.7.https://pkg.go.dev/io#Reader
- Package io — Writer, EOF, ReadFullOfficial documentation. The Write contract is stricter: «Write must return a non-nil error if it returns n < len(p)». How to compare with EOF: «Read must return EOF itself, not an error wrapping EOF, because callers will test for EOF using ==». ReadFull: «ReadFull reads exactly len(buf) bytes from r into buf» and «If an EOF happens after reading some but not all the bytes, ReadFull returns ErrUnexpectedEOF». Checked against go doc, go1.24.7.https://pkg.go.dev/io#ReadFull
- Package io — Copy, Pipe, TeeReaderOfficial documentation. What Copy copies with: «If src implements WriterTo, the copy is implemented by calling src.WriteTo(dst). Otherwise, if dst implements ReaderFrom, the copy is implemented by calling dst.ReadFrom(src)». Why a successful Copy does not return EOF: «A successful Copy returns err == nil, not err == EOF». For TeeReader: «There is no internal buffering - the write must complete before the read completes». For Pipe: «The data is copied directly from the Write to the corresponding Read (or Reads); there is no internal buffering». Checked against go doc, go1.24.7.https://pkg.go.dev/io#Copy
- io/io.go — copyBufferGo source code. Lines 407–419 on go1.24.7 (the same numbers on go1.25.1): first the src.(WriterTo) check on line 410, then dst.(ReaderFrom) on line 414, and only then the buffer — line 418:
size := 32 * 1024; if the source is a *LimitedReader with a smaller remainder, the buffer shrinks to it (lines 419–425). The comment on the first check: «Avoids an allocation and a copy».https://github.com/golang/go/blob/go1.24.7/src/io/io.go#L407-L450 - Package bufio — WriterOfficial documentation. Why Flush: «After all data has been written, the client should call the Writer.Flush method to guarantee all data has been forwarded to the underlying io.Writer». The error is sticky: «If an error occurs writing to a Writer, no more data will be accepted and all subsequent writes, and Writer.Flush, will return the error». The default buffer size is
defaultBufSize = 4096, bufio/bufio.go, line 19 on go1.24.7.https://pkg.go.dev/bufio#Writer - Packages compress/gzip and os — CloseOfficial documentation. gzip.Writer.Close writes: «Close closes the Writer by flushing any unwritten data to the underlying io.Writer and writing the GZIP footer». os.File.Close: «Close will return an error if it has already been called». Checked against go doc, go1.24.7.https://pkg.go.dev/compress/gzip#Writer.Close