Signals, zombies and PID 1: why a container ignores SIGTERM
The questions come as a ladder: what a zombie is — who adopts an orphan — why a container does not stop at once — what exit code 137 means. The first two rest on a process dying in two separate events; the last two rest on one kernel rule that is almost never said out loud: for the process with ID 1 in its namespace, the default action of a signal is not carried out.
Full technical treatment
TL;DR
A process dies in two events, not one. First it stops running, then somebody collects its status; between those two events what is left of it is a record, and that record is called a zombie. And the kernel shields the process that came first inside a container: a signal it has installed no handler for does nothing to it at all. That is a kernel rule, not a Docker quirk.
Hence the consequence that matters: an image whose first process does not
handle SIGTERM does not stop on docker stop. It sits out the whole grace
period and is killed by SIGKILL at the end of it. Measured with a two-second
grace period: 11 ms with a handler against 2009 ms without one. The second
failure is quieter: unreaped children do not clear up on their own — a thousand
finished children with no wait left a thousand zombies, 100% of them, and zero
after wait.
Beyond that is what separates knowing from having read. An orphan is adopted
by a process rather than by "the system" — by init or by the nearest subreaper:
measured, a grandchild's parent changes from a real PID to 1 at the moment its
father dies. For the process with ID 1 in its own namespace, default signal
actions are not carried out, and the single exception is SIGKILL and
SIGSTOP: sent from an ancestor namespace they are delivered forcibly and
cannot be caught by anything, so the orchestrator always wins and only the delay
is in question. 137 is 128 + 9 and 143 is 128 + 15, and the addition is the
shell's rather than the kernel's: 137 means "killed by SIGKILL" and says
nothing about who sent it. And a zombie "sits in memory" only as a figure of
speech: the process's address space is gone, but the kernel keeps a small
record — PID, termination status, resource usage — and that record holds a slot
in the process table.
- a program gets started and eventually ends — by itself or because something stopped it;
- one running program can start another;
- somebody starts a container and at some point asks it to stop — during a rollout, for instance.
- what a zombie, an orphan, adoption and a subreaper are;
- what a process namespace is and what makes the number 1 special in one;
SIGTERM,SIGKILL, a signal's disposition, exit codes 137 and 143.
What is actually being asked
The ladder is almost always the same, and it rests on a single mechanism even though the questions look unrelated:
- "What is a zombie process?" — the warm-up that filters out anyone who thinks it means a hung process.
- "Why is it a problem if the process itself is already gone?" — where the substance starts.
- "Who becomes the parent of an orphaned process?" — a check on adoption and on subreapers.
- "Why does a container not stop at once?" — where the tempting answer is "it closes connections slowly": plausible, and past the mechanism.
- "What does exit code 137 mean?" — whether you know the arithmetic and what the code does not tell you.
- "Why do people put
tiniin a container?" — whether you have connected the first five into one thing.
The lesson climbs that ladder in one direction: the rule first, its consequences
after. Every number below was produced on one machine by
bench/signals/lifecycle.py and bench/signals/practice.py; your process IDs
and millisecond timings will differ, while the states, the exit codes and the
outcome of SIGTERM for PID 1 will not.
Base: a process, its number, and the message you send it
Before talking about zombies and about the number 1, it is worth naming, in ordinary words, the four things the whole lesson rests on.
A process is a running program. Not the file on disk but its run: the same program started twice is two processes, and they live independently of each other.
A process has a number. The kernel hands it out at startup, and that number is how the process is addressed afterwards: to look at it, to stop it, or to wait for its end.
A signal is a short message to a process. The kernel or another process sends it, and it carries exactly as much meaning as its own number: "finish up nicely", "stop right now", "suspend". No text, no reply: a signal either did something to the process or did nothing at all.
Processes have a parent and a child — the one that did the starting and the one that was started. That link is not a formality: it is the parent that later receives the answer to how the child ended.
The first rung of the ladder follows straight from those last two points. A child has finished; the parent has not yet asked how it ended; the kernel needs somewhere to keep the answer to that future question — and it keeps it. The state "the process is gone but the record of how it ended is still there" is what a zombie is.
And one more thing, without which the second half of the lesson reads as magic: a container has its own space of numbers. Processes inside it are numbered from one, as in a machine that has just booted, while the very same process has a completely different number outside. One process, two numbers: 1 inside, an ordinary one outside. Whoever stops the container lives outside and signals the external number — and the rules that apply to the process are the ones that apply to the number 1.
That is already enough to answer the basic interview question. What follows is about why the same signal acts differently on an ordinary process and on the process numbered 1, who gets the child when its parent dies first, and where the 137 in a report comes from.
Mechanism 1: a process dies twice, not once
Now the same thing in the documentation's words — starting with what does not depend on the distribution or the kernel version.
A process ends in two separate events. First it stops running — it calls
exit, takes a fatal signal, crashes. Then somebody collects its status by
calling wait. Between those two events the process is gone and the record of
it is not.
That record is the zombie, and wait(2) says so directly:
A child that terminates, but has not been waited for becomes a "zombie". The
kernel maintains a minimal set of information about the zombie process (PID,
termination status, resource usage information) in order to allow the parent to
later perform a wait to obtain information about the child.
That answers the second rung immediately. A zombie is not dangerous because it "sits in memory": the process's address space is gone, and what remains is the minimal set of information listed in the quotation above. It is dangerous because that record holds a slot in the process table, and the table is finite:
As long as a zombie is not removed from the system via a wait, it will consume a
slot in the kernel process table, and if this table fills, it will not be
possible to create further processes.
This is a leak with nothing to see on the memory graph. The application runs,
memory is flat, and then fork starts failing — in some process that has
nothing to do with the one at fault.
Here it is in a run. The child called _exit(7) and was left unwaited:
1. ZOMBIE: the child is gone, the record is not
-----------------------------------------------
child finished, status not collected Z (zombie)
why the kernel keeps the record so the parent can read the exit code
state after wait no such process
exit code the record was held for 7
Four lines that read as one story: state Z before wait, nothing after it,
and the seven the record was being held for. What removes a zombie is not time
and not a collector — it is the question "how did it end", asked by the parent.
Mechanism 2: an orphan gets a new parent, and it is a real one
The mirror case: the parent dies before the child. The child neither dies nor becomes parentless — it is adopted.
If a parent process terminates, then its "zombie" children (if any) are adopted
by init(1), (or by the nearest "subreaper" process as defined through the use of
the prctl(2) PR_SET_CHILD_SUBREAPER operation).
The run catches the substitution itself. A grandchild prints its parent twice — right after it starts, and after its father has exited:
2. ORPHAN: the parent died first
--------------------------------
grandchild: parent right after start 5359
grandchild: parent after father exits 1
The 5359 will be different on your machine; the 1 will not. And here is what follows for real work: the new parent is the one who will collect the orphan's status, not the lost one. If the new parent knows how, zombies never pile up. If it does not, every orphan piles up at once — which is exactly what happens in a container whose first process is an ordinary application.
The word "subreaper" is not decoration. PR_SET_CHILD_SUBREAPER is how a
process tells the kernel: orphans from my subtree come to me, not to PID 1.
Session managers and supervision systems use it; a container usually has none,
so orphans go straight to PID 1.
Mechanism 3: the PID 1 rule that everything else grows from
Now the important part. A process that received ID 1 in its own namespace
handles signals differently, and pid_namespaces(7) writes it down:
Only signals for which the "init" process has established a signal handler can
be sent to the "init" process by other members of the PID namespace. This
restriction applies even to privileged processes, and prevents other members of
the PID namespace from accidentally killing the "init" process.
From the outside it works the same way, and that is precisely the orchestrator's case: it lives in the ancestor namespace and sends a signal to your first process.
a process in an ancestor namespace can—subject to the usual permission checks
described in kill(2)—send signals to the "init" process of a child PID namespace
only if the "init" process has established a handler for that signal.
Put in the words worth answering with: for PID 1 the default action of a
signal is not carried out. A signal's disposition is what the process does
with it: the default action from the signal(7) table (SIGTERM is listed as
Term — terminate), ignoring it, or a handler. An ordinary process without a
handler gets the first. PID 1 has no first: with a handler the signal works,
without one it does nothing.
The same SIGTERM, two processes that differ by one line:
3. PID 1: one SIGTERM, two outcomes
-----------------------------------
PID inside the namespace (no handler) 1
SIGTERM from outside, no handler (PID 5362) ALIVE: the signal did nothing
PID inside the namespace (with handler) 1
SIGTERM from outside, with handler (PID 5364) exited
Note that the signal is sent from outside and by the external ID (5362), while inside the namespace that process is number 1. The orchestrator does the same: it does not step inside the container, it signals a process that looks ordinary from where it stands.
There is exactly one exception:
SIGKILL or SIGSTOP are treated exceptionally: these signals are forcibly
delivered when sent from an ancestor PID namespace.
So "the container cannot be stopped" is the wrong phrasing. It can always be stopped; the question is whether the orchestrator burns the whole grace period first.
Mechanism 4: what a missing handler costs, and where 143 and 137 come from
Stopping a container is three steps: SIGTERM, a grace period, SIGKILL. The
length of that period is set by whoever is doing the stopping and is named in
its own settings; what matters for the lesson is not the particular number but
that the period is finite and that what follows it cannot be caught. The
difference between "stopped in milliseconds" and "stopped after the full period"
is exactly the presence of a handler.
The script reproduces that sequence with a two-second period:
stop time, SIGTERM handled 11 ms
stop time, no handler 2009 ms
The second number is not "a slow shutdown". It is the entire grace period plus the
time it takes the SIGKILL to land: the process is alive and well throughout,
the SIGTERM had no effect on it, and then SIGKILL arrives, which nothing
catches.
Then the arithmetic of exit codes. The kernel reports two different things: the code a process exited with, and the number of the signal that killed it. Whoever prints the result is the one who adds them up:
When a command terminates on a fatal signal N, bash uses the value of 128+N as
the exit status.
Hence the table people ask about:
4. EXIT CODES: 128 + signal number
----------------------------------
exited on its own 3
SIGTERM, default disposition -15 (the shell reports 143)
SIGKILL -9 (the shell reports 137)
143 is 128 + 15, SIGTERM. 137 is 128 + 9, SIGKILL. The minus fifteen is how
subprocess returns a signal number: negative, so that it cannot be mistaken
for an exit code.
And the most useful part is what 137 does not say. It says the process was
killed by SIGKILL. It does not say who sent it. The OOM killer, the
orchestrator after the grace period, a human with kill -9 — indistinguishable
in that field, and the application log will hold nothing, because SIGKILL runs
no line of code. Telling them apart takes other evidence: the kernel message,
the orchestrator's event, the timing.
Deeper: why containers ship an init
Now the parts fit together. Your application became PID 1, which means:
- default signal actions are not carried out for it, so
SIGTERMwithout a handler does nothing; - every orphan in the container is adopted by it, so collecting their statuses is its job too.
An ordinary application does neither: it never expected to be the first process. Hence the two classic failures — a container that only stops on timeout, and a container quietly accumulating zombies from every subprocess it spawns.
Hence the role that small init processes (tini, dumb-init, the
docker run --init flag) are written for: take PID 1 and cover both duties —
forward signals to the child and reap orphans. The rule about default
actions becomes that process's problem rather than yours; what any particular
one of them does is stated in its own documentation.
The alternative is not a library but two lines in the application: install a
SIGTERM handler and wait for children. If the application spawns no
processes, the first line is enough.
How to answer in an interview
Short answer: for the process with ID 1 in its own namespace the default
action of a signal is not carried out, so SIGTERM without a handler does
nothing to it — the orchestrator waits out the grace period and finishes the
job with SIGKILL, leaving exit code 137, that is 128 + 9, in the report. A
zombie, meanwhile, is not a process but the record of a death: it is held for the
exit code, it occupies a slot in the process table, and what removes it is
wait, not time.
That is enough for a correct answer. What follows is what you add when the interviewer digs.
If the interviewer digs deeper
Two things separate a good answer. First, you call the rule a kernel rule
rather than a Docker quirk, and you know the exception: SIGKILL and SIGSTOP
from an ancestor namespace are delivered forcibly, so stopping always works and
only the delay is in question. Second, you say what 137 does not report: who
sent the signal. "137 means OOM" turns a piece of
evidence into a diagnosis, and those are different things.
Next they ask
Why is the application log empty when a container "crashes"?
Because SIGKILL runs no line of code: it has no handler, the deferred log
write never happens, buffers are never flushed. Silence in the log is not lost
messages — it is a sign of which signal ended things.
Had SIGTERM arrived with a handler in place, the log would show the start of a
shutdown. The practical consequence: such a line is how you tell an orderly stop
from a kill, and it is worth writing.
Does the shell forward the signal if ENTRYPOINT says sh -c "app"?
Usually not: PID 1 is then the shell, not the application, and the shell is not
obliged to forward signals to its child. SIGTERM reaches a shell that has no
handler for it, and the application never learns of it at all.
That is why the shell form is replaced by the exec form (ENTRYPOINT ["app"]),
where the application itself becomes PID 1 — or an init is put in front of it,
since forwarding signals is what an init does.
Can thousands of zombies exhaust memory?
Not memory in the sense counted for a live process: the address space was
released at termination, and what the kernel keeps is a small record — PID,
termination status, resource usage. What runs out first is not memory but the
process table: wait(2) states that a zombie consumes a slot, and once the
table fills, no new process can be created.
It shows up as a failure where nobody is looking: fork starts failing in some
unrelated process. The memory graph stays flat, which is exactly why this leak
takes so long to find. Measured: a thousand finished children with no wait
give a thousand zombies — 100% of them — and zero after wait.
What happens if PID 1 itself exits?
The namespace ends with it, and that is written on the same page:
If the "init" process of a PID namespace terminates, the kernel terminates all
of the processes in the namespace via a SIGKILL signal.
This is the mechanism behind the advice that no work should outlive the container's main process — it will not outlive it, whether or not it was ready to stop.
The practical consequence: draining connections and flushing data must happen before PID 1 exits, not in some background thread that "will get to it".
Common misconceptions
a zombie process sits in memory and has to be killed
There is nothing to kill: the process is gone and what remains is the record of its death — PID, termination status, resource usage. Its address space is gone, but a slot in the process table is not, and wait(2) states plainly that once the table fills "it will not be possible to create further processes". The record is removed by the parent's wait; kill -9 on a zombie changes nothing, since signals are not delivered to a dead process.
a container that ignores SIGTERM is a Docker quirk
It is a kernel rule, written down in pid_namespaces(7): the first process of a namespace only receives signals for which it has installed a handler. Docker has nothing to do with it — it merely starts your process first. Measured: the same SIGTERM, sent from outside, leaves the handler-less process alive and ends the one with a handler.
exit code 137 means the process ran out of memory
137 is 128 + 9, that is "killed by SIGKILL", and the code does not report who sent it. The OOM killer, the orchestrator after the grace period and a human with kill -9 all produce 137. The diagnosis comes from other evidence: the kernel message, the orchestrator's event, the timing relative to a rollout.
handling SIGTERM is optional — the orchestrator will stop the process anyway
It will, but only after the full grace period and by SIGKILL, which has no handler: open connections are cut, unwritten data stays unwritten, the log holds nothing. Measured with a two-second grace period: 11 ms with a handler against 2009 ms without — and the second figure is waiting, nothing else.
the operating system adopts orphans and clears zombies away
A specific process adopts them: init or the nearest subreaper (PR_SET_CHILD_SUBREAPER). The difference is practical: a system's init collects statuses, while your application in the role of PID 1 does not. That is why orphans turn into zombies inside a container and not outside one.
tini and dumb-init are about starting the application correctly
They are not about starting anything. They cover the two duties of PID 1 that an application does not have: forwarding signals to the child and reaping orphans. An application that does both needs no init; one that spawns no processes needs only a SIGTERM handler.
SIGKILL can be caught if you really need to
It cannot: signal(7) says it outright — "SIGKILL and SIGSTOP cannot be caught, blocked, or ignored". That is precisely why an orchestrator finishes with it: it is the one signal whose outcome does not depend on application code. And for the same reason it never leaves a log line behind.
Practice
Two exercises. Answer first, then check against the real output: in both, the correct answer comes from a script's run rather than from an editor.
Practice · predict the output
pid = os.fork() if pid == 0: os._exit(7) time.sleep(0.2) print(state_of(pid)) print(os.waitstatus_to_exitcode(os.waitpid(pid, 0)[1])) print(orphan_ppid()) print(pid1_state_after_sigterm(SLEEPER)) killed = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(5)"]) time.sleep(0.2) killed.send_signal(signal.SIGKILL) print(128 - killed.wait())
Practice · estimate
Knowledge check
A child process has terminated and the parent never called wait. How much memory does the child hold?
This is neither a retelling nor a separate text: everything below is taken from the article itself — its own summary, the section headings, the “actually” column and the version table. Which is why these theses cannot drift from the article.
The gist
- A process dies in two events, not one. First it stops running, then somebody collects its status; between those two events what is left of it is a record, and that record is called a zombie. And the kernel shields the process that came first inside a container: a signal it has installed no handler for does nothing to it at all. That is a kernel rule, not a Docker quirk.
- Hence the consequence that matters: an image whose first process does not handle
SIGTERMdoes not stop ondocker stop. It sits out the whole grace period and is killed bySIGKILLat the end of it. Measured with a two-second grace period: 11 ms with a handler against 2009 ms without one. The second failure is quieter: unreaped children do not clear up on their own — a thousand finished children with nowaitleft a thousand zombies, 100% of them, and zero afterwait. - Beyond that is what separates knowing from having read. An orphan is adopted by a process rather than by "the system" — by init or by the nearest subreaper: measured, a grandchild's parent changes from a real PID to
1at the moment its father dies. For the process with ID 1 in its own namespace, default signal actions are not carried out, and the single exception isSIGKILLandSIGSTOP: sent from an ancestor namespace they are delivered forcibly and cannot be caught by anything, so the orchestrator always wins and only the delay is in question. 137 is 128 + 9 and 143 is 128 + 15, and the addition is the shell's rather than the kernel's: 137 means "killed bySIGKILL" and says nothing about who sent it. And a zombie "sits in memory" only as a figure of speech: the process's address space is gone, but the kernel keeps a small record — PID, termination status, resource usage — and that record holds a slot in the process table.
In fact
- There is nothing to kill: the process is gone and what remains is the record of its death — PID, termination status, resource usage. Its address space is gone, but a slot in the process table is not, and
wait(2)states plainly that once the table fills "it will not be possible to create further processes". The record is removed by the parent'swait;kill -9on a zombie changes nothing, since signals are not delivered to a dead process. - It is a kernel rule, written down in
pid_namespaces(7): the first process of a namespace only receives signals for which it has installed a handler. Docker has nothing to do with it — it merely starts your process first. Measured: the sameSIGTERM, sent from outside, leaves the handler-less process alive and ends the one with a handler. - 137 is 128 + 9, that is "killed by
SIGKILL", and the code does not report who sent it. The OOM killer, the orchestrator after the grace period and a human withkill -9all produce 137. The diagnosis comes from other evidence: the kernel message, the orchestrator's event, the timing relative to a rollout. - It will, but only after the full grace period and by
SIGKILL, which has no handler: open connections are cut, unwritten data stays unwritten, the log holds nothing. Measured with a two-second grace period: 11 ms with a handler against 2009 ms without — and the second figure is waiting, nothing else. - A specific process adopts them:
initor the nearest subreaper (PR_SET_CHILD_SUBREAPER). The difference is practical: a system's init collects statuses, while your application in the role of PID 1 does not. That is why orphans turn into zombies inside a container and not outside one. - They are not about starting anything. They cover the two duties of PID 1 that an application does not have: forwarding signals to the child and reaping orphans. An application that does both needs no init; one that spawns no processes needs only a
SIGTERMhandler. - It cannot:
signal(7)says it outright — "SIGKILL and SIGSTOP cannot be caught, blocked, or ignored". That is precisely why an orchestrator finishes with it: it is the one signal whose outcome does not depend on application code. And for the same reason it never leaves a log line behind.
What is covered
- What is actually being asked
- Base: a process, its number, and the message you send it
- Mechanism 1: a process dies twice, not once
- Mechanism 2: an orphan gets a new parent, and it is a real one
- Mechanism 3: the PID 1 rule that everything else grows from
- Mechanism 4: what a missing handler costs, and where 143 and 137 come from
- Deeper: why containers ship an init
- How to answer in an interview
- Next they ask
- Common misconceptions
- Practice
- Knowledge check
Sources & further reading
4 SOURCES
- wait(2), Linux man-pages 6.7Official documentation. What a zombie is and why the kernel keeps the record: "A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child". And where it ends: "As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes". The adoption rule comes from the same page: "If a parent process terminates, then its "zombie" children (if any) are adopted by init(1), (or by the nearest "subreaper" process as defined through the use of the prctl(2) PR_SET_CHILD_SUBREAPER operation)".https://man7.org/linux/man-pages/man2/wait.2.html
- pid_namespaces(7), Linux man-pages 6.7Official documentation. The rule the whole lesson grows from: "Only signals for which the "init" process has established a signal handler can be sent to the "init" process by other members of the PID namespace. This restriction applies even to privileged processes, and prevents other members of the PID namespace from accidentally killing the "init" process". The same holds from the outside, which is the orchestrator's case: "a process in an ancestor namespace can—subject to the usual permission checks described in kill(2)—send signals to the "init" process of a child PID namespace only if the "init" process has established a handler for that signal". And the exception the orchestrator relies on: "SIGKILL or SIGSTOP are treated exceptionally: these signals are forcibly delivered when sent from an ancestor PID namespace".https://man7.org/linux/man-pages/man7/pid_namespaces.7.html
- signal(7), Linux man-pages 6.7Official documentation. The table of default dispositions, where SIGTERM is listed as `Term`, and the boundary no handler crosses: "SIGKILL and SIGSTOP cannot be caught, blocked, or ignored". The same page defines disposition — what a process does with a signal: the default action, ignoring it, or a handler.https://man7.org/linux/man-pages/man7/signal.7.html
- Bash Reference Manual, Exit StatusOfficial documentation. Where the 128+N convention printed by the shell and by the reports around it comes from: "When a command terminates on a fatal signal N, bash uses the value of 128+N as the exit status". This is a shell convention rather than a kernel one: the kernel reports the signal number separately from the exit code, and the addition is done by whoever prints the result.https://www.gnu.org/software/bash/manual/bash.html#Exit-Status