Deep Engineering
Intermediate·Published·25 MIN

Signals, zombies and PID 1: why a container ignores SIGTERM

The questions come as a ladder: what a zombie is — who adopts an orphan — why a container does not stop at once — what exit code 137 means. The first two rest on a process dying in two separate events; the last two rest on one kernel rule that is almost never said out loud: for the process with ID 1 in its namespace, the default action of a signal is not carried out.

Full technical treatment

TL;DR

A process dies in two events, not one. First it stops running, then somebody collects its status; between those two events what is left of it is a record, and that record is called a zombie. And the kernel shields the process that came first inside a container: a signal it has installed no handler for does nothing to it at all. That is a kernel rule, not a Docker quirk.

Hence the consequence that matters: an image whose first process does not handle SIGTERM does not stop on docker stop. It sits out the whole grace period and is killed by SIGKILL at the end of it. Measured with a two-second grace period: 11 ms with a handler against 2009 ms without one. The second failure is quieter: unreaped children do not clear up on their own — a thousand finished children with no wait left a thousand zombies, 100% of them, and zero after wait.

Beyond that is what separates knowing from having read. An orphan is adopted by a process rather than by "the system" — by init or by the nearest subreaper: measured, a grandchild's parent changes from a real PID to 1 at the moment its father dies. For the process with ID 1 in its own namespace, default signal actions are not carried out, and the single exception is SIGKILL and SIGSTOP: sent from an ancestor namespace they are delivered forcibly and cannot be caught by anything, so the orchestrator always wins and only the delay is in question. 137 is 128 + 9 and 143 is 128 + 15, and the addition is the shell's rather than the kernel's: 137 means "killed by SIGKILL" and says nothing about who sent it. And a zombie "sits in memory" only as a figure of speech: the process's address space is gone, but the kernel keeps a small record — PID, termination status, resource usage — and that record holds a slot in the process table.

Where to start
Before this lesson it is enough to understand
  • a program gets started and eventually ends — by itself or because something stopped it;
  • one running program can start another;
  • somebody starts a container and at some point asks it to stop — during a rollout, for instance.
You do not need to know in advance
  • what a zombie, an orphan, adoption and a subreaper are;
  • what a process namespace is and what makes the number 1 special in one;
  • SIGTERM, SIGKILL, a signal's disposition, exit codes 137 and 143.

What is actually being asked

The ladder is almost always the same, and it rests on a single mechanism even though the questions look unrelated:

  1. "What is a zombie process?" — the warm-up that filters out anyone who thinks it means a hung process.
  2. "Why is it a problem if the process itself is already gone?" — where the substance starts.
  3. "Who becomes the parent of an orphaned process?" — a check on adoption and on subreapers.
  4. "Why does a container not stop at once?" — where the tempting answer is "it closes connections slowly": plausible, and past the mechanism.
  5. "What does exit code 137 mean?" — whether you know the arithmetic and what the code does not tell you.
  6. "Why do people put tini in a container?" — whether you have connected the first five into one thing.

The lesson climbs that ladder in one direction: the rule first, its consequences after. Every number below was produced on one machine by bench/signals/lifecycle.py and bench/signals/practice.py; your process IDs and millisecond timings will differ, while the states, the exit codes and the outcome of SIGTERM for PID 1 will not.

Base: a process, its number, and the message you send it

Before talking about zombies and about the number 1, it is worth naming, in ordinary words, the four things the whole lesson rests on.

A process is a running program. Not the file on disk but its run: the same program started twice is two processes, and they live independently of each other.

A process has a number. The kernel hands it out at startup, and that number is how the process is addressed afterwards: to look at it, to stop it, or to wait for its end.

A signal is a short message to a process. The kernel or another process sends it, and it carries exactly as much meaning as its own number: "finish up nicely", "stop right now", "suspend". No text, no reply: a signal either did something to the process or did nothing at all.

Processes have a parent and a child — the one that did the starting and the one that was started. That link is not a formality: it is the parent that later receives the answer to how the child ended.

The first rung of the ladder follows straight from those last two points. A child has finished; the parent has not yet asked how it ended; the kernel needs somewhere to keep the answer to that future question — and it keeps it. The state "the process is gone but the record of how it ended is still there" is what a zombie is.

And one more thing, without which the second half of the lesson reads as magic: a container has its own space of numbers. Processes inside it are numbered from one, as in a machine that has just booted, while the very same process has a completely different number outside. One process, two numbers: 1 inside, an ordinary one outside. Whoever stops the container lives outside and signals the external number — and the rules that apply to the process are the ones that apply to the number 1.

That is already enough to answer the basic interview question. What follows is about why the same signal acts differently on an ordinary process and on the process numbered 1, who gets the child when its parent dies first, and where the 137 in a report comes from.

Mechanism 1: a process dies twice, not once

Now the same thing in the documentation's words — starting with what does not depend on the distribution or the kernel version.

language contractBehaviour specified in wait(2) and pid_namespaces(7). The numbers later in the lesson were taken on Linux 6.18, but the rules below are older than any kernel version still in service.

A process ends in two separate events. First it stops running — it calls exit, takes a fatal signal, crashes. Then somebody collects its status by calling wait. Between those two events the process is gone and the record of it is not.

That record is the zombie, and wait(2) says so directly:

A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child.

wait(2)

That answers the second rung immediately. A zombie is not dangerous because it "sits in memory": the process's address space is gone, and what remains is the minimal set of information listed in the quotation above. It is dangerous because that record holds a slot in the process table, and the table is finite:

As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes.

wait(2)

This is a leak with nothing to see on the memory graph. The application runs, memory is flat, and then fork starts failing — in some process that has nothing to do with the one at fault.

Here it is in a run. The child called _exit(7) and was left unwaited:

measured observationbench/signals/lifecycle.py, Linux 6.18.44. The state letter, the exit code and the orphan's parent ID reproduce on any Linux; the process IDs do not.
1. ZOMBIE: the child is gone, the record is not
-----------------------------------------------
  child finished, status not collected       Z (zombie)
  why the kernel keeps the record            so the parent can read the exit code
  state after wait                           no such process
  exit code the record was held for          7

Four lines that read as one story: state Z before wait, nothing after it, and the seven the record was being held for. What removes a zombie is not time and not a collector — it is the question "how did it end", asked by the parent.

Mechanism 2: an orphan gets a new parent, and it is a real one

The mirror case: the parent dies before the child. The child neither dies nor becomes parentless — it is adopted.

If a parent process terminates, then its "zombie" children (if any) are adopted by init(1), (or by the nearest "subreaper" process as defined through the use of the prctl(2) PR_SET_CHILD_SUBREAPER operation).

wait(2)

The run catches the substitution itself. A grandchild prints its parent twice — right after it starts, and after its father has exited:

2. ORPHAN: the parent died first
--------------------------------
  grandchild: parent right after start    5359
  grandchild: parent after father exits   1

The 5359 will be different on your machine; the 1 will not. And here is what follows for real work: the new parent is the one who will collect the orphan's status, not the lost one. If the new parent knows how, zombies never pile up. If it does not, every orphan piles up at once — which is exactly what happens in a container whose first process is an ordinary application.

The word "subreaper" is not decoration. PR_SET_CHILD_SUBREAPER is how a process tells the kernel: orphans from my subtree come to me, not to PID 1. Session managers and supervision systems use it; a container usually has none, so orphans go straight to PID 1.

Mechanism 3: the PID 1 rule that everything else grows from

Now the important part. A process that received ID 1 in its own namespace handles signals differently, and pid_namespaces(7) writes it down:

Only signals for which the "init" process has established a signal handler can be sent to the "init" process by other members of the PID namespace. This restriction applies even to privileged processes, and prevents other members of the PID namespace from accidentally killing the "init" process.

pid_namespaces(7)

From the outside it works the same way, and that is precisely the orchestrator's case: it lives in the ancestor namespace and sends a signal to your first process.

a process in an ancestor namespace can—subject to the usual permission checks described in kill(2)—send signals to the "init" process of a child PID namespace only if the "init" process has established a handler for that signal.

pid_namespaces(7)

Put in the words worth answering with: for PID 1 the default action of a signal is not carried out. A signal's disposition is what the process does with it: the default action from the signal(7) table (SIGTERM is listed as Term — terminate), ignoring it, or a handler. An ordinary process without a handler gets the first. PID 1 has no first: with a handler the signal works, without one it does nothing.

The same SIGTERM, two processes that differ by one line:

3. PID 1: one SIGTERM, two outcomes
-----------------------------------
  PID inside the namespace (no handler)      1
  SIGTERM from outside, no handler (PID 5362) ALIVE: the signal did nothing
  PID inside the namespace (with handler)    1
  SIGTERM from outside, with handler (PID 5364) exited
measured observationbench/signals/lifecycle.py, Linux 6.18.44, unshare --fork --pid --mount-proc. Process IDs will differ on your machine; the outcome on both lines will not.

Note that the signal is sent from outside and by the external ID (5362), while inside the namespace that process is number 1. The orchestrator does the same: it does not step inside the container, it signals a process that looks ordinary from where it stands.

There is exactly one exception:

SIGKILL or SIGSTOP are treated exceptionally: these signals are forcibly delivered when sent from an ancestor PID namespace.

pid_namespaces(7)

So "the container cannot be stopped" is the wrong phrasing. It can always be stopped; the question is whether the orchestrator burns the whole grace period first.

Mechanism 4: what a missing handler costs, and where 143 and 137 come from

Stopping a container is three steps: SIGTERM, a grace period, SIGKILL. The length of that period is set by whoever is doing the stopping and is named in its own settings; what matters for the lesson is not the particular number but that the period is finite and that what follows it cannot be caught. The difference between "stopped in milliseconds" and "stopped after the full period" is exactly the presence of a handler.

The script reproduces that sequence with a two-second period:

stop time, SIGTERM handled           11 ms
stop time, no handler                2009 ms
measured observationbench/signals/practice.py, with a 2 s grace period in this run. Your milliseconds will differ: the first number depends on the scheduler, the second on the period chosen. What holds is the point of the measurement: with a handler the stop takes milliseconds, without one it takes the whole period.

The second number is not "a slow shutdown". It is the entire grace period plus the time it takes the SIGKILL to land: the process is alive and well throughout, the SIGTERM had no effect on it, and then SIGKILL arrives, which nothing catches.

Then the arithmetic of exit codes. The kernel reports two different things: the code a process exited with, and the number of the signal that killed it. Whoever prints the result is the one who adds them up:

When a command terminates on a fatal signal N, bash uses the value of 128+N as the exit status.

Bash Reference Manual

Hence the table people ask about:

4. EXIT CODES: 128 + signal number
----------------------------------
  exited on its own                          3
  SIGTERM, default disposition               -15 (the shell reports 143)
  SIGKILL                                    -9 (the shell reports 137)

143 is 128 + 15, SIGTERM. 137 is 128 + 9, SIGKILL. The minus fifteen is how subprocess returns a signal number: negative, so that it cannot be mistaken for an exit code.

And the most useful part is what 137 does not say. It says the process was killed by SIGKILL. It does not say who sent it. The OOM killer, the orchestrator after the grace period, a human with kill -9 — indistinguishable in that field, and the application log will hold nothing, because SIGKILL runs no line of code. Telling them apart takes other evidence: the kernel message, the orchestrator's event, the timing.

Deeper: why containers ship an init

Now the parts fit together. Your application became PID 1, which means:

  • default signal actions are not carried out for it, so SIGTERM without a handler does nothing;
  • every orphan in the container is adopted by it, so collecting their statuses is its job too.

An ordinary application does neither: it never expected to be the first process. Hence the two classic failures — a container that only stops on timeout, and a container quietly accumulating zombies from every subprocess it spawns.

Hence the role that small init processes (tini, dumb-init, the docker run --init flag) are written for: take PID 1 and cover both duties — forward signals to the child and reap orphans. The rule about default actions becomes that process's problem rather than yours; what any particular one of them does is stated in its own documentation.

The alternative is not a library but two lines in the application: install a SIGTERM handler and wait for children. If the application spawns no processes, the first line is enough.

How to answer in an interview

Short answer: for the process with ID 1 in its own namespace the default action of a signal is not carried out, so SIGTERM without a handler does nothing to it — the orchestrator waits out the grace period and finishes the job with SIGKILL, leaving exit code 137, that is 128 + 9, in the report. A zombie, meanwhile, is not a process but the record of a death: it is held for the exit code, it occupies a slot in the process table, and what removes it is wait, not time.

That is enough for a correct answer. What follows is what you add when the interviewer digs.

If the interviewer digs deeper

Two things separate a good answer. First, you call the rule a kernel rule rather than a Docker quirk, and you know the exception: SIGKILL and SIGSTOP from an ancestor namespace are delivered forcibly, so stopping always works and only the delay is in question. Second, you say what 137 does not report: who sent the signal. "137 means OOM" turns a piece of evidence into a diagnosis, and those are different things.

Next they ask

Next they ask

Why is the application log empty when a container "crashes"?

Short answer

Because SIGKILL runs no line of code: it has no handler, the deferred log write never happens, buffers are never flushed. Silence in the log is not lost messages — it is a sign of which signal ended things.

Had SIGTERM arrived with a handler in place, the log would show the start of a shutdown. The practical consequence: such a line is how you tell an orderly stop from a kill, and it is worth writing.

Next they ask

Does the shell forward the signal if ENTRYPOINT says sh -c "app"?

Short answer

Usually not: PID 1 is then the shell, not the application, and the shell is not obliged to forward signals to its child. SIGTERM reaches a shell that has no handler for it, and the application never learns of it at all.

That is why the shell form is replaced by the exec form (ENTRYPOINT ["app"]), where the application itself becomes PID 1 — or an init is put in front of it, since forwarding signals is what an init does.

Next they ask

Can thousands of zombies exhaust memory?

Short answer

Not memory in the sense counted for a live process: the address space was released at termination, and what the kernel keeps is a small record — PID, termination status, resource usage. What runs out first is not memory but the process table: wait(2) states that a zombie consumes a slot, and once the table fills, no new process can be created.

It shows up as a failure where nobody is looking: fork starts failing in some unrelated process. The memory graph stays flat, which is exactly why this leak takes so long to find. Measured: a thousand finished children with no wait give a thousand zombies — 100% of them — and zero after wait.

Next they ask

What happens if PID 1 itself exits?

Short answer

The namespace ends with it, and that is written on the same page:

If the "init" process of a PID namespace terminates, the kernel terminates all of the processes in the namespace via a SIGKILL signal.

pid_namespaces(7)

This is the mechanism behind the advice that no work should outlive the container's main process — it will not outlive it, whether or not it was ready to stop.

The practical consequence: draining connections and flushing data must happen before PID 1 exits, not in some background thread that "will get to it".

Common misconceptions

Claim

a zombie process sits in memory and has to be killed

Actually

There is nothing to kill: the process is gone and what remains is the record of its death — PID, termination status, resource usage. Its address space is gone, but a slot in the process table is not, and wait(2) states plainly that once the table fills "it will not be possible to create further processes". The record is removed by the parent's wait; kill -9 on a zombie changes nothing, since signals are not delivered to a dead process.

Claim

a container that ignores SIGTERM is a Docker quirk

Actually

It is a kernel rule, written down in pid_namespaces(7): the first process of a namespace only receives signals for which it has installed a handler. Docker has nothing to do with it — it merely starts your process first. Measured: the same SIGTERM, sent from outside, leaves the handler-less process alive and ends the one with a handler.

Claim

exit code 137 means the process ran out of memory

Actually

137 is 128 + 9, that is "killed by SIGKILL", and the code does not report who sent it. The OOM killer, the orchestrator after the grace period and a human with kill -9 all produce 137. The diagnosis comes from other evidence: the kernel message, the orchestrator's event, the timing relative to a rollout.

Claim

handling SIGTERM is optional — the orchestrator will stop the process anyway

Actually

It will, but only after the full grace period and by SIGKILL, which has no handler: open connections are cut, unwritten data stays unwritten, the log holds nothing. Measured with a two-second grace period: 11 ms with a handler against 2009 ms without — and the second figure is waiting, nothing else.

Claim

the operating system adopts orphans and clears zombies away

Actually

A specific process adopts them: init or the nearest subreaper (PR_SET_CHILD_SUBREAPER). The difference is practical: a system's init collects statuses, while your application in the role of PID 1 does not. That is why orphans turn into zombies inside a container and not outside one.

Claim

tini and dumb-init are about starting the application correctly

Actually

They are not about starting anything. They cover the two duties of PID 1 that an application does not have: forwarding signals to the child and reaping orphans. An application that does both needs no init; one that spawns no processes needs only a SIGTERM handler.

Claim

SIGKILL can be caught if you really need to

Actually

It cannot: signal(7) says it outright — "SIGKILL and SIGSTOP cannot be caught, blocked, or ignored". That is precisely why an orchestrator finishes with it: it is the one signal whose outcome does not depend on application code. And for the same reason it never leaves a log line behind.

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer comes from a script's run rather than from an editor.

Practice · predict the output

Five observations in a row: the state of an unreaped child, its exit code, the parent ID of an orphan, the state of a handler-less PID 1 one second after SIGTERM, and the exit code of a process killed by SIGKILL. States are the single letters from /proc: S is sleeping, Z is a zombie. What does this code print?
pid = os.fork()
if pid == 0:
  os._exit(7)
time.sleep(0.2)
print(state_of(pid))
print(os.waitstatus_to_exitcode(os.waitpid(pid, 0)[1]))
print(orphan_ppid())
print(pid1_state_after_sigterm(SLEEPER))
killed = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(5)"])
time.sleep(0.2)
killed.send_signal(signal.SIGKILL)
print(128 - killed.wait())

Practice · estimate

The grace period is two seconds. How many seconds does it take to stop a PID 1 that does not handle SIGTERM?
s

Knowledge check

Question 1 of 6

A child process has terminated and the parent never called wait. How much memory does the child hold?

Sources & further reading

4 SOURCES

  1. wait(2), Linux man-pages 6.7Official documentation. What a zombie is and why the kernel keeps the record: "A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child". And where it ends: "As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes". The adoption rule comes from the same page: "If a parent process terminates, then its "zombie" children (if any) are adopted by init(1), (or by the nearest "subreaper" process as defined through the use of the prctl(2) PR_SET_CHILD_SUBREAPER operation)".https://man7.org/linux/man-pages/man2/wait.2.html
  2. pid_namespaces(7), Linux man-pages 6.7Official documentation. The rule the whole lesson grows from: "Only signals for which the "init" process has established a signal handler can be sent to the "init" process by other members of the PID namespace. This restriction applies even to privileged processes, and prevents other members of the PID namespace from accidentally killing the "init" process". The same holds from the outside, which is the orchestrator's case: "a process in an ancestor namespace can—subject to the usual permission checks described in kill(2)—send signals to the "init" process of a child PID namespace only if the "init" process has established a handler for that signal". And the exception the orchestrator relies on: "SIGKILL or SIGSTOP are treated exceptionally: these signals are forcibly delivered when sent from an ancestor PID namespace".https://man7.org/linux/man-pages/man7/pid_namespaces.7.html
  3. signal(7), Linux man-pages 6.7Official documentation. The table of default dispositions, where SIGTERM is listed as `Term`, and the boundary no handler crosses: "SIGKILL and SIGSTOP cannot be caught, blocked, or ignored". The same page defines disposition — what a process does with a signal: the default action, ignoring it, or a handler.https://man7.org/linux/man-pages/man7/signal.7.html
  4. Bash Reference Manual, Exit StatusOfficial documentation. Where the 128+N convention printed by the shell and by the reports around it comes from: "When a command terminates on a fatal signal N, bash uses the value of 128+N as the exit status". This is a shell convention rather than a kernel one: the kernel reports the signal number separately from the exit code, and the addition is done by whoever prints the result.https://www.gnu.org/software/bash/manual/bash.html#Exit-Status