Processes — Fork, Exec, Signals, Zombies, Cgroups

What it is

A process is the OS abstraction for a running program: a unique identifier (PID), a private virtual address space, a set of open file descriptors, a security context (UID/GID, capabilities), and a state machine the kernel transitions through scheduling decisions. On Unix, processes are created by cloning an existing one with fork() (the new child is a near-perfect copy of the parent) and optionally replacing the running program with execve(). That two-step model — fork then exec — is the foundation of every shell, every sudo, every container runtime, and every service manager. Reach for this article when something has gone wrong with a runaway service, a zombie process, a signal that isn't doing what you expect, or a cgroup limit that's killing your workload.

The process lifecycle

Every Unix process goes through the same five-state cycle from creation to reaping. Understanding which state a process is in tells you what to do about it: a sleeping process is fine, a zombie is your problem to reap, a stopped one is waiting on a signal you haven't sent.

ps -eo pid,state,comm | head

Output:

  PID S COMMAND
    1 S systemd
    2 S kthreadd
    3 I rcu_gp
   18 S ksoftirqd/0
  423 S systemd-journal
  611 S rsyslogd
  800 S sshd
 1234 S nginx
 8821 S bash
 9514 R ps

[!WARN] A process stuck in D (uninterruptible sleep) cannot be killed by any signal, not even SIGKILL. The usual cause is a hung NFS mount or a misbehaving block device. The fix is to recover the underlying I/O — terminating the device or the NFS server — not to escalate the signal.

fork — clone the current process

fork() is the canonical Unix syscall for creating a new process. It returns twice: zero in the child, and the new child's PID in the parent. The child inherits everything — file descriptors, memory, environment, signal mask, working directory — but begins with a fresh PID and a parent pointer back to the caller. Memory is shared copy-on-write so fork itself is cheap; the cost is paid only when the child writes a page.

// Minimal C example
pid_t pid = fork();
if (pid == 0) {
    // child
    printf("I am the child, my PID is %d\n", getpid());
} else if (pid > 0) {
    // parent
    printf("I forked child %d\n", pid);
    waitpid(pid, NULL, 0);    // reap when it exits
} else {
    perror("fork");
}

In shells, you observe fork every time you run a command: the shell forks, the child execs the command, the parent waits.

# Show parent and PID of every child the shell launches
( echo "child PID: $$"; echo "parent PID: $PPID" )

Output:

child PID: 12031
parent PID: 8821

The parentheses run the block in a subshell — a fresh process whose PPID is the original shell.

exec — replace the current program

execve() (and its execvp, execlp, execle wrappers) replaces the current process's memory image with a new program. The PID does not change; everything else does — code, data, heap, even most of /proc/self/*. Reach for this in C when you want to run another program but keep the current process record (PID, parent, open FDs that aren't O_CLOEXEC). Shells implement exec as a builtin that replaces the shell itself with the named command — useful for entrypoint scripts that should not leave a shell wrapper running.

# Inside a shell — replace the shell with python; the shell no longer exists.
exec python3 -m http.server 8000

# After the python process exits, the terminal session ends.

Output:

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

The fork+exec idiom is the standard pattern: fork() to spawn a new process, execve() in the child to load the desired program. Without exec, the child would still be running the parent's code.

wait — reap a child

After a child exits, its exit status and resource usage remain in the kernel as a zombie (state Z) until the parent calls wait() or waitpid(). A zombie is essentially nothing — no memory, no file descriptors — just an entry in the process table waiting to be reaped. Failing to reap is one of the most common bugs in long-running daemons that spawn children.

# Spot a zombie
ps -eo pid,ppid,state,comm | awk '$3=="Z"'

Output:

14201 14200 Z   defunct

The fix is to fix the parent (waitpid in code, or set SIGCHLD to SA_NOCLDWAIT). If the parent is gone, the zombie is automatically reparented to PID 1 (init/systemd), which reaps it immediately — see orphans below.

Zombies are harmless in small numbers — they consume one process-table entry each. They only become a problem when a leak runs the process table dry (the kernel's pid_max is typically 4 million but defaults to 32k on smaller systems).

PIDs and PPIDs

Every process has a unique numeric PID and a parent PID (PPID) pointing at the process that forked it. PIDs are assigned monotonically from 1 (init/systemd) and wrap around at pid_max. PID 0 is the kernel scheduler placeholder; PID 1 is init and is special — it inherits orphans and cannot be killed.

# Inspect PID + PPID for this shell and its parent
echo "shell PID=$$, parent PID=$PPID"

# Print the whole process tree
pstree -p | head -10

Output:

shell PID=8821, parent PID=8820

systemd(1)─┬─NetworkManager(412)
           ├─cron(523)
           ├─sshd(800)─┬─sshd(8800)─┬─sshd(8820)───bash(8821)───pstree(12053)
           │           │            └─sftp-server(8822)
           ├─systemd-journal(423)
           └─systemd-logind(522)

pstree is the human-readable view of /proc/<pid>/status:PPid. The first column of each row is the immediate parent.

Signals

A signal is a small, fixed-arity message the kernel (or another process via kill()) delivers to a target process. The receiving process either runs a registered handler, performs the default action (often termination), or — for SIGSTOP and SIGKILL — has no choice. Signals are how you ask a process to reload its config, terminate gracefully, or die immediately.

# List all signals on this system
kill -l | head

Output:

 1) SIGHUP   2) SIGINT   3) SIGQUIT  4) SIGILL   5) SIGTRAP
 6) SIGABRT  7) SIGBUS   8) SIGFPE   9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM

Sending signals

# Polite termination — the default
kill 1234

# Reload config (if the process handles SIGHUP)
kill -HUP 1234
kill -1 1234              # equivalent numeric form

# Force kill — only when SIGTERM didn't work
kill -KILL 1234
kill -9 1234

# By name (sends to every match — be careful)
pkill -HUP nginx
pkill -f "python.*server.py"   # match full command line

# Pause / resume
kill -STOP 1234
kill -CONT 1234

# Send to a process group (negative PID)
kill -TERM -1234

Output: (none — exits 0 on success)

[!WARN] kill -9 (SIGKILL) is not "kill harder" — it's "kill without giving the process a chance to clean up". The process cannot trap it, flush buffers, write a final log line, or remove its PID file. Always try kill (SIGTERM) first, wait a few seconds, then escalate.

Trapping signals in a script

A handler that runs on Ctrl-C, cleans up, then exits.

#!/usr/bin/env bash
tmp=$(mktemp -d)
cleanup() {
  echo "Caught signal, cleaning up $tmp"
  rm -rf "$tmp"
}
trap cleanup EXIT INT TERM
echo "Working in $tmp — press Ctrl-C to test"
sleep 1000

Output:

Working in /tmp/tmp.X9aQ2k — press Ctrl-C to test
^CCaught signal, cleaning up /tmp/tmp.X9aQ2k

trap CMD SIGNAL... is bash's signal handler. EXIT is the pseudo-signal that always fires at normal exit; combining it with INT TERM covers most cases.

Process groups and sessions

Every process belongs to a process group (PGID), and every process group belongs to a session (SID). The shell uses these to implement job control: when you type Ctrl-C, the kernel sends SIGINT to every process in the foreground process group at once; kill -9 -1234 sends SIGKILL to the whole group whose leader is PID 1234.

ps -e -o pid,ppid,pgid,sid,tty,comm | head

Output:

   PID    PPID    PGID     SID TT       COMMAND
     1       0       1       1 ?        systemd
   800       1     800     800 ?        sshd
  8800     800    8800    8800 ?        sshd
  8821    8800    8821    8821 pts/0    bash
  9530    8821    9530    8821 pts/0    sleep
  9531    8821    9531    8821 pts/0    ps

Each foreground command (sleep, ps) becomes its own process group leader; the shell remains the session leader. When the controlling terminal closes (pts/0 disconnects), the kernel sends SIGHUP to every process in the session — hence nohup, which ignores SIGHUP to keep a job running.

# Detach: ignore SIGHUP and redirect output, then disown
nohup ./long-running.sh > out.log 2>&1 &
disown

# Or use setsid to start a fresh session
setsid ./long-running.sh > out.log 2>&1 < /dev/null &

Output: (none — exits 0 on success)

Orphans

When a parent exits before its child, the child becomes an orphan. The kernel reparents orphans to PID 1 (init/systemd), which calls wait() on them as they exit, preventing zombie buildup. This is the basis of the daemon pattern: fork, the parent exits, the orphaned child runs forever under init.

# Watch reparenting happen
( sleep 100 & echo "child=$!" ) ; sleep 1 ; ps -eo pid,ppid,comm | grep sleep

Output:

child=14801
14801      1 sleep

The outer shell exited (the parens), and sleep was reparented to PID 1.

/proc/ — the per-process filesystem

Linux exposes everything about a running process under /proc/<pid>/. This is how ps, top, lsof, pgrep, and friends actually work — they read these pseudo-files. It's also a powerful debugging tool.

pid=$(pgrep -n bash)
ls /proc/$pid

Output:

attr        cgroup    cwd      environ  fd      io        limits  loginuid
maps        mem       mounts   net      ns      pagemap   root    sched
schedstat   smaps     stat     statm    status  syscall   task    wchan

# Show argv with spaces visible
tr '\0' ' ' < /proc/$pid/cmdline ; echo

# Reveal environment of a foreign process (root)
sudo tr '\0' '\n' < /proc/$pid/environ | head

# What is this PID actually running?
readlink /proc/$pid/exe

# Truncate a deleted log a process is still holding
sudo : > /proc/1234/fd/2

Output:

nginx: worker process
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/var/cache/nginx
USER=www-data
/usr/sbin/nginx

Cgroups

Control groups (cgroups) are the Linux kernel feature that lets you bound a set of processes' use of CPU, memory, I/O, and other resources. They are the layer beneath containers, systemd's resource controls, and OOM accounting. Every process is in exactly one cgroup per controller in the unified hierarchy (cgroup v2).

# What cgroup is this shell in?
cat /proc/self/cgroup

Output (cgroup v2):

0::/user.slice/user-1000.slice/session-1.scope

# Tree of cgroups on the system
systemctl status --no-pager | head -20

# Per-cgroup CPU and memory
systemd-cgtop

Output:

myhost
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Tue 2026-05-20 09:14:02 UTC; 5 days ago
   CGroup: /
           ├─user.slice
           │ └─user-1000.slice
           │   └─session-1.scope
           │     ├─2104 -bash
           │     └─2210 systemd-cgtop
           └─system.slice
             ├─sshd.service
             └─nginx.service

What's in a cgroup

Setting limits via systemd

The simplest way to constrain a service in production is via the unit file. systemd writes the cgroup limits for you.

# /etc/systemd/system/myapp.service
[Service]
ExecStart=/opt/myapp/bin/myapp
MemoryMax=512M             # hard ceiling — kernel kills the process if exceeded
MemoryHigh=400M            # throttle when above this
CPUQuota=50%               # 0.5 CPU of bandwidth
TasksMax=200               # cap PID count
IOWeight=50                # block I/O priority (1-10000, default 100)

sudo systemctl daemon-reload
sudo systemctl restart myapp
systemctl status myapp | head -20

Output:

● myapp.service - My Application
     Active: active (running) since Sun 2026-05-25 09:14:02 EDT; 12s ago
   Main PID: 4821 (myapp)
      Tasks: 7 (limit: 200)
     Memory: 312.4M (max: 512.0M, high: 400.0M)
        CPU: 312ms
     CGroup: /system.slice/myapp.service
             └─4821 /opt/myapp/bin/myapp

Ad-hoc resource limits

systemd-run makes a transient cgroup on the fly — useful for sandboxing a one-off heavy job.

# Run a build under a memory cap
systemd-run --user --scope -p MemoryMax=2G -p CPUQuota=200% \
  ./build.sh

# Inspect what's running and how it's limited
systemd-cgls

Output: (none — exits 0 on success)

See the systemd unit files cheatsheet for the full set of resource-control directives.

Common pitfalls

1. kill -9 as a first resort — robs the process of cleanup. Always start with SIGTERM, wait a few seconds, then escalate.
2. Zombies you can't kill — a zombie is already dead; you must kill its parent to make init reap it. ps -o pid,ppid,state,comm to find the parent.
3. Process stuck in D — uninterruptible sleep cannot be killed. Recover the underlying I/O (often NFS); a reboot may be the only escape.
4. SIGKILL doesn't always kill threads cleanly — it kills the whole process. To stop a single thread, the process must implement it; the kernel does not let you kill an individual thread from outside.
5. Forgot nohup and the SSH session died — the controlling terminal closed, SIGHUP fanned out to the session, and your job died. Use nohup, setsid, tmux, or systemd-run --scope next time.
6. Signal lost during forking — between fork() and execve(), signal handlers from the parent are still installed. Use pthread_sigmask or reset to SIG_DFL for safety before exec.
7. pkill matched too much — pkill bash kills every bash on the box, including your shell. Use pkill -f with a specific regex, and try pgrep first to preview matches.
8. Fork bomb collapses the system — without TasksMax on user sessions, a runaway script can exhaust pid_max. Set a per-user TasksMax= in /etc/systemd/system/user-.slice.d/ or use cgroup limits.
9. Daemon doesn't reload on SIGHUP — many tools (nginx, postgres, sshd) do; others (Go binaries, Node) don't unless coded to. kill -HUP is a convention, not a guarantee. Check the man page.
10. pstree -p doesn't show threads — pass -T to suppress them or -h to highlight the current process. Threads share the parent's PID and only differ in TID (thread ID).
11. OOM killer chose the wrong victim — by default it picks the highest-scoring process by RSS + child contribution. Tune oom_score_adj for critical services (-1000 makes the process immune).

Real-world recipes

Graceful shutdown — try SIGTERM, escalate to SIGKILL

The canonical "kill it nicely first" recipe. Useful in init scripts and service controllers.

kill -TERM 1234
for i in {1..10}; do
  kill -0 1234 2>/dev/null || break    # process gone — done
  sleep 1
done
kill -0 1234 2>/dev/null && kill -KILL 1234

Output: (none — exits 0 on success)

kill -0 PID doesn't send a signal — it just probes whether the PID exists.

Find every process matching a pattern

pgrep previews; pkill kills. -f matches the full command line, not just the binary name.

pgrep -af "python.*server.py"

Output:

9200 python3 /opt/myapp/server.py --port 8080
9201 python3 /opt/myapp/server.py --worker

# Kill them all gracefully
pkill -TERM -f "python.*server.py"

Output: (none — exits 0 if at least one match was signalled)

Reload a service without restarting

For long-lived daemons that handle SIGHUP. The PID file gives you the right target on a multi-instance box.

sudo kill -HUP "$(cat /run/nginx.pid)"

Output: (none — exits 0 on success)

Or via systemd, which already knows the main PID:

sudo systemctl reload nginx

Output: (none — exits 0 on success)

Pause a runaway process for forensic capture

SIGSTOP freezes the process without killing it — ideal for grabbing memory, FDs, or a stack trace before deciding what to do.

sudo kill -STOP 1234
sudo gcore 1234              # capture a core file (gdb-tools)
sudo ls -l /proc/1234/fd/    # peek at open files
sudo cat /proc/1234/status   # full state snapshot
sudo kill -CONT 1234         # resume
# or kill -TERM 1234         # terminate after autopsy

Output:

Saved corefile core.1234
total 0
lrwx------ 1 alice alice 64 May 25 10:14 0 -> /dev/pts/2
lrwx------ 1 alice alice 64 May 25 10:14 1 -> /dev/pts/2
lr-x------ 1 alice alice 64 May 25 10:14 3 -> /home/alice/data.log
Name:   myproc
State:  T (stopped)
Pid:    1234
PPid:   8820

Find what spawned a child

Walk up the PPID chain to discover whose code is launching surprise processes.

pid=9200
while [[ "$pid" != "0" ]]; do
  read -r ppid cmd < <(ps -o ppid=,comm= -p "$pid")
  echo "$pid -> $cmd (parent $ppid)"
  pid="$ppid"
done

Output:

9200 -> node (parent 8821)
8821 -> bash (parent 8820)
8820 -> sshd (parent 800)
800 -> sshd (parent 1)
1 -> systemd (parent 0)
0 ->  (parent )

Limit a build to 4 cores and 2 GB

Sandbox a CPU-/memory-heavy command without touching unit files.

systemd-run --user --scope -p CPUQuota=400% -p MemoryMax=2G -- \
  make -j 8 all

Output:

Running scope as unit: run-r91a2.scope

CPUQuota=400% is "4 CPUs of bandwidth"; the kernel throttles at the cgroup boundary.

Spot the top memory consumers in a cgroup

For diagnosing "why did my container hit OOM?". systemd-cgtop is top per-cgroup.

systemd-cgtop --order=memory --depth=3 -n 1 | head

Output:

Control Group                                 Tasks   %CPU   Memory  Input/s Output/s
/                                                458    1.4    7.1G       -       -
user.slice                                       182    0.3    3.4G       -       -
user.slice/user-1000.slice                       182    0.3    3.4G       -       -
system.slice                                     142    0.8    2.4G       -       -
system.slice/myapp.service                         7    0.4  412.0M       -       -

Snapshot the entire process tree

For incident reports and post-mortems — paste this whole block.

{
  echo '=== uptime ==='     ; uptime
  echo '=== pstree ==='     ; pstree -aTp
  echo '=== top 20 RSS ===' ; ps -eo pid,ppid,user,rss,vsz,stat,start,time,comm --sort=-rss | head -21
  echo '=== top 20 CPU ===' ; ps -eo pid,ppid,user,pcpu,stat,start,time,comm --sort=-pcpu | head -21
  echo '=== zombies ===' ; ps -eo pid,ppid,state,comm | awk 'NR==1 || $3=="Z"'
} | tee /tmp/process-snapshot-$(date +%Y%m%d-%H%M).txt

Output: (none — exits 0 on success)

Protect a critical process from the OOM killer

oom_score_adj ranges from -1000 (never kill) to 1000 (kill me first). Default is 0.

echo -1000 | sudo tee /proc/1234/oom_score_adj
cat /proc/1234/oom_score

Output:

-1000
0

systemd's equivalent in a unit file is OOMScoreAdjust=-1000.

Detach a long-running task from the current shell

Three ways, in order of robustness.

# 1. nohup + & — POSIX, works everywhere
nohup ./job.sh > job.log 2>&1 &
disown

# 2. setsid — creates a new session, cleanly detached
setsid -f ./job.sh > job.log 2>&1 < /dev/null

# 3. systemd-run --user --scope — full cgroup isolation
systemd-run --user --unit=myjob ./job.sh
journalctl --user -u myjob -f

Output: (none — exits 0 on success)

Tips

htop and btop are interactive replacements for top that show the tree, signals, and cgroup hierarchy without typing flags. Both come with key bindings to send signals to the highlighted PID.

When debugging "what is this process doing right now?", reach for strace -fp PID (syscall trace), pstack PID (stack snapshot), and lsof -p PID (open FDs) in that order. The combination usually reveals where the process is stuck within seconds.

[!WARN] Never kill PID 1. On systemd systems it cannot be killed anyway (SIGKILL is masked for init), but if your container uses a plain shell as PID 1 and you kill it, the whole container exits.