Bash Redirection & Pipes

Standard streams

Redirection operators

# Redirect stdout to file (overwrite)
command > file.txt

# Redirect stdout (append)
command >> file.txt

# Redirect stderr
command 2> error.log

# Redirect both stdout and stderr to same file
command &> all.log
command > all.log 2>&1   # older form, POSIX portable

# Discard output
command > /dev/null 2>&1

# Redirect stdin from file
command < input.txt

# Here-doc (multiline stdin)
cat <<EOF
line one
line two
EOF

# Here-string (single line stdin)
base64 <<< "hello world"

Output (cat <<EOF … EOF):

line one
line two

Output (base64 <<< "hello world"):

aGVsbG8gd29ybGQK

Pipes

# Basic pipe
ps aux | grep nginx

# Pipe stderr through pipe (bash 4+)
command 2>&1 | grep ERROR

# Pipe with tee (write to file AND stdout)
make 2>&1 | tee build.log

# Process substitution (pipe without subshell for IDs)
diff <(sort file1.txt) <(sort file2.txt)

# Named pipe (FIFO)
mkfifo /tmp/mypipe
tail -f /var/log/syslog > /tmp/mypipe &
grep "ERROR" < /tmp/mypipe

Output (ps aux | grep nginx):

www-data  1234  0.0  0.1  55680  2048 ?  S  09:01  0:00 nginx: worker process
www-data  1235  0.0  0.1  55680  2048 ?  S  09:01  0:00 nginx: worker process

Output (diff <(sort file1.txt) <(sort file2.txt)):

3c3
< banana
---
> blueberry
5a6
> mango

Output (make 2>&1 | tee build.log):

gcc -o main main.c utils.c
Linking...
Build complete.

Output is written to the terminal in real time AND saved to build.log simultaneously.

Output (tail -f /var/log/auth.log | grep --color=always "Failed"):

Apr 26 10:33:01 server sshd[4821]: Failed password for invalid user admin from 203.0.113.42 port 51234 ssh2
Apr 26 10:33:08 server sshd[4822]: Failed password for root from 198.51.100.7 port 60412 ssh2

Useful patterns

Capture stderr into a variable (discard stdout):

err=$(command 2>&1 >/dev/null)

Run command, capture all output, and check exit code:

if ! output=$(some-command 2>&1); then
  echo "Failed: $output" >&2
  exit 1
fi

Swap stdout and stderr:

command 3>&1 1>&2 2>&3 3>&-

Tail a live log with colour preserved through the pipe:

tail -f /var/log/auth.log | grep --color=always "Failed"

File descriptors in depth

Every Unix process inherits three open file descriptors from its parent: 0 (stdin), 1 (stdout), 2 (stderr). The kernel doesn't treat them specially — they're just the first three slots in the process's file-descriptor table, and you can open more (3, 4, 5, …) whenever you need to. Redirection operators in bash are syntactic sugar over the dup2(2) system call: they wire one descriptor to point at the same file as another descriptor or to a file on disk.

# Show this shell's open descriptors
ls -l /proc/$$/fd

# Open a new descriptor 3 for reading from a file
exec 3< /etc/hostname
read -r hostname <&3
exec 3<&-             # close fd 3
echo "$hostname"

# Open fd 4 for writing to a log
exec 4> /tmp/script.log
echo "starting" >&4
date >&4
exec 4>&-             # close

Output (ls -l /proc/$$/fd):

lrwx------ 1 alice alice 64 May 24 10:00 0 -> /dev/pts/3
lrwx------ 1 alice alice 64 May 24 10:00 1 -> /dev/pts/3
lrwx------ 1 alice alice 64 May 24 10:00 2 -> /dev/pts/3
lrwx------ 1 alice alice 64 May 24 10:00 255 -> /dev/pts/3

All three standard descriptors point at the same pty (/dev/pts/3) — that's why typing in the terminal shows the same place where output appears.

Higher descriptors (3+) survive sub-shell creation and process-substitution boundaries. They're the right tool for shell scripts that need a persistent log channel without colliding with subcommands that write to stdout/stderr.

Closing a descriptor

>&- and <&- close an open descriptor. Closing inherited fds is occasionally important — when you spawn a long-running child you don't want it holding open a pipe to your script.

# Close stdin in the child
some-daemon <&-

# Close fd 3 after using it
exec 3>&-

>, >>, and >| — overwrite, append, noclobber

> opens the target file with O_WRONLY | O_CREAT | O_TRUNC — it creates the file if it doesn't exist and truncates it to zero length if it does. >> uses O_APPEND instead, so writes go to the end. >| is the same as > but bypasses the noclobber shell option (set -o noclobber), which otherwise forbids overwriting an existing file.

# Default: overwrite
echo "hello" > /tmp/greeting.txt

# Append
echo "world" >> /tmp/greeting.txt

# Enable noclobber to prevent accidental overwrites
set -o noclobber
echo "oops" > /tmp/greeting.txt          # error: cannot overwrite
echo "force" >| /tmp/greeting.txt        # explicit override
set +o noclobber

Output (with noclobber):

bash: /tmp/greeting.txt: cannot overwrite existing file

set -o noclobber is a cheap script-safety win: it turns an accidental > output.txt into an error instead of silently losing previous data. Pair with >| for the rare case where overwriting is intentional.

2>&1 — order matters

2>&1 makes file descriptor 2 (stderr) point at whatever fd 1 (stdout) currently points at. Redirections are processed left-to-right, so the order you write them in changes the result. This is the single most surprising thing about bash redirection.

# Send BOTH stdout and stderr to all.log
command > all.log 2>&1
#         ↑ first: fd 1 → all.log
#                  ↑ second: fd 2 → wherever fd 1 points = all.log
# Result: both go to all.log. ✓

# Same operators, different order — does NOT do the same thing
command 2>&1 > all.log
#       ↑ first: fd 2 → wherever fd 1 points (still the terminal)
#            ↑ second: fd 1 → all.log
# Result: stdout goes to all.log, stderr still goes to the terminal. ✗

The mental model: think of > and 2>&1 not as "merge streams" but as "set this fd to point at the same place as that one right now".

&> and |& — the bash shortcuts

bash 4 adds &> as a one-token shorthand for > ... 2>&1 and |& as shorthand for 2>&1 |. They're not POSIX, so don't use them in /bin/sh scripts; they're fine in any #!/usr/bin/env bash script.

# Both stdout and stderr to a file
command &> all.log

# Both stdout and stderr through a pipe
command |& grep ERROR

# Append both to a file (bash 4+)
command &>> all.log

# POSIX equivalents
command > all.log 2>&1
command 2>&1 | grep ERROR
command >> all.log 2>&1

Process substitution

<(...) and >(...) make a command's stdout (or stdin) look like a regular file path that other commands can open. bash implements them with /dev/fd/<n> (or named FIFOs on systems without /dev/fd). This lets you feed multiple command outputs to a tool that takes file arguments — diff, comm, paste, and anything else that won't read from stdin.

# Diff two sorted streams without temp files
diff <(sort file1.txt) <(sort file2.txt)

# Compare command outputs across hosts
diff <(ssh host1 'systemctl list-units --no-pager') \
     <(ssh host2 'systemctl list-units --no-pager')

# Three-way input: paste matched lines from three streams
paste <(cut -d, -f1 names.csv) <(cut -d, -f2 names.csv) <(cut -d, -f3 names.csv)

# Output substitution: send a command's input to multiple consumers via tee
ls -la | tee >(grep '.txt$' > text.list) >(grep '.log$' > log.list) >/dev/null

# Inspect what bash actually creates
echo <(true)
# /dev/fd/63

Output (echo <(true)):

/dev/fd/63

That /dev/fd/63 is the descriptor bash opened on the pipe between the shell and the substituted command. Tools that fopen() it get exactly the same byte stream the substituted command writes.

[!WARN] Process substitution does not propagate exit status. diff <(failing-cmd) <(other-cmd) returns the exit code of diff, not of failing-cmd. If you need the inner command's status, capture it via a temp file or coproc instead.

Here-docs

A here-document feeds a literal block of text into a command's stdin, ending at a sentinel line. The most common use is templating config files or scripts; the variants control quoting, indentation, and where the body comes from.

# Standard here-doc — variables ARE expanded
cat <<EOF
Hostname: $(hostname)
User: $USER
Date: $(date)
EOF

# Quoted delimiter — variables and command substitution are NOT expanded
# (use this for embedded shell snippets, JSON templates, awk programs, etc.)
cat <<'EOF'
Literal $USER and $(date) — no expansion.
This script doesn't try to evaluate anything inside.
EOF

# Leading-tab stripping with <<- so the heredoc can be indented in source
if true; then
    cat <<-EOF
	    one
	    two
	    three
	EOF
fi

# Redirect the heredoc output to a file
cat > /tmp/config.toml <<EOF
[server]
host = "$(hostname)"
port = 8080
EOF

Output (first heredoc, cat <<EOF):

Hostname: myhost
User: alicedev
Date: Sat May 24 10:00:00 EDT 2026

Use <<'EOF' (quoted) whenever the body contains shell metacharacters you do not want bash to interpret — for example, when embedding a Python or awk script. Reserve unquoted <<EOF for templates that genuinely need variable expansion.

<<< here-strings

A here-string feeds a single line of text on stdin without spawning echo. It's slightly faster than echo "x" | cmd and avoids the subshell that | creates, which lets you use read to capture the result into the current shell.

# base64-encode a literal string
base64 <<< "hello world"

# Feed a value into read without losing it in a subshell
read -r year month day <<< "2026 05 24"
echo "$year-$month-$day"

# Quick JSON pretty-print of a string
jq . <<< "$json_payload"

Output (base64 <<< "hello world"):

aGVsbG8gd29ybGQK

Named pipes (FIFOs)

mkfifo creates a special file that one process writes to and another reads from — like an anonymous pipe but persistent on disk, so the two processes don't need a common parent. Useful for connecting backgrounded jobs, building tee-like fan-out across machines, and giving one process the ability to send commands to another.

# Create a FIFO
mkfifo /tmp/mypipe

# Producer in the background
tail -f /var/log/auth.log > /tmp/mypipe &

# Consumer in the foreground
grep "Failed" < /tmp/mypipe

# Many-writer, one-reader pattern
mkfifo /tmp/cmd
while read -r line < /tmp/cmd; do
    echo "received: $line"
done &
echo "hello" > /tmp/cmd
echo "world" > /tmp/cmd

# Clean up
rm /tmp/mypipe /tmp/cmd

Output (consumer):

received: hello
received: world

A FIFO blocks until both ends are open: opening for reading blocks until someone opens for writing, and vice versa. For producer-consumer patterns where you don't want that, use O_NONBLOCK with exec 3<> /tmp/pipe (open for read+write yourself).

[!WARN] FIFOs persist after the processes exit. Always rm them when you're done — leftover FIFOs in shared directories can hang other scripts that happen to find them.

tee and friends

tee reads stdin, writes to stdout, and also writes to one or more files — the T-pipe of Unix. It's how you save a command's output while also watching it scroll past in real time.

# Save output AND see it live
make 2>&1 | tee build.log

# Append instead of overwrite
make 2>&1 | tee -a build.log

# Tee to multiple files at once
some-cmd | tee out1.log out2.log

# Run a downstream command on the SAME stream that's being saved
some-cmd | tee build.log | grep -i error

# Write to a file that requires root, without making the whole pipeline root
echo "127.0.0.1 myhost.local" | sudo tee -a /etc/hosts

# Discard tee's stdout (useful for "fan out to N files only")
some-cmd | tee out1.log out2.log > /dev/null

Output (make 2>&1 | tee build.log):

gcc -o main main.c utils.c
Linking...
Build complete.

echo X | sudo tee -a /etc/file is the canonical idiom for appending to a root-owned file from a non-root shell — it avoids bash: /etc/file: Permission denied because the redirection runs in the shell process before sudo takes effect.

exec — global redirection inside a script

exec followed by redirections (and no command) modifies the current shell's open descriptors permanently — every subsequent command in the script inherits them. This is how you say "send everything from here on to /tmp/script.log".

#!/usr/bin/env bash
# /home/alice/bin/job.sh — redirect ALL output from this point forward

# Redirect stdout to a log file (everything after this goes to the log)
exec >> /home/alice/log/job.log

# Redirect stderr to the same place
exec 2>&1

echo "[$(date '+%F %T')] starting"
do-work
echo "[$(date '+%F %T')] done"

# Variant: ALSO see output on the terminal via tee + process substitution
exec > >(tee -a /home/alice/log/job.log)
exec 2>&1
echo "this appears in both the terminal and the log"

# Open a dedicated logging fd that's separate from stdout/stderr
exec 4>> /home/alice/log/script-events.log
echo "starting" >&4
some-cmd                       # stdout/stderr unaffected
echo "finished" >&4

The first form (single redirect) is the cron-friendly idiom — set up logging at the top of the script and forget about it. The tee form gives you the live terminal output as well, useful for interactive runs.

coproc — bidirectional pipes

A coprocess is a background process whose stdin and stdout are connected to two file descriptors in the parent shell. Reach for coproc when you need to interact with a long-running child (send a query, read the response, send another query) — a one-way pipe can't do that.

# Launch bc as a coprocess
coproc CALC { bc -l; }

# Send a query and read the answer
echo "scale=4; sqrt(2)" >&"${CALC[1]}"
read -r answer <&"${CALC[0]}"
echo "sqrt(2) = $answer"

# Send another query — same process, no restart cost
echo "2^10" >&"${CALC[1]}"
read -r answer <&"${CALC[0]}"
echo "2^10 = $answer"

# Close down
exec {CALC[1]}>&-
wait "$CALC_PID"

Output:

sqrt(2) = 1.4142
2^10 = 1024

coproc is overkill for one-shot command output (use var=$(cmd)), but it's the right tool for "spawn a REPL once, ask many questions". Common targets: bc -l, sqlite3, an SSH master, a long-lived python3 -u.

set -o pipefail and the exit-status trap

By default, a pipeline's exit status is the status of the last command only. That means failing-cmd | tee log exits 0 even when failing-cmd failed, because tee succeeded. set -o pipefail changes the rule: the pipeline exits with the status of the rightmost non-zero command (or 0 if all succeeded). Combine with set -e and set -u for the standard "strict mode" script preamble.

#!/usr/bin/env bash
set -euo pipefail        # exit on error, error on unset var, pipefail

# Without pipefail, this would exit 0 even though `false` returned 1:
false | tee /tmp/out

# With pipefail set, the pipeline exits 1 and `set -e` kills the script.

Inspect ${PIPESTATUS[@]} for the status of every command in the most recent pipeline — useful when you want fine-grained reporting beyond pass/fail.

false | true | true
echo "${PIPESTATUS[@]}"
# 1 0 0

set -euo pipefail is the single best preamble for shell scripts. It turns silent failures into loud ones and makes debugging dramatically easier. Combine with trap 'echo "error on line $LINENO" >&2' ERR for line-number reporting.

Common pitfalls

1. cmd > file 2>&1 vs cmd 2>&1 > file — the first works; the second does not. Order is left-to-right.
2. echo "$x" | read y — the right-hand side runs in a subshell, so $y is unset back in the parent. Use read y <<< "$x" (here-string) or process substitution instead.
3. tee losing exit status — without pipefail, failing | tee log reports success. Always set -o pipefail in production scripts, or check ${PIPESTATUS[0]}.
4. Unquoted heredoc expanding $variables unintentionally — embedding a Python or awk program in <<EOF will mangle $1/$NF/$VAR. Use <<'EOF' for verbatim bodies.
5. <<-EOF indentation requires tabs — leading spaces are NOT stripped; only tab characters are. Editors that auto-convert tabs to spaces will silently break the heredoc body.
6. Process substitution exit codes ignored — diff <(may-fail) <(other) returns diff's status; the inner command's failure is invisible. Capture via a temp file when you need it.
7. /dev/null 2>&1 order — > /dev/null 2>&1 discards both; 2>&1 > /dev/null discards stdout only and leaves stderr on the terminal.
8. Forgetting to close descriptors — exec 3> log without a later exec 3>&- means descriptor 3 leaks into every child process you spawn. Usually harmless but occasionally surprising.
9. FIFOs blocking unexpectedly — opening a FIFO for reading blocks until something opens it for writing (and vice versa). tail -f log > fifo & is the right pattern; cat fifo alone hangs forever.
10. >> file vs tee -a file — when running under sudo, only the latter works for a root-owned file. The >> redirection happens in the unprivileged shell process.

Real-world recipes

Log a script's full output to file + terminal

The standard "run interactively but also save the transcript" pattern. Everything (stdout and stderr) is captured to the log while still visible on the terminal.

#!/usr/bin/env bash
set -euo pipefail

LOG=/home/alice/log/install.log
exec > >(tee -a "$LOG") 2>&1

echo "[$(date '+%F %T')] install starting on $(hostname)"
sudo apt-get update
sudo apt-get install -y nginx postgresql
echo "[$(date '+%F %T')] install finished"

The tee -a keeps the log when the script is re-run; the process substitution lets exec redirect the parent's stdout to tee's stdin transparently.

Capture stdout and stderr to separate files

For build tools that produce useful warnings on stderr but mountains of progress on stdout, you may want them in separate files for triage.

build-tool > build.out 2> build.err

# With pipefail-aware exit reporting
{ build-tool 2>&3 | tee build.out >/dev/null; } 3>&1 1>&2 | tee build.err >/dev/null

The second form is the canonical "tee both streams without merging" idiom. Read it right-to-left: fd 3 is opened to point at the original stdout, fd 2 of build-tool is redirected to 3 (so stderr now goes where the outer pipe is), and the outer pipeline tees the visible stderr while the inner tees stdout.

Capture stderr into a variable, discard stdout

For checking error messages from a command whose stdout output is irrelevant. The swap pattern (3>&1 1>&2 2>&3 3>&-) is the magic spell.

err=$(some-cmd 2>&1 >/dev/null)

# OR — keep stdout in $out and stderr in $err
{
    IFS= read -rd '' out
    IFS= read -rd '' err
} < <({ some-cmd 2> >(printf '%s\0' "$(cat)" >&2) ; printf '\0'; } 2>&1)

The simpler err=$(cmd 2>&1 >/dev/null) works for most cases; the second form is when you need both streams independently in the parent shell.

Diff two command outputs

The classic process-substitution use case. Especially handy when you can't (or don't want to) write the outputs to disk first.

# Compare package lists between two hosts
diff <(ssh host1 dpkg -l | awk '{print $2}' | sort) \
     <(ssh host2 dpkg -l | awk '{print $2}' | sort)

# Spot what changed in a sysctl after a tuning patch
diff <(sysctl -a 2>/dev/null | sort) /tmp/sysctl.before

Fan-out a stream to multiple consumers

When one upstream produces a stream that several downstream tools need to consume in parallel.

# Save raw + run two analyses, all from one pcap capture
tcpdump -i eth0 -U -w - 2>/dev/null \
  | tee >(tcpdump -r - 'tcp port 80' > http.pcap) \
        >(tcpdump -r - 'udp port 53' > dns.pcap) \
        > /tmp/full.pcap

tee writes the stream to each >(...) consumer; bash sets each one up as a sub-process reading on a /dev/fd/<n>.

Atomic file write with a temp file + rename

Don't redirect directly to the destination — write to a temporary, then mv into place. This way a partial write never appears as the target file.

tmp=$(mktemp /tmp/config.XXXXXX)
trap 'rm -f "$tmp"' EXIT
generate-config > "$tmp"
mv "$tmp" /etc/myapp/config.toml

Append to a root-owned file from a normal shell

sudo echo >> /etc/file does NOT do what you want — >> runs as your user before sudo is involved. The fix is sudo tee -a.

echo "127.0.0.1 myhost.local" | sudo tee -a /etc/hosts > /dev/null

The trailing > /dev/null keeps tee from echoing the line back to your terminal.

Build a one-liner that returns the right exit code

When piping through tee (e.g. for a CI build), make sure a failing build actually fails the pipeline.

set -o pipefail
make 2>&1 | tee build.log
# Pipeline now exits with make's status, not tee's.

When in doubt about a redirection, prepend set -x (or run bash -x script.sh) — bash prints every expanded command and its redirections to stderr, which makes order-of-evaluation bugs immediately obvious. Pair with shellcheck for static analysis before running.

[!WARN] > file is destructive and instantaneous. There is no undo. Pair set -o noclobber with >| for explicit overwrites in interactive shells, and write to temp files plus mv in scripts.