cheat sheet
journalctl
Query and follow systemd's structured journal. Covers unit filters, time ranges, priority levels, boot logs, namespaces, invocations, output formats, persistence, configuration, and disk-vacuum.
journalctl — systemd Journal Query
What it is
journalctl is the query interface for the systemd journal — a binary, indexed, structured log store that captures every line of stdout/stderr from systemd-managed services, plus kernel ring-buffer messages, audit records, and any program that writes to syslog(3). It's installed by default on every systemd-based distribution (Debian, Ubuntu, Fedora, Arch, RHEL, openSUSE, and most cloud images). Reach for journalctl whenever you'd otherwise tail -f /var/log/syslog or cat /var/log/nginx/error.log; for plain-text log files outside systemd's control, grep and tail remain the right tools.
Install
journalctl ships with systemd, so it's already installed on every systemd distribution — there's nothing to add. The only common decision is whether to enable on-disk persistence so logs survive a reboot.
# Verify it's present
journalctl --version
# Enable persistent storage (default on most distros, but check)
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
Output:
systemd 258 (258-1ubuntu1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK ...
Several flags below were introduced in recent systemd releases:
-i/--file=shortcut,-T/--exclude-identifier=,--list-namespaces(v256);--list-invocations/-I(v257);--synchronize-on-exit=(v258). Checkjournalctl --versionif a flag isn't recognized.
Syntax
journalctl takes any combination of unit filters, time ranges, priority filters, and field matches; with no arguments it pages the entire journal newest-last. Multiple filters of the same kind are OR-combined; different kinds are AND-combined.
journalctl [OPTIONS] [MATCHES...]
journalctl -u UNIT
journalctl FIELD=VALUE [FIELD=VALUE ...]
journalctl --since "TIMESPEC" --until "TIMESPEC"
Output: (none — exits 0 on success)
Essential options
| Option | Meaning |
|---|---|
-u UNIT | Show logs only for the given systemd unit |
-f, --follow | Stream new entries as they arrive (like tail -f) |
-n N | Show the last N entries (default 10 with -n, or all) |
-r, --reverse | Newest first |
--since TS | Entries on or after timestamp TS |
--until TS | Entries on or before timestamp TS |
-p PRIO | Filter by priority (emerg…debug or 0…7) |
-b [N] | Logs for boot N (-b = current, -b -1 = previous, -b 0 = current) |
-k, --dmesg | Kernel ring-buffer messages only |
-g PATTERN | Grep messages with PCRE2 regex; pair with --case-sensitive=yes|no |
-T, --exclude-identifier=ID | Exclude entries with the given SYSLOG_IDENTIFIER (v256+) |
-i FILE, --file=FILE | Read entries from a specific .journal file (v256 added -i shortcut) |
-o FMT | Output format (short, cat, json, json-pretty, json-seq, verbose, …) |
--no-pager | Don't pipe through less |
--no-hostname | Hide the hostname column |
-S, -U | Short aliases for --since / --until |
--list-boots | Show indexed list of recorded boots |
--namespace=NS | Query a journal namespace (* = all, +NS = NS plus default) |
--list-namespaces | List all configured journal namespaces (v256+) |
--list-invocations / -I IDX | List unit invocations, or pick a specific one (v257+) |
--root=DIR / --image=PATH | Read journals from an alternate root tree or a disk image (v247+) |
--synchronize-on-exit=yes | With --follow, flush pending entries before exiting on SIGINT (v258+) |
--disk-usage | Print total size of journal on disk |
--vacuum-size / --vacuum-time / --vacuum-files | Prune the journal |
--verify | Verify journal file integrity |
Filtering by unit
-u accepts a unit name (with or without the .service suffix) or a glob, and is the single most-used flag — almost every troubleshooting session begins by narrowing the entire journal to one service. You can pass -u multiple times to OR several units together.
journalctl -u nginx # all nginx logs
journalctl -u nginx.service # same (explicit suffix)
journalctl -u 'nginx*' # any unit matching the glob
journalctl -u nginx -u php-fpm # both, merged by timestamp
journalctl --user -u syncthing # user-level units
Output (journalctl -u nginx -n 5):
May 24 10:14:01 myhost systemd[1]: Started nginx.service - A high performance web server.
May 24 10:14:01 myhost nginx[1234]: nginx: [warn] conflicting server name "myhost" on 0.0.0.0:80
May 24 10:14:02 myhost nginx[1234]: 10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET / HTTP/1.1" 200 1024
May 24 10:14:02 myhost nginx[1234]: 10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET /style.css HTTP/1.1" 200 4096
May 24 10:14:03 myhost nginx[1234]: 10.0.0.5 - alicedev [24/May/2026:10:14:03 +0000] "GET /favicon.ico HTTP/1.1" 404 162
Following logs in real time
-f streams new entries as they're written, the systemd equivalent of tail -f. Combine with -u and -n to first dump the last N lines, then keep streaming.
journalctl -u nginx -f # live tail
journalctl -u nginx -n 50 -f # last 50 lines, then follow
journalctl -f -p err # live tail of all errors system-wide
journalctl -f _SYSTEMD_UNIT=ssh.service
Output:
May 24 10:15:33 myhost sshd[8800]: Accepted publickey for alicedev from 10.0.0.5 port 52314 ssh2: RSA SHA256:abc...
May 24 10:15:33 myhost sshd[8800]: pam_unix(sshd:session): session opened for user alicedev by (uid=0)
^C
Time-range filters
--since and --until accept anything systemd.time(7) understands: absolute timestamps ("2026-05-24 09:00"), human phrases ("1 hour ago", "yesterday", "today"), and special words (now). Quote any value that contains spaces.
journalctl --since "1 hour ago"
journalctl --since today --until "10 minutes ago"
journalctl --since "2026-05-24 09:00" --until "2026-05-24 10:00"
journalctl --since yesterday --until today
journalctl -u nginx -S "30 min ago" -U now
Output (journalctl --since "5 min ago" -p err):
May 24 10:11:22 myhost kernel: usb 1-2: USB disconnect, device number 7
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] connect() failed (111: Connection refused)
May 24 10:13:55 myhost myapp[4521]: ERROR Database pool exhausted (size=20, waiting=3)
Priority filtering
-p filters by syslog priority — useful for cutting through the noise of info chatter when you only care about warnings or errors. Levels are inclusive upward: -p warning returns warning plus everything more severe (err, crit, alert, emerg).
| Level | Number | Meaning |
|---|---|---|
emerg | 0 | System is unusable |
alert | 1 | Action must be taken immediately |
crit | 2 | Critical conditions |
err | 3 | Error conditions |
warning | 4 | Warning conditions |
notice | 5 | Normal but significant |
info | 6 | Informational |
debug | 7 | Debug-level messages |
journalctl -p err # err + crit + alert + emerg
journalctl -p warning..err # range (warning, err only)
journalctl -p 3 # same as -p err
journalctl -u nginx -p err -S "1h ago"
Output (journalctl -p err -S "1h ago"):
May 24 09:32:11 myhost myapp[4521]: ERROR Failed to connect to redis: timeout
May 24 09:48:09 myhost kernel: I/O error, dev sda, sector 12345
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] connect() failed (111: Connection refused)
Boot logs (-b)
The journal indexes every boot, so -b gives you "just this boot" without timestamps. Negative offsets walk backwards through boot history; --list-boots shows the indexed list.
journalctl -b # current boot only
journalctl -b -1 # previous boot
journalctl -b -2 -p err # errors from two boots ago
journalctl --list-boots # list all stored boots
journalctl -k -b # kernel messages this boot
journalctl -b -1 -u nginx # nginx logs from previous boot
Output (journalctl --list-boots):
IDX BOOT ID FIRST ENTRY LAST ENTRY
-3 a1b2c3d4e5f6... Mon 2026-05-20 08:01:14 UTC Mon 2026-05-20 22:14:55 UTC
-2 b2c3d4e5f6a7... Tue 2026-05-21 07:55:01 UTC Tue 2026-05-21 23:02:11 UTC
-1 c3d4e5f6a7b8... Wed 2026-05-22 08:11:30 UTC Wed 2026-05-22 21:48:09 UTC
0 d4e5f6a7b8c9... Thu 2026-05-23 08:00:14 UTC Sat 2026-05-24 10:14:33 UTC
Field matches and structured data
The journal stores structured fields alongside every entry, not just a free-text line. You can filter on any field with FIELD=VALUE, and combine multiple matches: same field = OR, different fields = AND. Run journalctl -o verbose to see every field on a sample record.
| Field | Meaning |
|---|---|
_PID | Process ID that emitted the record |
_UID / _GID | User / group ID |
_COMM | Executable basename (nginx, sshd) |
_EXE | Full path to the binary |
_SYSTEMD_UNIT | Owning systemd unit |
_BOOT_ID | Boot the entry came from |
_HOSTNAME | Hostname |
PRIORITY | Syslog priority (0–7) |
SYSLOG_IDENTIFIER | Tag (often the program name) |
MESSAGE | The free-text message |
journalctl _PID=1234 # one process
journalctl _COMM=sshd # any sshd invocation
journalctl _SYSTEMD_UNIT=nginx.service _PID=1234 # both must match
journalctl _COMM=sshd _COMM=login # either sshd OR login
journalctl SYSLOG_IDENTIFIER=cron --since today
journalctl _UID=1000 -S yesterday # everything user 1000 did
Output (journalctl _COMM=sshd -n 3):
May 24 10:15:33 myhost sshd[8800]: Accepted publickey for alicedev from 10.0.0.5 port 52314 ssh2
May 24 10:15:33 myhost sshd[8800]: pam_unix(sshd:session): session opened for user alicedev
May 24 10:18:12 myhost sshd[8800]: pam_unix(sshd:session): session closed for user alicedev
Pattern matching with -g (PCRE2)
-g filters by a PCRE2 regular expression against the MESSAGE field — much faster than piping into grep because the match runs inside the journal reader before formatting. Pair with --case-sensitive=no for case-insensitive search; the default is smart-case (case-insensitive when the pattern is all-lowercase).
journalctl -g 'timeout|refused|denied' # any of three words
journalctl -u nginx -g '50[0-9]' # 5xx responses only
journalctl -g 'segfault' --case-sensitive=no
journalctl -u myapp -g 'request_id=abc[0-9]+' -S "1h ago"
Output (journalctl -u nginx -g '50[0-9]' -n 3):
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] 502 upstream prematurely closed
May 24 10:13:11 myhost nginx[1234]: 10.0.0.5 - alicedev "GET /api HTTP/1.1" 503 92
May 24 10:14:02 myhost nginx[1234]: 10.0.0.5 - alicedev "GET /healthz HTTP/1.1" 504 0
Excluding identifiers with -T (v256+)
-T / --exclude-identifier= drops entries whose SYSLOG_IDENTIFIER matches — the inverse of -t/--identifier=. Use it to mute one chatty source while still seeing everything else. Pass -T multiple times to exclude several identifiers.
journalctl -T sudo # everything except sudo
journalctl -p err -T kernel # all errors but kernel ones
journalctl -T cron -T systemd -S "1h ago" # mute two noisy sources
Output: (none — exits 0 on success)
Log namespaces
A journal namespace is an independent log stream with its own storage, retention, and journald instance. Namespaces both isolate noisy services (their data doesn't bloat the default journal) and reduce write contention on shared storage. The default namespace has no identifier; named namespaces are created by symlinking systemd-journald@NAME.service and adding LogNamespace=NAME to a unit.
journalctl --list-namespaces # all configured namespaces (v256+)
journalctl --namespace=myapp # only the myapp namespace
journalctl --namespace='*' # all namespaces, interleaved
journalctl --namespace=+myapp # myapp + default, interleaved
journalctl --namespace=myapp -u worker -f # follow one unit in a namespace
Output (journalctl --list-namespaces):
NAMESPACE ACTIVE STATE PERSISTENT VOLATILE
default yes running 482.3M 16.0M
myapp yes running 128.0M 8.0M
audit yes running 256.0M 4.0M
Unit invocations (v257+)
--list-invocations shows every time a unit was started, ended, and what exit status it produced — without manually correlating boot IDs. -I/--invocation= then jumps to the logs for one specific invocation, identified by either an index (most recent = ^1) or a 128-bit invocation ID.
journalctl -u myapp --list-invocations # table of recent runs
journalctl -u myapp -I ^1 # the most recent invocation
journalctl -u myapp -I ^3 # three invocations ago
journalctl -u myapp -I 9f8e7d6c5b4a... # by invocation ID
Output (journalctl -u myapp --list-invocations -n 3):
IDX INVOCATION ID FIRST ENTRY LAST ENTRY STATE
^3 3a2b1c0d4e5f6a7b8c9d0e1f2a3b4c5d Thu 2026-05-22 14:01:22 UTC Thu 2026-05-22 14:14:08 UTC dead
^2 4b3c2d1e5f6a7b8c9d0e1f2a3b4c5d6e Fri 2026-05-23 09:12:54 UTC Fri 2026-05-23 18:33:01 UTC dead
^1 5c4d3e2f6a7b8c9d0e1f2a3b4c5d6e7f Sat 2026-05-24 08:00:14 UTC - running
Offline journals: --root and --image
When recovering logs from a mounted backup, a chroot, or a disk image (e.g. a VM snapshot, an installer ISO, or a raw block device), point journalctl at the source instead of running it inside the system. --root=DIR operates on a mounted tree; --image=PATH mounts a disk image read-only and reads its /var/log/journal.
journalctl --root=/mnt/recovery -b -1 # last boot of a mounted disk
journalctl --image=/srv/backups/server.img -k # kernel log from a VM image
journalctl --image=/dev/sdb1 -p err -S "1 day ago"
journalctl --image=/var/lib/machines/web.raw --list-boots
Output (journalctl --root=/mnt/recovery --list-boots):
IDX BOOT ID FIRST ENTRY LAST ENTRY
-1 c3d4e5f6a7b8... Wed 2026-05-22 08:11:30 UTC Wed 2026-05-22 21:48:09 UTC
0 d4e5f6a7b8c9... Thu 2026-05-23 08:00:14 UTC Sat 2026-05-24 02:14:33 UTC
Output formats
-o switches between rendering modes. cat strips all metadata for grep-friendly output; json and json-pretty emit one structured object per record for downstream tools like jq; json-seq prefixes each record with ␞ (RS, 0x1E) so streaming parsers can frame messages reliably even when fields contain embedded newlines; verbose shows every field on a record.
journalctl -u nginx -o cat -n 5 # message only, no timestamps
journalctl -u nginx -o json -n 1 # one-line JSON
journalctl -u nginx -o json-pretty -n 1 # multi-line JSON
journalctl -u nginx -o json-seq -n 5 # RFC 7464 record-separator framed
journalctl -u nginx -o verbose -n 1 # all fields
journalctl -u nginx -o short-iso # ISO-8601 timestamps
journalctl -u nginx -o short-iso-precise # ISO-8601 with microseconds
journalctl -u nginx -o short-monotonic # monotonic clock seconds since boot
Output (journalctl -u nginx -o cat -n 3):
Started nginx.service - A high performance web server.
10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET / HTTP/1.1" 200 1024
10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET /style.css HTTP/1.1" 200 4096
Output (journalctl -u nginx -o json-pretty -n 1):
{
"__REALTIME_TIMESTAMP" : "1748081643000123",
"PRIORITY" : "6",
"_PID" : "1234",
"_UID" : "33",
"_COMM" : "nginx",
"_SYSTEMD_UNIT" : "nginx.service",
"SYSLOG_IDENTIFIER" : "nginx",
"MESSAGE" : "10.0.0.5 - alicedev [24/May/2026:10:14:03 +0000] \"GET /favicon.ico HTTP/1.1\" 404 162"
}
Disk usage and vacuum
The journal can grow large on busy hosts. --disk-usage reports its current footprint and --vacuum-* flags prune old data. Pruning never touches the live in-memory ring buffer, only on-disk files.
journalctl --disk-usage # total size on disk
sudo journalctl --vacuum-size=1G # keep at most 1 GiB on disk
sudo journalctl --vacuum-time=7d # delete entries older than 7 days
sudo journalctl --vacuum-files=10 # keep at most 10 journal files
Output:
Archived and active journals take up 482.3M in the file system.
Deleted archived journal /var/log/journal/abcd…/system@001.journal (32.0M).
Deleted archived journal /var/log/journal/abcd…/system@002.journal (32.0M).
Vacuuming done, freed 64.0M of archived journals from /var/log/journal/abcd….
Configuration
systemd-journald reads its configuration from /etc/systemd/journald.conf, then merges any drop-in files under /etc/systemd/journald.conf.d/*.conf (system-admin overrides) and /run/systemd/journald.conf.d/*.conf (runtime/vendor overrides). Drop-ins take precedence in lexical order, so prefer creating a small drop-in over editing the main file — package upgrades will clobber edits to journald.conf but leave *.conf.d/ alone. Each namespace has its own config: /etc/systemd/journald@NAME.conf plus matching journald@NAME.conf.d/*.conf. man journald.conf documents every setting.
# /etc/systemd/journald.conf.d/50-jockey.conf — admin overrides
[Journal]
Storage=persistent # auto | persistent | volatile | none
Compress=yes
SystemMaxUse=2G # cap on /var/log/journal
SystemKeepFree=500M # leave at least 500M free on the filesystem
SystemMaxFileSize=128M # rotate at this size
MaxRetentionSec=30day # delete entries older than this
ForwardToSyslog=no # don't forward to rsyslog
ForwardToSocket=192.0.2.10:5140 # v256+: stream to an AF_INET/AF_VSOCK/AF_UNIX sink
By default many distributions store the journal only in /run/log/journal (RAM), which is wiped on reboot. Creating /var/log/journal plus Storage=persistent moves it to disk. After editing config:
sudo mkdir -p /etc/systemd/journald.conf.d
sudo systemd-analyze cat-config systemd/journald.conf # show effective merged config
sudo systemctl restart systemd-journald
Output (systemd-analyze cat-config systemd/journald.conf):
# /etc/systemd/journald.conf
[Journal]
#Storage=auto
#Compress=yes
...
# /etc/systemd/journald.conf.d/50-jockey.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
ForwardToSocket=192.0.2.10:5140
Per-unit namespaces
Attach a unit to a named journal namespace by adding LogNamespace= to its service file; systemd auto-starts the matching systemd-journald@NAME.service instance, which reads /etc/systemd/journald@NAME.conf.
# /etc/systemd/system/myapp.service.d/10-namespace.conf
[Service]
LogNamespace=myapp
sudo systemctl daemon-reload
sudo systemctl restart myapp
journalctl --namespace=myapp -u myapp -f
Output: (none — exits 0 on success)
Common pitfalls
- "No journal files were found" — persistent storage isn't enabled.
sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald. - Unit name typo —
journalctl -u ngnixsilently returns nothing. Verify withsystemctl list-units 'nginx*'. - Quotes around
--since—--since 1 hour agois three tokens and fails; use--since "1 hour ago". - Trying to grep before
-o cat— by defaultjournalctladds timestamp and hostname columns;grepanchors won't match the message body. Use-o cator filter with_COMM/SYSLOG_IDENTIFIERdirectly. -fexits immediately underless—journalctl -f | grep ERRORworks but loses live tailing colors; for live grep usejournalctl -f -u myapp | grep --line-buffered ERROR.- Permissions — non-root users only see their own user-scope units plus system logs whose
_UIDmatches. Add the user to thesystemd-journal(oradmon Debian) group to read everything. - Clock skew —
--since "1 hour ago"is computed in the host's local time. If your VM is in UTC and your logs span time zones, prefer absolute timestamps.
Real-world recipes
"Show me everything nginx logged in the last 10 minutes"
The most common triage query: one unit, one time window, paged in reverse for newest-first reading.
journalctl -u nginx --since "10 min ago" --no-pager
journalctl -u nginx --since "10 min ago" -r # newest first
journalctl -u nginx --since "10 min ago" -p err # only errors
Output:
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] connect() failed (111: Connection refused)
May 24 10:13:01 myhost nginx[1234]: 2026/05/24 10:13:01 [warn] upstream server temporarily disabled
May 24 10:14:33 myhost nginx[1234]: 10.0.0.5 - alicedev "GET / HTTP/1.1" 200 1024
Boot-time post-mortem
After a crash, walk the previous boot's logs and look for the kernel's last words and any service that failed.
journalctl -b -1 -p err # all errors from last boot
journalctl -b -1 -k # kernel ring buffer last boot
journalctl -b -1 --no-pager | tail -100
systemctl --failed # current state of services
Output (systemctl --failed):
UNIT LOAD ACTIVE SUB DESCRIPTION
* myapp.service loaded failed failed My Application
* postgresql.service loaded failed failed PostgreSQL RDBMS
2 loaded units listed.
Save the last hour of one service to a file
-o cat strips metadata for downstream tools; --no-pager prevents paging when stdout is a pipe inside scripts.
journalctl -u myapp --since "1 hour ago" -o cat --no-pager > /tmp/myapp.log
Output: (none — exits 0 on success)
Stream JSON into jq without losing the tail (v258+)
When you need structured fields for an alert pipeline, -o json plus jq is the cleanest path. Add --synchronize-on-exit=yes so that pressing Ctrl-C asks systemd-journald to flush everything queued before the signal arrived — without this, the last few events can be lost between the kernel buffer and disk.
journalctl -u myapp -f -o json --synchronize-on-exit=yes | \
jq -r 'select(.PRIORITY|tonumber < 4) | "\(.__REALTIME_TIMESTAMP) \(.MESSAGE)"'
Output:
1748081643000123 Database pool exhausted (size=20, waiting=3)
1748081709112301 Failed to connect to redis: timeout
Count errors per service over the last day
The classic "what's noisy?" report — grouped by unit, sorted by count.
journalctl --since "1 day ago" -p err -o json --no-pager | \
jq -r '._SYSTEMD_UNIT // "kernel"' | sort | uniq -c | sort -rn | head
Output:
142 myapp.service
38 nginx.service
12 cron.service
3 kernel
Find what filled the disk this hour
When df shows /var filling up, the journal itself is sometimes the culprit. Cap its size and then identify the chatty service.
journalctl --disk-usage
journalctl --since "1 hour ago" -o json --no-pager | \
jq -r '._SYSTEMD_UNIT // ._COMM' | sort | uniq -c | sort -rn | head
sudo journalctl --vacuum-size=500M
Output:
Archived and active journals take up 1.8G in the file system.
8421 myapp.service
1502 nginx.service
214 sshd.service
Pair
journalctl -u myapp -fwithtmuxin one pane andhtop -p $(pgrep myapp)in another — you get live logs alongside live resource usage without leaving the terminal.
If a service logs structured fields via
sd_journal_send(), those fields show up as first-class filterable keys (e.g.REQUEST_ID=…). Searching by request ID is how you trace one transaction through dozens of services on a single host.
Sources
- journalctl(1) — systemd upstream man page
- journald.conf(5) — systemd upstream man page
- systemd v256 release notes (LWN) —
--list-namespaces,-i/--file=shortcut,-T/--exclude-identifier=,ForwardToSocket= - systemd v257 release notes (LWN) —
--list-invocations,-I/--invocation= - systemd v258 release notes (LWN) —
--synchronize-on-exit=, follow-exit semantics,--setup-keysJSON output - Highlights from systemd v258 — part one (LWN)
- journalctl(1) — Linux manual pages (man7.org)
- systemd/Journal — Arch Wiki