cheat sheet

journalctl

Query and follow systemd's structured journal. Covers unit filters, time ranges, priority levels, boot logs, namespaces, invocations, output formats, persistence, configuration, and disk-vacuum.

journalctl — systemd Journal Query

What it is

journalctl is the query interface for the systemd journal — a binary, indexed, structured log store that captures every line of stdout/stderr from systemd-managed services, plus kernel ring-buffer messages, audit records, and any program that writes to syslog(3). It's installed by default on every systemd-based distribution (Debian, Ubuntu, Fedora, Arch, RHEL, openSUSE, and most cloud images). Reach for journalctl whenever you'd otherwise tail -f /var/log/syslog or cat /var/log/nginx/error.log; for plain-text log files outside systemd's control, grep and tail remain the right tools.

Install

journalctl ships with systemd, so it's already installed on every systemd distribution — there's nothing to add. The only common decision is whether to enable on-disk persistence so logs survive a reboot.

bash
# Verify it's present
journalctl --version

# Enable persistent storage (default on most distros, but check)
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Output:

text
systemd 258 (258-1ubuntu1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK ...

Several flags below were introduced in recent systemd releases: -i/--file= shortcut, -T/--exclude-identifier=, --list-namespaces (v256); --list-invocations/-I (v257); --synchronize-on-exit= (v258). Check journalctl --version if a flag isn't recognized.

Syntax

journalctl takes any combination of unit filters, time ranges, priority filters, and field matches; with no arguments it pages the entire journal newest-last. Multiple filters of the same kind are OR-combined; different kinds are AND-combined.

bash
journalctl [OPTIONS] [MATCHES...]
journalctl -u UNIT
journalctl FIELD=VALUE [FIELD=VALUE ...]
journalctl --since "TIMESPEC" --until "TIMESPEC"

Output: (none — exits 0 on success)

Essential options

OptionMeaning
-u UNITShow logs only for the given systemd unit
-f, --followStream new entries as they arrive (like tail -f)
-n NShow the last N entries (default 10 with -n, or all)
-r, --reverseNewest first
--since TSEntries on or after timestamp TS
--until TSEntries on or before timestamp TS
-p PRIOFilter by priority (emergdebug or 07)
-b [N]Logs for boot N (-b = current, -b -1 = previous, -b 0 = current)
-k, --dmesgKernel ring-buffer messages only
-g PATTERNGrep messages with PCRE2 regex; pair with --case-sensitive=yes|no
-T, --exclude-identifier=IDExclude entries with the given SYSLOG_IDENTIFIER (v256+)
-i FILE, --file=FILERead entries from a specific .journal file (v256 added -i shortcut)
-o FMTOutput format (short, cat, json, json-pretty, json-seq, verbose, …)
--no-pagerDon't pipe through less
--no-hostnameHide the hostname column
-S, -UShort aliases for --since / --until
--list-bootsShow indexed list of recorded boots
--namespace=NSQuery a journal namespace (* = all, +NS = NS plus default)
--list-namespacesList all configured journal namespaces (v256+)
--list-invocations / -I IDXList unit invocations, or pick a specific one (v257+)
--root=DIR / --image=PATHRead journals from an alternate root tree or a disk image (v247+)
--synchronize-on-exit=yesWith --follow, flush pending entries before exiting on SIGINT (v258+)
--disk-usagePrint total size of journal on disk
--vacuum-size / --vacuum-time / --vacuum-filesPrune the journal
--verifyVerify journal file integrity

Filtering by unit

-u accepts a unit name (with or without the .service suffix) or a glob, and is the single most-used flag — almost every troubleshooting session begins by narrowing the entire journal to one service. You can pass -u multiple times to OR several units together.

bash
journalctl -u nginx                  # all nginx logs
journalctl -u nginx.service          # same (explicit suffix)
journalctl -u 'nginx*'               # any unit matching the glob
journalctl -u nginx -u php-fpm       # both, merged by timestamp
journalctl --user -u syncthing       # user-level units

Output (journalctl -u nginx -n 5):

text
May 24 10:14:01 myhost systemd[1]: Started nginx.service - A high performance web server.
May 24 10:14:01 myhost nginx[1234]: nginx: [warn] conflicting server name "myhost" on 0.0.0.0:80
May 24 10:14:02 myhost nginx[1234]: 10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET / HTTP/1.1" 200 1024
May 24 10:14:02 myhost nginx[1234]: 10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET /style.css HTTP/1.1" 200 4096
May 24 10:14:03 myhost nginx[1234]: 10.0.0.5 - alicedev [24/May/2026:10:14:03 +0000] "GET /favicon.ico HTTP/1.1" 404 162

Following logs in real time

-f streams new entries as they're written, the systemd equivalent of tail -f. Combine with -u and -n to first dump the last N lines, then keep streaming.

bash
journalctl -u nginx -f                 # live tail
journalctl -u nginx -n 50 -f           # last 50 lines, then follow
journalctl -f -p err                   # live tail of all errors system-wide
journalctl -f _SYSTEMD_UNIT=ssh.service

Output:

text
May 24 10:15:33 myhost sshd[8800]: Accepted publickey for alicedev from 10.0.0.5 port 52314 ssh2: RSA SHA256:abc...
May 24 10:15:33 myhost sshd[8800]: pam_unix(sshd:session): session opened for user alicedev by (uid=0)
^C

Time-range filters

--since and --until accept anything systemd.time(7) understands: absolute timestamps ("2026-05-24 09:00"), human phrases ("1 hour ago", "yesterday", "today"), and special words (now). Quote any value that contains spaces.

bash
journalctl --since "1 hour ago"
journalctl --since today --until "10 minutes ago"
journalctl --since "2026-05-24 09:00" --until "2026-05-24 10:00"
journalctl --since yesterday --until today
journalctl -u nginx -S "30 min ago" -U now

Output (journalctl --since "5 min ago" -p err):

text
May 24 10:11:22 myhost kernel: usb 1-2: USB disconnect, device number 7
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] connect() failed (111: Connection refused)
May 24 10:13:55 myhost myapp[4521]: ERROR    Database pool exhausted (size=20, waiting=3)

Priority filtering

-p filters by syslog priority — useful for cutting through the noise of info chatter when you only care about warnings or errors. Levels are inclusive upward: -p warning returns warning plus everything more severe (err, crit, alert, emerg).

LevelNumberMeaning
emerg0System is unusable
alert1Action must be taken immediately
crit2Critical conditions
err3Error conditions
warning4Warning conditions
notice5Normal but significant
info6Informational
debug7Debug-level messages
bash
journalctl -p err                    # err + crit + alert + emerg
journalctl -p warning..err           # range (warning, err only)
journalctl -p 3                      # same as -p err
journalctl -u nginx -p err -S "1h ago"

Output (journalctl -p err -S "1h ago"):

text
May 24 09:32:11 myhost myapp[4521]: ERROR    Failed to connect to redis: timeout
May 24 09:48:09 myhost kernel: I/O error, dev sda, sector 12345
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] connect() failed (111: Connection refused)

Boot logs (-b)

The journal indexes every boot, so -b gives you "just this boot" without timestamps. Negative offsets walk backwards through boot history; --list-boots shows the indexed list.

bash
journalctl -b                        # current boot only
journalctl -b -1                     # previous boot
journalctl -b -2 -p err              # errors from two boots ago
journalctl --list-boots              # list all stored boots
journalctl -k -b                     # kernel messages this boot
journalctl -b -1 -u nginx            # nginx logs from previous boot

Output (journalctl --list-boots):

text
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -3 a1b2c3d4e5f6...                  Mon 2026-05-20 08:01:14 UTC Mon 2026-05-20 22:14:55 UTC
 -2 b2c3d4e5f6a7...                  Tue 2026-05-21 07:55:01 UTC Tue 2026-05-21 23:02:11 UTC
 -1 c3d4e5f6a7b8...                  Wed 2026-05-22 08:11:30 UTC Wed 2026-05-22 21:48:09 UTC
  0 d4e5f6a7b8c9...                  Thu 2026-05-23 08:00:14 UTC Sat 2026-05-24 10:14:33 UTC

Field matches and structured data

The journal stores structured fields alongside every entry, not just a free-text line. You can filter on any field with FIELD=VALUE, and combine multiple matches: same field = OR, different fields = AND. Run journalctl -o verbose to see every field on a sample record.

FieldMeaning
_PIDProcess ID that emitted the record
_UID / _GIDUser / group ID
_COMMExecutable basename (nginx, sshd)
_EXEFull path to the binary
_SYSTEMD_UNITOwning systemd unit
_BOOT_IDBoot the entry came from
_HOSTNAMEHostname
PRIORITYSyslog priority (0–7)
SYSLOG_IDENTIFIERTag (often the program name)
MESSAGEThe free-text message
bash
journalctl _PID=1234                              # one process
journalctl _COMM=sshd                             # any sshd invocation
journalctl _SYSTEMD_UNIT=nginx.service _PID=1234  # both must match
journalctl _COMM=sshd _COMM=login                 # either sshd OR login
journalctl SYSLOG_IDENTIFIER=cron --since today
journalctl _UID=1000 -S yesterday                 # everything user 1000 did

Output (journalctl _COMM=sshd -n 3):

text
May 24 10:15:33 myhost sshd[8800]: Accepted publickey for alicedev from 10.0.0.5 port 52314 ssh2
May 24 10:15:33 myhost sshd[8800]: pam_unix(sshd:session): session opened for user alicedev
May 24 10:18:12 myhost sshd[8800]: pam_unix(sshd:session): session closed for user alicedev

Pattern matching with -g (PCRE2)

-g filters by a PCRE2 regular expression against the MESSAGE field — much faster than piping into grep because the match runs inside the journal reader before formatting. Pair with --case-sensitive=no for case-insensitive search; the default is smart-case (case-insensitive when the pattern is all-lowercase).

bash
journalctl -g 'timeout|refused|denied'           # any of three words
journalctl -u nginx -g '50[0-9]'                 # 5xx responses only
journalctl -g 'segfault' --case-sensitive=no
journalctl -u myapp -g 'request_id=abc[0-9]+' -S "1h ago"

Output (journalctl -u nginx -g '50[0-9]' -n 3):

text
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] 502 upstream prematurely closed
May 24 10:13:11 myhost nginx[1234]: 10.0.0.5 - alicedev "GET /api HTTP/1.1" 503 92
May 24 10:14:02 myhost nginx[1234]: 10.0.0.5 - alicedev "GET /healthz HTTP/1.1" 504 0

Excluding identifiers with -T (v256+)

-T / --exclude-identifier= drops entries whose SYSLOG_IDENTIFIER matches — the inverse of -t/--identifier=. Use it to mute one chatty source while still seeing everything else. Pass -T multiple times to exclude several identifiers.

bash
journalctl -T sudo                               # everything except sudo
journalctl -p err -T kernel                      # all errors but kernel ones
journalctl -T cron -T systemd -S "1h ago"        # mute two noisy sources

Output: (none — exits 0 on success)

Log namespaces

A journal namespace is an independent log stream with its own storage, retention, and journald instance. Namespaces both isolate noisy services (their data doesn't bloat the default journal) and reduce write contention on shared storage. The default namespace has no identifier; named namespaces are created by symlinking systemd-journald@NAME.service and adding LogNamespace=NAME to a unit.

bash
journalctl --list-namespaces                     # all configured namespaces (v256+)
journalctl --namespace=myapp                     # only the myapp namespace
journalctl --namespace='*'                       # all namespaces, interleaved
journalctl --namespace=+myapp                    # myapp + default, interleaved
journalctl --namespace=myapp -u worker -f        # follow one unit in a namespace

Output (journalctl --list-namespaces):

text
NAMESPACE  ACTIVE STATE   PERSISTENT VOLATILE
default    yes    running 482.3M     16.0M
myapp      yes    running 128.0M      8.0M
audit      yes    running 256.0M      4.0M

Unit invocations (v257+)

--list-invocations shows every time a unit was started, ended, and what exit status it produced — without manually correlating boot IDs. -I/--invocation= then jumps to the logs for one specific invocation, identified by either an index (most recent = ^1) or a 128-bit invocation ID.

bash
journalctl -u myapp --list-invocations           # table of recent runs
journalctl -u myapp -I ^1                        # the most recent invocation
journalctl -u myapp -I ^3                        # three invocations ago
journalctl -u myapp -I 9f8e7d6c5b4a...           # by invocation ID

Output (journalctl -u myapp --list-invocations -n 3):

text
IDX INVOCATION ID                    FIRST ENTRY                LAST ENTRY                 STATE
^3  3a2b1c0d4e5f6a7b8c9d0e1f2a3b4c5d Thu 2026-05-22 14:01:22 UTC Thu 2026-05-22 14:14:08 UTC dead
^2  4b3c2d1e5f6a7b8c9d0e1f2a3b4c5d6e Fri 2026-05-23 09:12:54 UTC Fri 2026-05-23 18:33:01 UTC dead
^1  5c4d3e2f6a7b8c9d0e1f2a3b4c5d6e7f Sat 2026-05-24 08:00:14 UTC -                          running

Offline journals: --root and --image

When recovering logs from a mounted backup, a chroot, or a disk image (e.g. a VM snapshot, an installer ISO, or a raw block device), point journalctl at the source instead of running it inside the system. --root=DIR operates on a mounted tree; --image=PATH mounts a disk image read-only and reads its /var/log/journal.

bash
journalctl --root=/mnt/recovery -b -1            # last boot of a mounted disk
journalctl --image=/srv/backups/server.img -k    # kernel log from a VM image
journalctl --image=/dev/sdb1 -p err -S "1 day ago"
journalctl --image=/var/lib/machines/web.raw --list-boots

Output (journalctl --root=/mnt/recovery --list-boots):

text
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -1 c3d4e5f6a7b8...                  Wed 2026-05-22 08:11:30 UTC Wed 2026-05-22 21:48:09 UTC
  0 d4e5f6a7b8c9...                  Thu 2026-05-23 08:00:14 UTC Sat 2026-05-24 02:14:33 UTC

Output formats

-o switches between rendering modes. cat strips all metadata for grep-friendly output; json and json-pretty emit one structured object per record for downstream tools like jq; json-seq prefixes each record with (RS, 0x1E) so streaming parsers can frame messages reliably even when fields contain embedded newlines; verbose shows every field on a record.

bash
journalctl -u nginx -o cat -n 5            # message only, no timestamps
journalctl -u nginx -o json -n 1           # one-line JSON
journalctl -u nginx -o json-pretty -n 1    # multi-line JSON
journalctl -u nginx -o json-seq -n 5       # RFC 7464 record-separator framed
journalctl -u nginx -o verbose -n 1        # all fields
journalctl -u nginx -o short-iso           # ISO-8601 timestamps
journalctl -u nginx -o short-iso-precise   # ISO-8601 with microseconds
journalctl -u nginx -o short-monotonic     # monotonic clock seconds since boot

Output (journalctl -u nginx -o cat -n 3):

text
Started nginx.service - A high performance web server.
10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET / HTTP/1.1" 200 1024
10.0.0.5 - alicedev [24/May/2026:10:14:02 +0000] "GET /style.css HTTP/1.1" 200 4096

Output (journalctl -u nginx -o json-pretty -n 1):

json
{
  "__REALTIME_TIMESTAMP" : "1748081643000123",
  "PRIORITY" : "6",
  "_PID" : "1234",
  "_UID" : "33",
  "_COMM" : "nginx",
  "_SYSTEMD_UNIT" : "nginx.service",
  "SYSLOG_IDENTIFIER" : "nginx",
  "MESSAGE" : "10.0.0.5 - alicedev [24/May/2026:10:14:03 +0000] \"GET /favicon.ico HTTP/1.1\" 404 162"
}

Disk usage and vacuum

The journal can grow large on busy hosts. --disk-usage reports its current footprint and --vacuum-* flags prune old data. Pruning never touches the live in-memory ring buffer, only on-disk files.

bash
journalctl --disk-usage              # total size on disk
sudo journalctl --vacuum-size=1G     # keep at most 1 GiB on disk
sudo journalctl --vacuum-time=7d     # delete entries older than 7 days
sudo journalctl --vacuum-files=10    # keep at most 10 journal files

Output:

text
Archived and active journals take up 482.3M in the file system.
Deleted archived journal /var/log/journal/abcd…/system@001.journal (32.0M).
Deleted archived journal /var/log/journal/abcd…/system@002.journal (32.0M).
Vacuuming done, freed 64.0M of archived journals from /var/log/journal/abcd….

Configuration

systemd-journald reads its configuration from /etc/systemd/journald.conf, then merges any drop-in files under /etc/systemd/journald.conf.d/*.conf (system-admin overrides) and /run/systemd/journald.conf.d/*.conf (runtime/vendor overrides). Drop-ins take precedence in lexical order, so prefer creating a small drop-in over editing the main file — package upgrades will clobber edits to journald.conf but leave *.conf.d/ alone. Each namespace has its own config: /etc/systemd/journald@NAME.conf plus matching journald@NAME.conf.d/*.conf. man journald.conf documents every setting.

ini
# /etc/systemd/journald.conf.d/50-jockey.conf — admin overrides
[Journal]
Storage=persistent           # auto | persistent | volatile | none
Compress=yes
SystemMaxUse=2G              # cap on /var/log/journal
SystemKeepFree=500M          # leave at least 500M free on the filesystem
SystemMaxFileSize=128M       # rotate at this size
MaxRetentionSec=30day        # delete entries older than this
ForwardToSyslog=no           # don't forward to rsyslog
ForwardToSocket=192.0.2.10:5140   # v256+: stream to an AF_INET/AF_VSOCK/AF_UNIX sink

By default many distributions store the journal only in /run/log/journal (RAM), which is wiped on reboot. Creating /var/log/journal plus Storage=persistent moves it to disk. After editing config:

bash
sudo mkdir -p /etc/systemd/journald.conf.d
sudo systemd-analyze cat-config systemd/journald.conf   # show effective merged config
sudo systemctl restart systemd-journald

Output (systemd-analyze cat-config systemd/journald.conf):

text
# /etc/systemd/journald.conf
[Journal]
#Storage=auto
#Compress=yes
...

# /etc/systemd/journald.conf.d/50-jockey.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
ForwardToSocket=192.0.2.10:5140

Per-unit namespaces

Attach a unit to a named journal namespace by adding LogNamespace= to its service file; systemd auto-starts the matching systemd-journald@NAME.service instance, which reads /etc/systemd/journald@NAME.conf.

ini
# /etc/systemd/system/myapp.service.d/10-namespace.conf
[Service]
LogNamespace=myapp
bash
sudo systemctl daemon-reload
sudo systemctl restart myapp
journalctl --namespace=myapp -u myapp -f

Output: (none — exits 0 on success)

Common pitfalls

  1. "No journal files were found" — persistent storage isn't enabled. sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald.
  2. Unit name typojournalctl -u ngnix silently returns nothing. Verify with systemctl list-units 'nginx*'.
  3. Quotes around --since--since 1 hour ago is three tokens and fails; use --since "1 hour ago".
  4. Trying to grep before -o cat — by default journalctl adds timestamp and hostname columns; grep anchors won't match the message body. Use -o cat or filter with _COMM / SYSLOG_IDENTIFIER directly.
  5. -f exits immediately under lessjournalctl -f | grep ERROR works but loses live tailing colors; for live grep use journalctl -f -u myapp | grep --line-buffered ERROR.
  6. Permissions — non-root users only see their own user-scope units plus system logs whose _UID matches. Add the user to the systemd-journal (or adm on Debian) group to read everything.
  7. Clock skew--since "1 hour ago" is computed in the host's local time. If your VM is in UTC and your logs span time zones, prefer absolute timestamps.

Real-world recipes

"Show me everything nginx logged in the last 10 minutes"

The most common triage query: one unit, one time window, paged in reverse for newest-first reading.

bash
journalctl -u nginx --since "10 min ago" --no-pager
journalctl -u nginx --since "10 min ago" -r        # newest first
journalctl -u nginx --since "10 min ago" -p err    # only errors

Output:

text
May 24 10:12:48 myhost nginx[1234]: 2026/05/24 10:12:48 [error] connect() failed (111: Connection refused)
May 24 10:13:01 myhost nginx[1234]: 2026/05/24 10:13:01 [warn] upstream server temporarily disabled
May 24 10:14:33 myhost nginx[1234]: 10.0.0.5 - alicedev "GET / HTTP/1.1" 200 1024

Boot-time post-mortem

After a crash, walk the previous boot's logs and look for the kernel's last words and any service that failed.

bash
journalctl -b -1 -p err              # all errors from last boot
journalctl -b -1 -k                  # kernel ring buffer last boot
journalctl -b -1 --no-pager | tail -100
systemctl --failed                   # current state of services

Output (systemctl --failed):

text
  UNIT                LOAD   ACTIVE SUB    DESCRIPTION
* myapp.service       loaded failed failed My Application
* postgresql.service  loaded failed failed PostgreSQL RDBMS

2 loaded units listed.

Save the last hour of one service to a file

-o cat strips metadata for downstream tools; --no-pager prevents paging when stdout is a pipe inside scripts.

bash
journalctl -u myapp --since "1 hour ago" -o cat --no-pager > /tmp/myapp.log

Output: (none — exits 0 on success)

Stream JSON into jq without losing the tail (v258+)

When you need structured fields for an alert pipeline, -o json plus jq is the cleanest path. Add --synchronize-on-exit=yes so that pressing Ctrl-C asks systemd-journald to flush everything queued before the signal arrived — without this, the last few events can be lost between the kernel buffer and disk.

bash
journalctl -u myapp -f -o json --synchronize-on-exit=yes | \
  jq -r 'select(.PRIORITY|tonumber < 4) | "\(.__REALTIME_TIMESTAMP) \(.MESSAGE)"'

Output:

text
1748081643000123 Database pool exhausted (size=20, waiting=3)
1748081709112301 Failed to connect to redis: timeout

Count errors per service over the last day

The classic "what's noisy?" report — grouped by unit, sorted by count.

bash
journalctl --since "1 day ago" -p err -o json --no-pager | \
  jq -r '._SYSTEMD_UNIT // "kernel"' | sort | uniq -c | sort -rn | head

Output:

text
    142 myapp.service
     38 nginx.service
     12 cron.service
      3 kernel

Find what filled the disk this hour

When df shows /var filling up, the journal itself is sometimes the culprit. Cap its size and then identify the chatty service.

bash
journalctl --disk-usage
journalctl --since "1 hour ago" -o json --no-pager | \
  jq -r '._SYSTEMD_UNIT // ._COMM' | sort | uniq -c | sort -rn | head
sudo journalctl --vacuum-size=500M

Output:

text
Archived and active journals take up 1.8G in the file system.
   8421 myapp.service
   1502 nginx.service
    214 sshd.service

Pair journalctl -u myapp -f with tmux in one pane and htop -p $(pgrep myapp) in another — you get live logs alongside live resource usage without leaving the terminal.

If a service logs structured fields via sd_journal_send(), those fields show up as first-class filterable keys (e.g. REQUEST_ID=…). Searching by request ID is how you trace one transaction through dozens of services on a single host.

Sources