User Units and Debugging

User Units

Every logged-in user gets their own systemd manager. Same syntax, no root, and unit files live under your home.

mkdir -p ~/.config/systemd/user
$EDITOR ~/.config/systemd/user/syncthing.service
systemctl --user daemon-reload
systemctl --user enable --now syncthing.service
systemctl --user status syncthing
journalctl --user -u syncthing -f

# ~/.config/systemd/user/syncthing.service
[Unit]
Description=Syncthing

[Service]
ExecStart=%h/.local/bin/syncthing serve --no-browser
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=default.target

Differences from a system unit:

• WantedBy=default.target, not multi-user.target. The user manager's default.target is an alias for graphical-session.target or just a plain target, depending on the session.
• No User= or Group=. The manager already runs as you.
• Search path is ~/.config/systemd/user/, then /etc/systemd/user/, then /usr/lib/systemd/user/.
• RuntimeDirectory= maps to $XDG_RUNTIME_DIR, StateDirectory= to ~/.local/state, CacheDirectory= to ~/.cache.
• Some sandboxing directives are unavailable without privileges, notably PrivateNetwork= and most of the Protect* kernel set.
• systemctl --global enable foo.service sets a default for every user.

Linger, or Your Unit Dies at Logout

By default the user manager starts on your first login and is killed when your last session ends. Your service goes with it, SSH out and it stops.

loginctl enable-linger $USER          # keep my manager running without a session
loginctl enable-linger deploy         # as root, for another account
loginctl disable-linger $USER
loginctl show-user $USER -p Linger
loginctl list-users
loginctl user-status $USER

With linger enabled the manager starts at boot and survives logout, which is what makes --user units usable for anything long-lived. Lingering state is a flag file under /var/lib/systemd/linger/.

The related knob is KillUserProcesses= in /etc/systemd/logind.conf. It defaults to no on most distributions, meaning stray background processes survive logout, but the manager still exits without linger. Do not rely on nohup semantics here, enable linger.

Reading a Failure

systemctl status tells you almost everything if you read all of it.

systemctl status myapp.service -l --no-pager
systemctl status myapp.service -n 50        # more log lines
systemctl show -p Result -p ExecMainStatus -p ExecMainCode myapp.service
journalctl -xeu myapp.service
journalctl -u myapp.service -p err -b

● myapp.service - My application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since ...
    Process: 5120 ExecStart=/usr/local/bin/myapp (code=exited, status=203/EXEC)
   Main PID: 5120 (code=exited, status=203/EXEC)

Read it in this order:

1. Loaded: confirms which file is in use and whether it is enabled or masked. not-found means a typo in the unit name.
2. Active: gives the state plus Result:, the machine-readable reason.
3. Process: gives the exit code. Anything in the 200s is systemd failing to launch, so the bug is in your unit file. 1/FAILURE and friends are your program.
4. The log tail underneath is the program's own output.
5. "changed on disk" means you forgot daemon-reload.

Common statuses: 203/EXEC bad path or not executable, 200/CHDIR missing WorkingDirectory=, 217/USER and 216/GROUP missing account, 226/NAMESPACE a sandboxing directive could not build its namespace, 238/STATE_DIRECTORY could not create the directory, start-limit-hit needs systemctl reset-failed.

The Debugging Toolkit

systemctl cat myapp.service                  # the loaded file plus every drop-in
systemctl show myapp.service                 # all resolved properties
systemctl show -p Environment -p ExecStart -p Restart myapp.service
systemctl show -p MainPID --value myapp.service
systemd-analyze verify /etc/systemd/system/myapp.service
systemd-analyze security myapp.service
systemd-analyze cat-config systemd/journald.conf
systemctl list-dependencies --reverse myapp.service
systemd-cgls -u myapp.service                # what is actually in the cgroup
systemctl status myapp.service | grep CGroup

systemd-delta is the one to run on a machine you inherited. It lists every /etc file that masks, extends or overrides something under /usr/lib.

Turning the Volume Up

# One unit, verbose child process
sudo systemctl edit myapp.service

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

# The manager itself
sudo systemd-analyze set-log-level debug
journalctl -f _PID=1
sudo systemd-analyze set-log-level info      # put it back
sudo systemd-analyze get-log-level

# From the boot prompt
systemd.log_level=debug systemd.log_target=console

# Per-invocation, for a manual run
SYSTEMD_LOG_LEVEL=debug systemctl start myapp.service

Manager debug logging is extremely loud. Set it, reproduce, read, and set it back. systemd-analyze set-log-target accepts console, journal, kmsg and null.

Reproducing Without Guessing

The fastest loop is to run the exact command in the exact context, outside the unit.

# Same user, same environment, as a transient unit
systemd-run --uid=myapp --pty --collect \
  --property=WorkingDirectory=/srv/myapp \
  --property=Environment=NODE_ENV=production \
  /usr/local/bin/myapp

# An interactive shell inside the service context
systemd-run --pty --uid=myapp --service-type=exec --collect /bin/bash

# Copy the real unit's sandboxing in one property at a time
systemd-run --pty --property=ProtectSystem=strict /bin/sh -c 'touch /etc/x'

--collect removes the transient unit afterwards so it does not linger in the failed list. --pty gives you a terminal, --wait returns the exit status, and --property= (-p) accepts any [Service] directive. This is how you bisect a sandbox: add one directive per run until it breaks.

Common Failure Modes