Output and Logging

Cron mails output, it does not log it

If a job writes anything to stdout or stderr, cron mails that output to the crontab owner. If it writes nothing, cron says nothing. That is the whole mechanism, and on a machine with no mail transfer agent installed, which is most machines now, the output is generated and then dropped.

(CRON) MAIL (mailed 42 bytes of output but got status 0x004b)

That syslog line means the job produced output, cron tried to mail it, and the mailer failed. The output is gone. Anywhere a job "runs but nothing happens", this is the first thing to check.

The fix is to redirect explicitly, always, on every line.

0 3 * * * /srv/app/backup.sh >> /var/log/backup.log 2>&1

Redirection patterns

2>&1 must come after the file redirect. 2>&1 >> file sends stderr to the old stdout, which is the mail pipe, and only stdout to the file.

# The only-mail-me-on-failure idiom
0 3 * * * /srv/app/backup.sh > /dev/null

# Log everything and still get mail on failure
0 3 * * * /srv/app/backup.sh >> /var/log/backup.log 2>&1 || \
            echo "backup failed, see /var/log/backup.log"

Do not use > /dev/null 2>&1 on a job you care about. It is the single most common reason a broken cron job goes unnoticed for months.

MAILTO

MAILTO=ops@example.com          # send this crontab's output here
MAILTO=""                       # send nothing, for any job in this crontab
MAILTO=root,ops@example.com     # several recipients, comma separated
MAILFROM=cron@$(hostname)       # cronie, sets the envelope sender

- MAILTO applies to every job below it in the file, so it can be changed partway through. - MAILTO="" is the correct way to silence a crontab. It stops the mail without stopping the output, so a per-line redirect still captures it. - Unset means mail goes to the crontab owner, which for a system crontab is root. - MAILTO in /etc/cron.d files works per file, and cronie expands variables in MAILTO and MAILFROM, so MAILFROM=cron-$USER@example.com resolves.

Why a job fails silently

In rough order of how often each one is the answer:

1. Output was mailed into the void. No MTA, so nothing was recorded anywhere. 2. > /dev/null 2>&1 on the crontab line threw the error away deliberately. 3. PATH. The command was not found, sh printed to stderr, see 1 and 2. 4. The command needs a TTY. Anything interactive, or a tool that calls tput, stty, or isatty, behaves differently or hangs. 5. The script is not executable, or has no shebang, or has CRLF line endings, which produce /bin/bash^M: bad interpreter. 6. An unescaped % truncated the command at the percent sign. 7. Permissions. A different user, a different umask, no sudo without a TTY. 8. The previous run is still going. Overlapping runs, no lock, contention.

The 2>&1 | logger form makes numbers 1 through 3 impossible to miss:

0 3 * * * /srv/app/backup.sh 2>&1 | /usr/bin/logger -t backup -p cron.info

Where cron itself logs

Cron logs the fact that it ran a command, never the command's output.

# What has cron run recently
journalctl -u cron --since '1 hour ago'
journalctl -u crond -f
grep CRON /var/log/syslog | tail -20
sudo tail -f /var/log/cron

# Only the job you care about, if you tagged it with logger
journalctl -t backup --since today

A successful run looks like this, and note there is no output and no exit status:

Sep 15 03:00:01 host CRON[12345]: (deploy) CMD (/srv/app/backup.sh)

To see the start line at all on Debian you may need cron logging enabled in rsyslog, and cronie's -x flag adds debug categories if the daemon itself is suspect. A root crontab line beginning with - suppresses cron's own syslog entry for that command, which is occasionally used for very chatty per-minute jobs.

Capture the exit status, not just the text

Cron ignores the exit status entirely. A job that fails every night looks exactly like a job that succeeds every night unless you record the status yourself.

#!/bin/bash
set -euo pipefail
exec >> /var/log/backup.log 2>&1

start=$(date +%s)
echo "=== $(date -Is) start ==="

if /srv/app/backup.sh; then
  status=ok
else
  status=failed
fi

echo "=== $(date -Is) $status in $(( $(date +%s) - start ))s ==="
[ "$status" = ok ] || logger -t backup -p cron.err "backup failed"

For anything that matters, push the result somewhere that alerts when it stops arriving, a dead-man's-switch style check-in:

0 3 * * * /srv/app/backup.sh && \
            /usr/bin/curl -fsS -m 10 https://example.com/ping/backup

The point of the trailing ping is that it catches the failure mode logs cannot: the job that never ran at all, because the daemon was down, the crontab was deleted, or the machine was off.

Rotate the log you just created

A cron job appending to a file forever will fill the disk, and a full disk breaks every other job on the box.

# /etc/logrotate.d/app-cron
/var/log/backup.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
    create 0640 deploy adm
}

logrotate itself runs from /etc/cron.daily/logrotate or a systemd timer, so the rotation is only as reliable as the schedule running it.