Linux logs: journald, auth logs, wtmp and btmp

On a Linux server, "who logged in, from where, when, and what did they do as root" is answered by text logs, the systemd journal and small binary login records. This lesson shows where each lives on Debian- and Red Hat-style systems, how to read them offline, how to turn their timestamps into exact UTC values, and how to corroborate one SSH login across all of them.

Text logs in /var/log

Text logs are written by a syslog daemon, usually rsyslog, following rules in /etc/rsyslog.conf and /etc/rsyslog.d/. Read them in the image: they show which file received what, and whether messages were forwarded to a remote log host the attacker could not reach.

Older Debian configurations also wrote daemon.log and user.log. Defaults change, so report what the image contains. In particular, new installations of Debian 12 and later do not necessarily include rsyslog; when it is absent, /var/log/auth.log does not exist and the journal is the only record of logins.

Rotation

logrotate (configured in /etc/logrotate.conf and /etc/logrotate.d/) renames logs on a schedule. On Debian the rsyslog files are typically rotated weekly with four old copies kept and delayed compression, giving auth.log, auth.log.1, auth.log.2.gz, auth.log.3.gz and auth.log.4.gz. RHEL-family systems usually add a date instead of a number (secure-20260308). Collect every rotated file, search compressed ones with zgrep, and order them by content, not name.

Timestamps

The classic syslog format (RFC 3164 style) starts each line with Mar 14 14:20:31: no year and no time zone. The time is the system's local time when the line was written. To convert it you must establish:

• Year: from the file's modification time, the rotation sequence, and dated sources such as the journal or wtmp. Watch for December lines in a file closed in January.
• Time zone: from /etc/localtime in the image (usually a symbolic link into /usr/share/zoneinfo/; resolve it inside the image, not on your analysis machine) and, on Debian, /etc/timezone.

rsyslog can instead write RFC 3339 timestamps, and the rsyslog packages of Debian 12 and Ubuntu 24.04 do so by default, with microseconds and an offset, e.g. 2026-03-14T10:43:52.104312-03:00. It carries the year and offset, so it converts without assumptions.

The systemd journal

systemd-journald stores logs in a binary, indexed format. Persistent files are in /var/log/journal/<machine-id>/, where the machine ID is the 32-hex-digit value in /etc/machine-id. Volatile files are in /run/log/journal/, RAM-backed, lost at reboot and absent from a disk image. The Storage= setting in /etc/systemd/journald.conf decides; its default auto stores persistently only if /var/log/journal exists.

The active file is system.journal (plus user-<UID>.journal for user sessions). Archived files are named system@….journal. A file not closed cleanly, for example after a crash, is renamed system@….journal~; such files are often still readable and must be collected too.

Each entry is a set of fields. Fields with one leading underscore are added by journald, so the logging program cannot forge them; two underscores mark addressing metadata.

Read a journal offline with journalctl -D /case/var/log/journal (a directory) or journalctl --file '/case/var/log/journal/<machine-id>/system@*.journal'. Use a recent journalctl; an older one can refuse files with newer format features. Useful options:

• --utc to print UTC; give --since "2026-03-14 13:00:00 UTC" and --until with an explicit UTC suffix.
• -u ssh for the OpenSSH unit on Debian and Ubuntu (-u sshd on RHEL), or _COMM=sshd. From OpenSSH 9.8 the per-connection work is done by sshd-session, so lines may carry that name instead.
• -o json or -o verbose to show every field, including __REALTIME_TIMESTAMP; -o short-iso-precise for readable microsecond times.
• --list-boots to list boot IDs with their first and last entries, then -b <boot ID>; --verify to check file consistency.

What the authentication lines mean

The same lines appear in the journal and, when rsyslog runs, in auth.log or secure.

To link a publickey login to a key, run ssh-keygen -lf on the account's ~/.ssh/authorized_keys from the image; it prints the SHA256 fingerprint of each key, and the comment field of the matching line often names its owner.

Binary login records

last and lastb print times in the analysis machine's time zone, hence TZ=UTC; -i prints ut_addr_v6 numerically; utmpdump wtmp dumps every field with microseconds.

utmp, wtmp and btmp share the glibc struct utmp, 384 bytes per record on x86-64 Linux: ut_type at offset 0 (7 USER_PROCESS login, 8 DEAD_PROCESS logout, 2 BOOT_TIME reboot), ut_pid at 4, ut_line (32 bytes, e.g. pts/0) at 8, ut_user (32 bytes) at 44, ut_host (256 bytes) at 76, ut_tv (32-bit seconds and microseconds since the Unix epoch, UTC) at 340 and ut_addr_v6 (16 bytes) at 348. A healthy file's size is a multiple of 384. last pairs a login with the next logout on the same ut_line. The 32-bit seconds field overflows in January 2038, so some recent distributions replace wtmp with wtmpdb, an SQLite database at /var/lib/wtmpdb/wtmp.db read with wtmpdb last; on such systems check which of btmp and lastlog are still written.

lastlog holds 292-byte records (ll_time 4 bytes, ll_line 32, ll_host 256) indexed by UID at offset UID × 292. It is a sparse file: a large UID makes its apparent size huge while little disk space is used. The lastlog command reads the running system's file by default, so on an image read the record at the computed offset.

Brute force and tampering

Brute force shows as hundreds of Failed password and Invalid user lines and btmp records from one IP within minutes, often cycling through names such as root, admin and test. The decisive finding is a later Accepted line from the same IP. Password spraying looks different: few attempts per account from many IPs. rsyslog can collapse identical consecutive lines into message repeated N times, so count btmp records as well as text lines.

Tampering indicators include gaps during known activity, an empty current log beside full rotated ones, missing rotation numbers, all-zero wtmp records, sizes not a multiple of 384, and journal and auth.log that disagree; Module 7 covers log clearing.

Worked example: reconstructing an SSH intrusion

A Debian server whose /etc/localtime points to America/Sao_Paulo (UTC−3, no daylight saving time in 2026) has rsyslog with the traditional format and a persistent journal.

1. Failures. TZ=UTC lastb -f btmp -i --time-format iso | grep -c 203.0.113.77 returns 1,240 records, first 2026-03-14T13:02:15Z, last 13:39:48Z. Duration: 37 min 33 s = 2,253 s, so 1,240 / 2,253 ≈ 0.55 attempts per second. If btmp holds only these records, its size must be 1,240 × 384 = 476,160 bytes.
2. Success. journalctl -D /case/var/log/journal _COMM=sshd -o json shows Accepted password for backup from 203.0.113.77 port 40122 ssh2 with __REALTIME_TIMESTAMP 1773495667482913. Split: 1,773,495,667 seconds and 482,913 microseconds. 1,773,495,667 / 86,400 = 20,526 days remainder 49,267 s; 49,267 s = 13 h 41 min 7 s; day 20,526 after 1970-01-01 is 2026-03-14. Result: 2026-03-14T13:41:07.482913Z, confirmed with python3 -c "import datetime as d; print(d.datetime(1970,1,1,tzinfo=d.timezone.utc)+d.timedelta(microseconds=1773495667482913))".
3. Privilege use. auth.log.1 contains Mar 14 10:43:52 srv01 sudo: backup : TTY=pts/0 ; PWD=/var/backups ; USER=root ; COMMAND=/usr/bin/tar czf /tmp/.c.tgz /etc /home. Assumptions: year 2026 (the file's modification time and the journal), local time UTC−3 from /etc/localtime, and a correct clock (lesson 3.5). 10:43:52 + 3 h = 2026-03-14T13:43:52Z, matching the journal entry for the same line.
4. Session. TZ=UTC last -f wtmp -i --time-format iso shows backup pts/0 203.0.113.77 2026-03-14T13:41:07+00:00 - 2026-03-14T14:12:40+00:00 (00:31), matching Disconnected from user backup 203.0.113.77 port 40122 at 14:12:40Z in the journal.
5. Last login. On Debian, backup is a system account with UID 34, so its lastlog record is at 34 × 292 = 9,928 bytes: ll_time decodes to 13:41:07Z, ll_host is 203.0.113.77. Since backup normally has a locked password and the shell /usr/sbin/nologin, check /etc/passwd, /etc/shadow and sudoers (lesson 6.3) for how it gained a password, shell and sudo rights.

Question → log → command

Exam focus: Expect to report a login source IP or a UTC time taken from Linux logs, and to convert a __REALTIME_TIMESTAMP value or a no-year syslog line correctly. Know which file holds failed logins (btmp), the login history (wtmp) and the RHEL name for the authentication log (secure). Be ready to state the assumptions behind a syslog conversion: year, time zone and clock.

Key points

• Authentication events go to auth.log (Debian/Ubuntu) or secure (RHEL family); collect every rotated copy, including .gz files.
• Classic syslog times have no year and no zone; recent Debian and Ubuntu write RFC 3339 times with an offset, and new Debian 12+ systems may have only the journal.
• The journal lives in /var/log/journal/<machine-id>/ (persistent) or /run/log/journal (lost at reboot); read it with journalctl -D and --utc; __REALTIME_TIMESTAMP is microseconds since the Unix epoch, UTC.
• utmp, wtmp and btmp use 384-byte records with UTC epoch times; run last and lastb with TZ=UTC.
• lastlog is indexed by UID at UID × 292 bytes; parse it directly from an image.
• Corroborate a login across btmp, the sshd line, wtmp, lastlog and sudo lines.