SelfLearning

Study Linux Forensics #1

Hi there! ( ˙꒳˙ )ﾉ

This is my study note on Linux forensics. For details, please see the contents below. It was a great starting point for me to learn about Linux forensics!

Hal Pomeranz Linux Forensics Intro

Table of Contents

File system

Linux directories

Pathobjective
/usrThe files under there should not change.
/etcThe primary configuration directory for the system
/varLogging, specific system processes.
/homePersonal directories
/tmp /var/tmpWritable temporary derectories
/dev/shm/World-writable directory that only exists in memory. When the system is shut off or rebooted, the content of /dev/shm is lost.
/runAn in-memory file system for storing data that does not need to persist across reboots. The attacker has to achieve elevated privileges before writing into this directory

Additional notes

  • Executables normally live under
    • /usr , /usr/bin , /usr/sbin, /usr/libexec, etc
    • Exceptions
      • include 3rd-party software under /opt
  • Be suspicious of programs in user-writable directories
    • /tmp, /var/tmp, /dev/shm
    • User profile dirs under /home

Hidden files and directory

  • File and directory names starting with . are hidden
    • Use ls –a or ls –A to see them
  • Look for EXE paths not under /usr
  • Look for (deleted) binaries
  • The process’ current working directory is a hidden directory(/.)
  • Look for hidden files and directories
  • Look for executables outside of /usr

Useful command line

  • awk

    • For processing text, especially useful for extracting and manipulating columns in structured data.

    • Example:

      awk '/cwd/ && !/\/$/' live_response/process/lsof_-nPl.txt
      • Contain the string cwd (which stands for “current working directory”)
      • Do NOT end with a / (i.e., not a directory path that ends with a slash)
      • Identify processes whose current working directory (cwd) is not a standard directory (which usually ends with /).
  • zgrep

    • Search inside compressed .gz files using grep-like syntax.

    • Example:

      zgrep -Fi usage: strings.txt.gz
      • -F = treat the search term as a fixed string (not a regex)
      • -i = ignore case (matches Usage:, USAGE:, etc.)
      • Searches for the term usage: (case-insensitive) in the compressed file strings.txt.gz.
      zgrep -Fi -A20 usage: strings.txt.gz
      • -A20 = show 20 lines of context After the matched line
      • Same as above, but also shows 20 lines after each match.

Processes

Linux Process Hierarchy

  • systemd PID 1 : The main init system that starts everything after the kernel boots.
    • sshd [Remote logins]
      • SSH daemon started by systemd. It handles incoming remote login connections.
      • Under it: the user’s shells or commands run after logging in via SSH.
    • login [Console logins]
      • Handles physical terminal logins (e.g. from Ctrl+Alt+F2).
      • Under it: the user’s shell and processes started in the console.
    • X Windows manager [Graphical logins]
      • Starts the graphical desktop environment (e.g. GNOME, KDE).
      • Under it: user’s GUI apps, terminals, and other desktop processes.
  • Background services should not spawn shells!

Orphan Processes

  1. Execute program as background task
  2. Exit parent shell
  3. Orphan processes (=Child process) show their PPID as 1 systemd.

Scheduled Tasks

  • Path

    • [root]/etc/*cron*
    • [root]/var/spool/*cron

  • Example:

    */5 * * * * /tmp/.ICEd-unix/.src.sh
    • */5 = Run every 5 minutes
    • * * * * = Every hour, day, month, and weekday (i.e. no restrictions)
    • /tmp/.ICEd-unix/.src.sh → Run the shell script located at that path

  • Check the code and configuration files for legitimate scheduled tasks to make sure they haven’t been hijacked by an attacker.

Networking

  • How to find?

    • Processes talking to other hosts?
    • Processes listening on unexpected ports?

  • Example:

    tcp … 0.0.0.0:1337 0.0.0.0:* LISTEN 89 638037 58719/lsof
    • It seems like “beacon” – make short connections on regular intervals
    • lsof shows which files are currently open and which processes are using them
    • lsof is not a command that should be listening on a network port, and 1337/tcp is not a standard port used by any network service.

Useful Command line 2

  • grep

    • Searches for patterns in text.
    • Example:
    grep -Ev '/(usr|lib|etc|bin|sbin|opt|boot|home|var/lib)/' list_of_executable_files.txt
    • -E = enables extended regular expressions (e.g. using | for OR)
    • -v = inverts the match, i.e. shows lines that do NOT match
    • Look for another executables in unexpected directories

  • find

    • Example:
    find \[root\]/ -name \*cron\* -ls
    • Search for files and directories under [root]/, whose names contain the word cron, and then list detailed information (like ls -l).

  • ls

    • Example:
    ls -lrt \[root\]/etc/cron.* \[root\]/etc/crontab
    • List and inspect the modification time of cron-related configuration files in the [root]/etc/ directory.
    • -l: Long format (shows permissions, owner, size, etc.).
    • -r: Reverse the order (oldest to newest).
    • -t: Sort by modification time (newest first, but r reverses it).
    • [root]/etc/cron.*: Matches files like cron.d, cron.daily, cron.hourly, etc.
    • [root]/etc/crontab: The main system-wide crontab file.

User Context

Identity Basics

  • Users have a default User ID (UID) and Group ID (GID)

    • Assigned in /etc/passwd

    • Example:

      # "sshd" user is UID 108 and has a default GID of 65534.
sshd:x:108:65534::/run/sshd:/usr/sbin/nologin

  • Users may belong to other groups

    • As listed in /etc/group

      • Fields in it : group_name:x:GID:members

    • Example:

      plugdev:x:46:lab
staff:x:50:
games:x:60:
users:x:100:
nogroup:x:65534:
# The group nogroup is associated with GID 65534, typically used for unprivileged or anonymous users/processes.
# For example, if a process or file has no matching group, it might be assigned to nogroup.

  • Newly created files inherit the UID and default GID of the user that creates the file.

Additional notes 2

  • Clues to the initial compromise

    • What user is that malicious process running as?
    • Who owns the malicious executable?
    • What other processes are running as this user?
    • What other files/directories do they own?

  • Example:

    $ grep -E '(lsof|xterm)' live_response/process/ps_-ef.txt
mail 1811 1 0 16:59 pts/0 00:00:00 lsof -l -k -p 1337
mail 1817 1 0 16:59 pts/0 00:00:00 xterm
    • Use grep to search for potentially suspicious processes within trace files (e.g., from /proc or process memory dumps).
      • xterm : A terminal emulator for the X Window System. Its presence may indicate unauthorized shell access in a GUI session.
      • lsof : Lists open files associated with running processes. Useful for identifying unusual file access or persistence mechanisms.
      • tail : Displays the last lines of a file. Commonly used to monitor logs, but in forensics, can help verify if an attacker reviewed or tailed specific files in real time.

UID notes

  • Accounts with UID 0 has ADMIN RIGHTS

    • Normally only root account has UID 0

    • Fields in etc/passwd: username:password_placeholder:UID:GID:comment:home_directory:shell

      $ sort -t: -k3,3 -n /etc/passwd
root:x:0:0:root:/root:/bin/bash
toor:x:0:0:surprise!:/tmp:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
[… snip …]
      • You can find suspicious toor user.
      • ’ GID and UID are both 0 ’ means toor has full root privileges, just like the real root user.

  • Accounts with UID < 1000 are SERVICE ACCOUNTS

    • Should be locked
    • No interactive logins!

  • /etc/shadow : user password hashes are stored

sudo

  • su commands: Grants admin access on a limited basis
  • Configured via /etc/sudoers
  • Some groups have special Sudo privileges
    • Look out for unauthorized group memberships from sudoers

SET-ID bits

  • Some processes need to run with privilege
    • Set-UID : Run as executable owner, not user
    • Set-GID : Group privs of executable, not user
  • Adding set-UID is a subtle post-exploitation back door

AUTHORIZED_KEYS

  • [root]/home/*/.ssh
  • [root]/root/.ssh
    • ~/.ssh/authorized_keys
      • Contains public keys for SSH authentication
      • Popular post-compromise back door

Post-exploitation checklist

  • Added $HOME/.ssh/authorized_keys entries

  • Additional unexpected group members (Audit /etc/group)

  • Unlocked service accounts

    • noshell = stop login using shell
    • no password hashes in /etc/shadow
      • an entry with a blank password field (e.g., username::…) means that the account has no password.
      • An attacker who has already gained a shell on the system can use this to (1) Bypass Authentication, (2) Gain Privileges, (3) Create Backdoors.

  • Extra UID 0 accounts in /etc/passwd

    • Sample search command:

      awk -F: '$3 == 0' /etc/passwd

  • Direct /etc/sudoers modifications

  • Unauthorized set-UID bits

Memory forensics

Find kernel, build profile

  • Find debugging kernel

    $ find / -name vmlinu\* -size +100M 2>/dev/null
/usr/lib/debug/boot/vmlinux-5.10.0-21-amd64

  • Verify debugging kernel

    $ file /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64

/usr/lib/debug/boot/vmlinux-5.10.0-21-amd64: ELF 64-bit LSB executable, x86- 64, version 1 (SYSV), statically linked, BuildID[sha1]=5e5d3209033f927baa64…, with debug_info, not stripped

  • Check file size

    $ ls -lh /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64

rw-r--r-- 1 root root 627M Jan 21 14:35 /usr/lib/debug/boot/vmlinux-5.10.0-…

Memory acquisition tools

  • AVML(Acquire Volatile Memory Linux) – Free, file output only
  • LiME – Free, kernel driver, output to file or network
  • F-Response – Costs money, agent for disk/memory access

Volatility

Volatility basics

  • Volatility allows you to analyze a memory image to uncover live system activity, detect hidden malware, and extract volatile artifacts that aren’t visible through disk analysis alone.

  • Basic query

    vol3.py -s <Profilr location> -f <Memory Image File> <Volatility plugin choice>

vol3.py -s . -f memory.lime linux.pslist.PsList

Search kernel version by volatiliy

$ vol3.py -f memory.lime banners.Banners
  • Banners will give you enough clues to begin tracking down the debugging kernel you need to build a profile to analyze the memory image.

Additional plugins

  • Process Info
    • linux.psaux.PsAux: View the full command line used to execute the process
    • linux.pslist.PsList: View the process’s basic information (such as PID)
    • linux.pstree.PsTree: View the parent-child relationship between processes
  • Digging deeper
    • linux.lsof.Lsof: View files and sockets opened by the process
    • linux.elfs.Elfs: View the binaries and shared libraries currently in use
    • linux.proc.Maps: View the process’s memory mappings
    • linux.bash.Bash : Vew the command histories
      vol3.py -s . -f memory-avml.lime linux.bash.Bash --pid 1325
      • Command history only written to disk when shell exits
        • In-memory history has all commands for session
      • bash_history commands are not normally timestamped
        • Full timestamp information visible with linux_bash

This is just a personal memo of key terms I found important—if you’re interested in Linux forensics, please refer to the content above! I’ll write another post soon, as I couldn’t cover everything I learned this time.

Thanks for reading—see you in the next post! ( ˙꒳​˙ )ﾉﾞ