Study Linux Forensics 3

Hi there! ( ˙꒳˙ )ﾉ

This is my last study note on Linux forensics. For details, please see Hal Pomeranz Linux Forensics Intro.

Core log analysis

Linux logs

Generally found under /var/log
Logs are primarily text
- ⚠️ Easy to modify and manipulate
Logging is discretionary
- Amount and format of logs left to developers

wtmp read with last -if /mnt/test/data/var/log/wtmp
- User logins and system reboots
- File may be truncated weekly or monthly
btmp read with lastb -if /mnt/test/data/var/log/btmp
- Failed logins
- Often not kept due to risk of password disclosure
lastlog read with lastlog
- Last login for each user
- Varying formats make decoding tricky https://github.com/tigerphoenixdragon/lastlog_parser
- Only show the last login about existing user

SYSLOG

Syslog is the background service that receives/routes logs
Destination is usually local log files
- Default is restart logs weekly, keep four previous weeks
Can also route logs to other hosts over the network
- Always a good idea to aggregate longer term log history

SYSLOG CONFIGURATION

Type of log messages by “facility” and “priority”	Local file destinations
auth,authpriv.*	/var/log/auth.log
.;auth,authpriv.none	-/var/log/syslog
#cron.*	/var/log/cron.log
#daemon.*	-/var/log/daemon.log
kern.*	-/var/log/kern.log
#lpr.*	-/var/log/lpr.log
mail.*	-/var/log/mail.log
auth,authpriv.*	@loghost
*.notice;auth,authpriv.none	@loghost

SAMPLE LOG MESSAGES

Timestamp      Host     Process[PID]
Oct 5 13:13:53 VulnOSv2 sshd[2624]: Accepted password for mail from 192.168.210.131 port 57686 ssh2
Oct 5 13:13:53 VulnOSv2 sshd[2624]: pam_unix(sshd:session): session opened for user mail by (uid=0)
Oct 5 13:14:04 VulnOSv2 sudo: mail : TTY=pts/1 ; PWD=/var/mail ; USER=root ; COMMAND=/bin/su -
Oct 5 13:14:04 VulnOSv2 sudo: pam_unix(sudo:session): session opened for user root by mail(uid=0)
Oct 5 13:14:04 VulnOSv2 su[2721]: pam_unix(su:session): session opened for user root by mail(uid=0)
Oct 5 13:18:48 VulnOSv2 sshd[2624]: pam_unix(sshd:session): session closed for user mail

USEFUL LOGS

auth,authpriv.* – All things security-related

kern.* – USB and other device info, firewall logs

cron.* – Scheduled task execution

daemon.* – Other applications and services

Exercise 12: Log Analysis

Before Get Started

root@LAB:~# cd /images/lab10/
root@LAB:/images/lab10# ls
Webserver.E01
root@LAB:/images/lab10# ls /mnt/test/img/
root@LAB:/images/lab10# ewfmount Webserver.E01 /mnt/test/img
ewfmount 20140807
root@LAB:/images/lab10# mmls /mnt/test/img/ewf1 
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)
001:  -------   0000000000   0000002047   0000002048   Unallocated
002:  000:000   0000002048   0000499711   0000497664   Linux (0x83)
003:  -------   0000499712   0000501759   0000002048   Unallocated
004:  Meta      0000501758   0066064383   0065562626   DOS Extended (0x05)
005:  Meta      0000501758   0000501758   0000000001   Extended Table (#1)
006:  001:000   0000501760   0066064383   0065562624   Linux Logical Volume Manager (0x8e)
007:  -------   0066064384   0066064607   0000000224   Unallocated
root@LAB:/images/lab10# losetup -rf -o $((501760*512)) /mnt/test/img/ewf1 
root@LAB:/images/lab10# vgscan
  Found volume group "LabVM" using metadata type lvm2
  WARNING: PV /dev/loop0 in VG VulnOSv2-vg is using an old PV header, modify the VG to update.
  Found volume group "VulnOSv2-vg" using metadata type lvm2
root@LAB:/images/lab10# vgchange -a y VulnOSv2-vg
  WARNING: PV /dev/loop0 in VG VulnOSv2-vg is using an old PV header, modify the VG to update.
  2 logical volume(s) in volume group "VulnOSv2-vg" now active
root@LAB:/images/lab10# lvscan | grep VulnOSv2-vg
  WARNING: PV /dev/loop0 in VG VulnOSv2-vg is using an old PV header, modify the VG to update.
  ACTIVE            '/dev/VulnOSv2-vg/root' [30.51 GiB] inherit
  ACTIVE            '/dev/VulnOSv2-vg/swap_1' [768.00 MiB] inherit
root@LAB:/images/lab10# ls /mnt/test/data/
root@LAB:/images/lab10# mount -o ro,noexec,noload /dev/VulnOSv2-vg/root /mnt/test/data
root@LAB:/images/lab10# mount -o ro,noexec,loop,offset=$((2048*512)),sizelimit=$((499712*512)) /mnt/test/img/ewf1 /mnt/test/data/boot
root@LAB:/images/lab10# ls /mnt/test/data/boot
abi-3.13.0-24-generic	      lost+found		System.map-3.13.0-24-generic
config-3.13.0-24-generic      memtest86+.bin		vmlinuz-3.13.0-24-generic
grub			      memtest86+.elf
initrd.img-3.13.0-24-generic  memtest86+_multiboot.bin

1. wtmp and btmp

cd /mnt/test/data/var/log
ls
last -if wtmp | head
lastb -if btmp
lastb -if btmp | grep 192.168.210.131 | wc -l

2. Getting More Data From Syslog

grep mail auth.log
grep 'password for mail' auth.log
grep mail auth.log | grep TTY=
grep php auth.log

3. Enhancing Timelines With Logs

Gather logs
- grep mail auth.log | head -6 >/images/lab11/syslogs
- grep 'password for mail' auth.log >>/images/lab11/syslogs
- grep mail auth.log | grep TTY= >>/images/lab11/syslogs
- grep php auth.log | grep 'Oct 5' >>/images/lab11/syslogs
Convert Syslog into body file style
- cd /images/lab11
- export TZ=CET
- syslog2mactime -y 2019 syslogs | gzip >bodyfile-syslog.gz
- unset TZ
Merge file into body files
- zcat bodyfile-* | mactime -d 2019-10-05 | grep -v deleted-realloc >timeline-with-logs.csv
- wc -l timeline-with-logs.csv

ADDITIONAL LOGS

OTHER USEFUL LOGS

Web server logs
- Often document the initial compromise
Kernel audit logs
- Optional mandatory logging, very detailed
Other application logs
- Databases, web proxies, …

WEB LOGS

Actual Logs
```
192.168.210.131 - - [05/Oct/2019:13:17:48 +0200]
	"GET /jabc/scripts/update.php HTTP/1.1" 200 223
	"http://192.168.210.135/jabc/scripts/"
	"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
```
- Source of request + Remote user and authenticated user (both usually “-”) + World’s most annoying time and date stamp
- Request method, path, and protocol + Returned result code + Bytes sent
- HTTP Referer and User Agent (optional)
Directories
- /var/log/httpd
- /var/log/apache
- /var/log/nginx
the log message is only put into the log file when the web server finishes processing the request.

DON’T FORGET ERROR LOGS!

[…]
PHP Notice: Use of undefined constant
	aygiTmxlbiIsICRsZW4pOyAkbGVuID0gJGFbJ2xlbiddOyAkYiA9ICcnOyB3aGlsZS
	Aoc3RybGVuKCRiKSA8ICRsZW4pIHsgc3dpdGNoICgkc190eXBlKSB7IGNhc2UgJ3N0
	cmVhbSc6ICRiIC49IGZyZWFkKCRzLCAkbGVuLXN0cmxlbigkYikpOyBicmVhazsgY2
	FzZSAnc29ja2V0JzogJGIgLj0gc29ja2V0X3JlYWQoJHMsICRsZW4tc3RybGVuKC…
[Sat Oct 05 13:17:48.483593 2019] [:error] [pid 1789]
	[client 192.168.210.131:41888] PHP Warning: system():
	Cannot execute a blank command in
	/var/www/html/jabc/scripts/update.php on line 2,
	referer: http://192.168.210.135/jabc/scripts/
[…]

LINUX KERNEL AUDITING

Kernel-level activity monitor can see everything /var/log/audit
- System booting
- User logins and privilege change/escalation
- Scheduled task execution
- SELINUX security policy violations
With additional configuration can log
- File access, modification, execution
- Any specific system call(s) across all processes
- User keystrokes
- Locally defined tags or keywords for later searching

ALL HAIL AUSEARCH!

How to use ausearch

# ausearch –if /mnt/evidence/var/log/audit -c useradd
----
time->Thu Feb 20 13:26:44 2020
type=PROCTITLE msg=audit(1582223204.906:342):
	proctitle=2F7573722F7362696E2F75736572616464002D64002F7573722F706870002D6D0
02D2D73797374656D002D2D7368656C6C002F62696E2F62617368002D2D736B656C002F6574
632F736B656C002D4700776865656C00706870
type=PATH msg=audit(1582223204.906:342): item=0 name="/etc/passwd"
	inode=135568 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
	obj=system_u:object_r:passwd_file_t:s0 objtype=NORMAL
	cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1582223204.906:342): cwd="/var/mail"
type=SYSCALL msg=audit(1582223204.906:342): arch=c000003e syscall=2
	success=yes exit=5 a0=55d79f171ce0 a1=20902 a2=0 a3=8 items=1 ppid=9425
	pid=9428 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
	tty=pts1 ses=3 comm="useradd" exe="/usr/sbin/useradd"
	subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="auth-files"

Pipe the hex encoded data

$ echo 2F7573722F7362696E2F75736572616464002D64002F7573722F706870002D6D0
02D2D73797374656D002D2D7368656C6C002F62696E2F62617368002D2D736B656C002F6574
632F736B656C002D4700776865656C00706870 | xxd -r -p | tr \\000 ' '; echo

/usr/sbin/useradd -d /usr/php -m --system --shell /bin/bash -
-skel /etc/skel -G wheel php

useful type=… messages found in audit logs:
- USER_AUTH, USER_LOGIN, USER_START , USER_END, USER_LOGOUT – user interactive logins (SSH sessions also use CRYPTO_KEY_USER, CRYPTO_SESSION)
- USER_CMD, PROCTITLE, PATH, CWD, SYSCALL – process execution and user activity
- ADD_USER, ADD_GROUP – account admin activity
- AVC – SELinux messages
- TTY, USER_TTY – keystroke logs (if enabled)
- LOGIN, USER_ACCT, USER_START, USER_END, CRED_ACQ, CRED_DISP, CRED_REFR – related to scheduled task start/stop
- SYSTEM_BOOT, SYSTEM_RUNLEVEL, KERN_MODULE, NETFILTER_CFG
- DAEMON_START, SERVICE_START, CFG_CHANGE – system boot and startup messages

OTHER TOOLS

aureport
- Generate summary reports for different event types
- Get detailed breakdowns with ausearch –a
- Example:
  - aureport -s -if /mnt/evidence/var/log/audit
- Dump the keystroke logs
  - aureport --tty
aulast
aulastlog
- Produce output like last and lastlog using audit logs

Sorry to repeat, but this is just a personal memo about something I found important.

Thanks for reading ( ˙꒳˙ )ﾉﾞ