Study Linux Forensics 3

Hi there! ( ˙꒳˙ )ﾉ

This is my last study note on Linux forensics. For details, please see Hal Pomeranz Linux Forensics Intro.

Core log analysis

Linux logs

Generally found under /var/log
Logs are primarily text
- ⚠️ Easy to modify and manipulate
Logging is discretionary
- Amount and format of logs left to developers

wtmp read with last -if /mnt/test/data/var/log/wtmp
- User logins and system reboots
- File may be truncated weekly or monthly
btmp read with lastb -if /mnt/test/data/var/log/btmp
- Failed logins
- Often not kept due to risk of password disclosure
lastlog read with lastlog
- Last login for each user
- Varying formats make decoding tricky https://github.com/tigerphoenixdragon/lastlog_parser
- Only show the last login about existing user

SYSLOG

Syslog is the background service that receives/routes logs
Destination is usually local log files
- Default is restart logs weekly, keep four previous weeks
Can also route logs to other hosts over the network
- Always a good idea to aggregate longer term log history

SYSLOG CONFIGURATION

Type of log messages by “facility” and “priority”	Local file destinations
auth,authpriv.*	/var/log/auth.log
.;auth,authpriv.none	-/var/log/syslog
#cron.*	/var/log/cron.log
#daemon.*	-/var/log/daemon.log
kern.*	-/var/log/kern.log
#lpr.*	-/var/log/lpr.log
mail.*	-/var/log/mail.log
auth,authpriv.*	@loghost
*.notice;auth,authpriv.none	@loghost

SAMPLE LOG MESSAGES

Timestamp      Host     Process[PID]
Oct 5 13:13:53 VulnOSv2 sshd[2624]: Accepted password for mail from 192.168.210.131 port 57686 ssh2
Oct 5 13:13:53 VulnOSv2 sshd[2624]: pam_unix(sshd:session): session opened for user mail by (uid=0)
Oct 5 13:14:04 VulnOSv2 sudo: mail : TTY=pts/1 ; PWD=/var/mail ; USER=root ; COMMAND=/bin/su -
Oct 5 13:14:04 VulnOSv2 sudo: pam_unix(sudo:session): session opened for user root by mail(uid=0)
Oct 5 13:14:04 VulnOSv2 su[2721]: pam_unix(su:session): session opened for user root by mail(uid=0)
Oct 5 13:18:48 VulnOSv2 sshd[2624]: pam_unix(sshd:session): session closed for user mail

USEFUL LOGS

auth,authpriv.* – All things security-related

kern.* – USB and other device info, firewall logs

cron.* – Scheduled task execution

daemon.* – Other applications and services

Exercise 12: Log Analysis

Goals For This Lab

Investigate successful and failed logins
Track activity for mail user
Include log data in file system timeline
Continue to enhance incident timeline with new information

Before Get Started

root@LAB:~# cd /images/lab10/
root@LAB:/images/lab10# ls
Webserver.E01
root@LAB:/images/lab10# ls /mnt/test/img/
root@LAB:/images/lab10# ewfmount Webserver.E01 /mnt/test/img
ewfmount 20140807
root@LAB:/images/lab10# mmls /mnt/test/img/ewf1 
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)
001:  -------   0000000000   0000002047   0000002048   Unallocated
002:  000:000   0000002048   0000499711   0000497664   Linux (0x83)
003:  -------   0000499712   0000501759   0000002048   Unallocated
004:  Meta      0000501758   0066064383   0065562626   DOS Extended (0x05)
005:  Meta      0000501758   0000501758   0000000001   Extended Table (#1)
006:  001:000   0000501760   0066064383   0065562624   Linux Logical Volume Manager (0x8e)
007:  -------   0066064384   0066064607   0000000224   Unallocated
root@LAB:/images/lab10# losetup -rf -o $((501760*512)) /mnt/test/img/ewf1 
root@LAB:/images/lab10# vgscan
  Found volume group "LabVM" using metadata type lvm2
  WARNING: PV /dev/loop0 in VG VulnOSv2-vg is using an old PV header, modify the VG to update.
  Found volume group "VulnOSv2-vg" using metadata type lvm2
root@LAB:/images/lab10# vgchange -a y VulnOSv2-vg
  WARNING: PV /dev/loop0 in VG VulnOSv2-vg is using an old PV header, modify the VG to update.
  2 logical volume(s) in volume group "VulnOSv2-vg" now active
root@LAB:/images/lab10# lvscan | grep VulnOSv2-vg
  WARNING: PV /dev/loop0 in VG VulnOSv2-vg is using an old PV header, modify the VG to update.
  ACTIVE            '/dev/VulnOSv2-vg/root' [30.51 GiB] inherit
  ACTIVE            '/dev/VulnOSv2-vg/swap_1' [768.00 MiB] inherit
root@LAB:/images/lab10# ls /mnt/test/data/
root@LAB:/images/lab10# mount -o ro,noexec,noload /dev/VulnOSv2-vg/root /mnt/test/data
root@LAB:/images/lab10# mount -o ro,noexec,loop,offset=$((2048*512)),sizelimit=$((499712*512)) /mnt/test/img/ewf1 /mnt/test/data/boot
root@LAB:/images/lab10# ls /mnt/test/data/boot
abi-3.13.0-24-generic	      lost+found		System.map-3.13.0-24-generic
config-3.13.0-24-generic      memtest86+.bin		vmlinuz-3.13.0-24-generic
grub			      memtest86+.elf
initrd.img-3.13.0-24-generic  memtest86+_multiboot.bin

1. wtmp and btmp

cd /mnt/test/data/var/log
ls
last -if wtmp | head
lastb -if btmp
lastb -if btmp | grep 192.168.210.131 | wc -l

2. Getting More Data From Syslog

grep mail auth.log
grep 'password for mail' auth.log
grep mail auth.log | grep TTY=
grep php auth.log

3. Enhancing Timelines With Logs

Gather logs
- grep mail auth.log | head -6 >/images/lab11/syslogs
- grep 'password for mail' auth.log >>/images/lab11/syslogs
- grep mail auth.log | grep TTY= >>/images/lab11/syslogs
- grep php auth.log | grep 'Oct 5' >>/images/lab11/syslogs
Convert Syslog into body file style
- cd /images/lab11
- export TZ=CET
- syslog2mactime -y 2019 syslogs | gzip >bodyfile-syslog.gz
- unset TZ
Merge file into body files
- zcat bodyfile-* | mactime -d 2019-10-05 | grep -v deleted-realloc >timeline-with-logs.csv
- wc -l timeline-with-logs.csv

ADDITIONAL LOGS

OTHER USEFUL LOGS

Web server logs
- Often document the initial compromise
Kernel audit logs
- Optional mandatory logging, very detailed
Other application logs
- Databases, web proxies, …

WEB LOGS

Actual Logs
```
192.168.210.131 - - [05/Oct/2019:13:17:48 +0200]
	"GET /jabc/scripts/update.php HTTP/1.1" 200 223
	"http://192.168.210.135/jabc/scripts/"
	"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
```
- Source of request + Remote user and authenticated user (both usually “-”) + World’s most annoying time and date stamp
- Request method, path, and protocol + Returned result code + Bytes sent
- HTTP Referer and User Agent (optional)
Directories
- /var/log/httpd
- /var/log/apache
- /var/log/nginx
the log message is only put into the log file when the web server finishes processing the request.

DON’T FORGET ERROR LOGS!

[…]
PHP Notice: Use of undefined constant
	aygiTmxlbiIsICRsZW4pOyAkbGVuID0gJGFbJ2xlbiddOyAkYiA9ICcnOyB3aGlsZS
	Aoc3RybGVuKCRiKSA8ICRsZW4pIHsgc3dpdGNoICgkc190eXBlKSB7IGNhc2UgJ3N0
	cmVhbSc6ICRiIC49IGZyZWFkKCRzLCAkbGVuLXN0cmxlbigkYikpOyBicmVhazsgY2
	FzZSAnc29ja2V0JzogJGIgLj0gc29ja2V0X3JlYWQoJHMsICRsZW4tc3RybGVuKC…
[Sat Oct 05 13:17:48.483593 2019] [:error] [pid 1789]
	[client 192.168.210.131:41888] PHP Warning: system():
	Cannot execute a blank command in
	/var/www/html/jabc/scripts/update.php on line 2,
	referer: http://192.168.210.135/jabc/scripts/
[…]

LINUX KERNEL AUDITING

Kernel-level activity monitor can see everything /var/log/audit
- System booting
- User logins and privilege change/escalation
- Scheduled task execution
- SELINUX security policy violations
With additional configuration can log
- File access, modification, execution
- Any specific system call(s) across all processes
- User keystrokes
- Locally defined tags or keywords for later searching

ALL HAIL AUSEARCH!

How to use ausearch

# ausearch –if /mnt/evidence/var/log/audit -c useradd
----
time->Thu Feb 20 13:26:44 2020
type=PROCTITLE msg=audit(1582223204.906:342):
	proctitle=2F7573722F7362696E2F75736572616464002D64002F7573722F706870002D6D0
02D2D73797374656D002D2D7368656C6C002F62696E2F62617368002D2D736B656C002F6574
632F736B656C002D4700776865656C00706870
type=PATH msg=audit(1582223204.906:342): item=0 name="/etc/passwd"
	inode=135568 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
	obj=system_u:object_r:passwd_file_t:s0 objtype=NORMAL
	cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1582223204.906:342): cwd="/var/mail"
type=SYSCALL msg=audit(1582223204.906:342): arch=c000003e syscall=2
	success=yes exit=5 a0=55d79f171ce0 a1=20902 a2=0 a3=8 items=1 ppid=9425
	pid=9428 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
	tty=pts1 ses=3 comm="useradd" exe="/usr/sbin/useradd"
	subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="auth-files"

Pipe the hex encoded data

$ echo 2F7573722F7362696E2F75736572616464002D64002F7573722F706870002D6D0
02D2D73797374656D002D2D7368656C6C002F62696E2F62617368002D2D736B656C002F6574
632F736B656C002D4700776865656C00706870 | xxd -r -p | tr \\000 ' '; echo

/usr/sbin/useradd -d /usr/php -m --system --shell /bin/bash -
-skel /etc/skel -G wheel php

useful type=… messages found in audit logs:
- USER_AUTH, USER_LOGIN, USER_START , USER_END, USER_LOGOUT – user interactive logins (SSH sessions also use CRYPTO_KEY_USER, CRYPTO_SESSION)
- USER_CMD, PROCTITLE, PATH, CWD, SYSCALL – process execution and user activity
- ADD_USER, ADD_GROUP – account admin activity
- AVC – SELinux messages
- TTY, USER_TTY – keystroke logs (if enabled)
- LOGIN, USER_ACCT, USER_START, USER_END, CRED_ACQ, CRED_DISP, CRED_REFR – related to scheduled task start/stop
- SYSTEM_BOOT, SYSTEM_RUNLEVEL, KERN_MODULE, NETFILTER_CFG
- DAEMON_START, SERVICE_START, CFG_CHANGE – system boot and startup messages

OTHER TOOLS

aureport
- Generate summary reports for different event types
- Get detailed breakdowns with ausearch –a
- Example:
  - aureport -s -if /mnt/evidence/var/log/audit
- Dump the keystroke logs
  - aureport --tty
aulast
aulastlog
- Produce output like last and lastlog using audit logs

Sorry to repeat, but this is just a personal memo about something I found important.

Thanks for reading ( ˙꒳˙ )ﾉﾞ