Forensic Artifacts
There exists a number of utility programs that can generate the information listed below such as wbinfo and getent. Since we are interested in forensic evidence artifacts that can be collected and reviewed offline the focus here is to examine the file contents directly. This is also required when we are presented with a disk image of a system and not the live running system.
Unexpected Local user accounts
Look for unusual user accounts in the /etc/passwd
#!/bin/bash
NI_SHELLS="/sbin/nologin|/sbin/false|/sbin/true|/bin/sync|/sbin/shutdown|/sbin/halt"
echo "Non Interactive / Service Accounts"
echo "**********************************"
egrep $NI_SHELLS /etc/passwd | cut -d: -f1
echo ""
echo "Interactive Users"
echo "*****************"
egrep -v $NI_SHELLS /etc/passwd | cut -d: -f1
echo ""
Sample output
[tlasso@rhel-richmondfc scripts]$ ./list_users.sh
Non Interactive / Service Accounts
**********************************
bin
daemon
adm
lp
sync
shutdown
halt
mail
operator
games
ftp
nobody
systemd-coredump
dbus
polkitd
avahi
tss
colord
clevis
rtkit
sssd
geoclue
libstoragemgmt
systemd-oom
setroubleshoot
pipewire
flatpak
gdm
cockpit-ws
cockpit-wsinstance
gnome-initial-setup
sshd
chrony
dnsmasq
tcpdump
Interactive Users
*****************
root
tlasso
The /etc/group, /etc/shadow, /etc/gshadow must be cross referenced to check for any user ids or groups that are not recognized.
This can be sign of an orphaned user account that was not removed correctly.
Account Earliest Activity & Deleted Accounts
The user home directory is created for all interactive users (unless explicitly configured).
Examine the /home/ folder for a list of home directories.
[tlasso@rhel-richmondfc ~]$ sudo ls -al /home
total 12
drwxr-xr-x. 5 root root 84 Jun 14 03:21 .
dr-xr-xr-x. 18 root root 235 Jun 14 02:26 ..
drwx------. 14 357201109 357200513 4096 Jun 14 03:23 nate@corp.tuxtriage.net
drwx------. 15 tlasso tlasso 4096 Jun 18 19:25 tlasso
drwx------. 14 357201108 357200513 4096 Jun 14 03:14 tlasso@corp.tuxtriage.net
Note that in the above example there are two tlasso users. The tlasso user is a local system user while the tlasso@corp.tuxtriage.net is an AD user.
Since one of the first things the system does when a user is created is the creation of the home directory, this information can be approximated to the earliest presense of this user on the host.
If the investigation timelines are outside the earliest activity date, the user account may not be part of the incident.
The Birth time of the folder can be used for this purpose.
[tlasso@rhel-richmondfc ~]$ stat /home/*
File: /home/nate@corp.tuxtriage.net
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 102693422 Links: 14
Access: (0700/drwx------) Uid: (357201109/ UNKNOWN) Gid: (357200513/ UNKNOWN)
Context: unconfined_u:object_r:user_home_dir_t:s0
Access: 2023-06-14 03:21:01.790738178 -0400
Modify: 2023-06-14 03:23:10.091326083 -0400
Change: 2023-06-14 03:23:10.091326083 -0400
Birth: 2023-06-14 03:21:01.790738178 -0400
File: /home/tlasso
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 69333680 Links: 15
Access: (0700/drwx------) Uid: ( 1000/ tlasso) Gid: ( 1001/ tlasso)
Context: unconfined_u:object_r:user_home_dir_t:s0
Access: 2023-06-18 19:32:30.473261644 -0400
Modify: 2023-06-18 19:25:15.008942407 -0400
Change: 2023-06-18 19:25:15.008942407 -0400
Birth: 2023-06-14 02:30:33.307059622 -0400
File: /home/tlasso@corp.tuxtriage.net
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 101569584 Links: 14
Access: (0700/drwx------) Uid: (357201108/ UNKNOWN) Gid: (357200513/ UNKNOWN)
Context: unconfined_u:object_r:user_home_dir_t:s0
Access: 2023-06-14 03:11:59.697315839 -0400
Modify: 2023-06-14 03:14:12.457619105 -0400
Change: 2023-06-14 03:14:12.457619105 -0400
Birth: 2023-06-14 03:11:59.697315839 -0400
Last logon time
Compile a comprehensive list detailing the latest logon times for all local accounts by parsing the /var/log/lastlog.
This following script, automates the traversal and data extraction. It reads the lastlog file, usually located at /var/log/lastlog, and iterates over it, decoding the binary data into human-readable format. For each record, it prints out the user ID along with the respective last logon timestamp.
#!/usr/bin/env python
# -*- coding:utf-8 -*-
import struct
import collections
import datetime
import os
LSTLOGSTRUCT = struct.Struct('=l32s256s')
LSTLOGRecord = collections.namedtuple(
"LLRecord",
'time line host'
)
lastlog_path='/var/log/lastlog'
with open(lastlog_path, 'rb') as fp:
uid = 0
seekto = 0
size = LSTLOGSTRUCT.size
end = os.fstat(fp.fileno()).st_size
while True:
fp.seek(seekto, 0)
bytes = fp.read(LSTLOGSTRUCT.size)
if not bytes:
break
if seekto > end:
break;
data = LSTLOGSTRUCT.unpack(bytes)
time, line, host = data
time = datetime.datetime.fromtimestamp(time) if time else None
line = line.strip(b'')
host = host.strip(b'')
if time is not None:
print("UID: {}. Last Logon: {}. Host: {}. Line:{}".format( uid, time, host, line))
uid = uid+1
seekto = seekto + size
[tlasso@rhel-richmondfc Documents]$ ./test.py
UID: 0. Last Logon: 2023-06-14 02:53:13. Host: b''. Line:b'pts/0'
UID: 42. Last Logon: 2023-07-02 12:04:28. Host: b''. Line:b'tty1'
UID: 1000. Last Logon: 2023-07-02 12:04:37. Host: b''. Line:b'tty2'
SSSD Cache
Any user that logs on via AD does not leave much trace on the system maintained files such as /etc/passwd etc. However, there are cache files that are updated as part of the logon process that can be analyzed to detect malicious user activity.
SSSD cache is located at /var/lib/sss/db.
[tlasso@rhel-richmondfc scripts]$ sudo ls /var/lib/sss/db
cache_corp.tuxtriage.net.ldb ccache_CORP.TUXTRIAGE.NET config.ldb sssd.ldb timestamps_corp.tuxtriage.net.ldb
Of interest to us is the cache_corp.tuxtriage.net.ldb ldb database. See details about lbd format here 1.
ldbsearch command can be used to search this database for records of interest.
tlasso@rhel-richmondfc scripts]$ sudo ldbsearch -H /var/lib/sss/db/cache_corp.tuxtriage.net.ldb
asq: Unable to register control with rootdse!
# record 1
dn: gpoGUID={31B2F340-016D-11D2-945F-00C04FB984F9},cn=gpos,cn=ad,cn=custom,cn=corp.tuxtriage.net,cn=sysdb
gpoGUID: {31B2F340-016D-11D2-945F-00C04FB984F9}
gpoVersion: 3
objectClass: gpo
gpoPolicyFileTimeout: 1686727266
distinguishedName: gpoGUID={31B2F340-016D-11D2-945F-00C04FB984F9},cn=gpos,cn=a
d,cn=custom,cn=corp.tuxtriage.net,cn=sysdb
...
# record 3
dn: name=tlasso@corp.tuxtriage.net,cn=users,cn=corp.tuxtriage.net,cn=sysdb
createTimestamp: 1686726711
fullName: Ted Lasso
gecos: Ted Lasso
gidNumber: 357200513
name: tlasso@corp.tuxtriage.net
objectCategory: user
uidNumber: 357201108
objectSIDString: S-1-5-21-2369507818-3716282536-1365522936-1108
uniqueID: 5e499899-1305-460b-82d9-87a949f32d18
originalDN: CN=Ted Lasso,CN=Users,DC=corp,DC=tuxtriage,DC=net
originalModifyTimestamp: 20230614062722.0Z
entryUSN: 41335
userPrincipalName: tlasso@CORP.TUXTRIAGE.NET
adAccountExpires: 9223372036854775807
adUserAccountControl: 512
nameAlias: tlasso@corp.tuxtriage.net
isPosix: TRUE
lastUpdate: 1686726711
dataExpireTimestamp: 1686732111
initgrExpireTimestamp: 0
ccacheFile: KCM:
cachedPasswordType: 1
failedLoginAttempts: 0
memberof: name=Domain Users@corp.tuxtriage.net,cn=groups,cn=corp.tuxtriage.net
,cn=sysdb
pacBlob:: BwAAAAAAAAABAAAA2AEAAHgAAAAAAAAABgAAABAAAABQAgAAAAAAAAcAAAAQAAAAYAIA
AAAAAAAKAAAAFgAAAHACAAAAAAAADAAAAKgAAACIAgAAAAAAABAAAAAQAAAAMAMAAAAAAAATAAAAE
AAAAEADAAAAAAAAARAIAMzMzMzIAQAAAAAAAAAAAgCpHPiCj57ZAf////////9//////////38lSO
9GiZ7ZASUIWXFSn9kBJchIPIq/2QEMAAwABAACABIAEgAIAAIAAAAAAAwAAgAAAAAAEAACAAAAAAA
UAAIAAAAAABgAAgABAAAAVAQAAAECAAABAAAAHAACACAAAAAAAAAAAAAAAAAAAAAAAAAAHgAgACAA
AgAIAAoAJAACACgAAgAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAA
CwAAgAAAAAAAAAAAAAAAAAGAAAAAAAAAAYAAAB0AGwAYQBzAHMAbwAJAAAAAAAAAAkAAABUAGUAZA
AgAEwAYQBzAHMAbwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAEAAAABAgAABwAAABAAAAAAAAAADwAAAFcASQBOAC0AVABVAEUAMQBWADIATQBKAEYANQBS
AAAABQAAAAAAAAAEAAAAQwBPAFIAUAAEAAAAAQQAAAAAAAUVAAAA6tE7jaj4gd34OWRRAQAAADAAA
gAHAAAAAQAAAAEBAAAAAAASAQAAABAAAABi2JyYFqoO2sq6CcwQAAAACawZ0ByD2C8dtvh8gE5Eq5
Ce2QEMAHQAbABhAHMAcwBvAAAAMgAYACQAUAACAAAADAB4ABwAiAAAAAAAdABsAGEAcwBzAG8AQAB
jAG8AcgBwAC4AdAB1AHgAdAByAGkAYQBnAGUALgBuAGUAdAAAAAAAAABDAE8AUgBQAC4AVABVAFgA
VABSAEkAQQBHAEUALgBOAEUAVAAAAAAAdABsAGEAcwBzAG8AAAAAAAEFAAAAAAAFFQAAAOrRO42o+
IHd+DlkUVQEAAAAAAAAEAAAAAElgqwi0pz73PmFIhAAAADlOc348iFviKxpfwI=
pacBlobExpireTimestamp: 1686727516
cachedPassword: $6$TuYfJdJ.n2nhNpsA$e8b8IGjnqoNmQCrWlit/Ucn7M2Iie2jq3DwOHEXTIz
FzY17zamTuy3loYoH/w5DVYqS7t8lM8v6Be2ijrMmLa0
lastCachedPasswordChange: 1686727216
lastOnlineAuth: 1686727216
lastOnlineAuthWithCurrentToken: 1686727216
lastLogin: 1686727216
distinguishedName: name=tlasso@corp.tuxtriage.net,cn=users,cn=corp.tuxtriage.n
et,cn=sysdb
...
# record 8
dn: name=nate@corp.tuxtriage.net,cn=users,cn=corp.tuxtriage.net,cn=sysdb
createTimestamp: 1686727258
fullName: Nathan Shelley
gecos: Nathan Shelley
gidNumber: 357200513
name: nate@corp.tuxtriage.net
objectCategory: user
uidNumber: 357201109
objectSIDString: S-1-5-21-2369507818-3716282536-1365522936-1109
uniqueID: a4d7ff67-c14f-4568-b803-b1344d456af8
originalDN: CN=Nathan Shelley,CN=Users,DC=corp,DC=tuxtriage,DC=net
originalModifyTimestamp: 20230614062944.0Z
entryUSN: 41342
userPrincipalName: nate@CORP.TUXTRIAGE.NET
adAccountExpires: 9223372036854775807
adUserAccountControl: 512
nameAlias: nate@corp.tuxtriage.net
isPosix: TRUE
lastUpdate: 1686727258
dataExpireTimestamp: 1686732658
initgrExpireTimestamp: 0
memberof: name=Domain Users@corp.tuxtriage.net,cn=groups,cn=corp.tuxtriage.net
,cn=sysdb
pacBlob:: BwAAAAAAAAABAAAA4AEAAHgAAAAAAAAABgAAABAAAABYAgAAAAAAAAcAAAAQAAAAaAIA
AAAAAAAKAAAAEgAAAHgCAAAAAAAADAAAAJgAAACQAgAAAAAAABAAAAAQAAAAKAMAAAAAAAATAAAAE
AAAADgDAAAAAAAAARAIAMzMzMzQAQAAAAAAAAAAAgAAAAAAAAAAAP////////9//////////38tnc
mbiZ7ZAS1dM8ZSn9kBLR0jkYq/2QEIAAgABAACABwAHAAIAAIAAAAAAAwAAgAAAAAAEAACAAAAAAA
UAAIAAAAAABgAAgAAAAAAVQQAAAECAAABAAAAHAACACAAAAAAAAAAAAAAAAAAAAAAAAAAHgAgACAA
AgAIAAoAJAACACgAAgAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAA
CwAAgAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAABuAGEAdABlAA4AAAAAAAAADgAAAE4AYQB0AGgAYQ
BuACAAUwBoAGUAbABsAGUAeQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAABAAAAAQIAAAcAAAAQAAAAAAAAAA8AAABXAEkATgAtAFQAVQBFADEAVgAyAE0ASgBG
ADUAUgAAAAUAAAAAAAAABAAAAEMATwBSAFAABAAAAAEEAAAAAAAFFQAAAOrRO42o+IHd+DlkUQEAA
AAwAAIABwAAAAEAAAABAQAAAAAAEgEAAAAAAAAAEAAAANNBR2M9XYa9xX8pNxAAAACF+5OW95HsbT
4mEPAAwxbGkJ7ZAQgAbgBhAHQAZQAAAAAAAAAuABgAJABIAAIAAAAIAHAAHAB4AAAAAABuAGEAdAB
lAEAAYwBvAHIAcAAuAHQAdQB4AHQAcgBpAGEAZwBlAC4AbgBlAHQAAABDAE8AUgBQAC4AVABVAFgA
VABSAEkAQQBHAEUALgBOAEUAVAAAAAAAbgBhAHQAZQABBQAAAAAABRUAAADq0TuNqPiB3fg5ZFFVB
AAAAAAAABAAAAAiqzSyLHV5O4/3XFsQAAAAMpv8w439cfo71DnY
pacBlobExpireTimestamp: 1686727561
ccacheFile: KCM:
cachedPassword: $6$.9jD7fiamZSK.KjF$IPjQgS76hMIMOjZ2K6/j3HFC7hies1Ezyiuur85cMU
ZjKSntkCgNcxydyL86uqhpRJk9k7fTfMnkkqcs9xEKb/
cachedPasswordType: 1
lastCachedPasswordChange: 1686727261
failedLoginAttempts: 0
lastOnlineAuth: 1686727261
lastOnlineAuthWithCurrentToken: 1686727261
lastLogin: 1686727261
distinguishedName: name=nate@corp.tuxtriage.net,cn=users,cn=corp.tuxtriage.net
,cn=sysdb
...
# returned 14 records
# 14 entries
# 0 referrals
From the above results, we can locate domain users that have logged on by searching for entries with pacBlob value set. Once the records have been located, various attributes about the user activity can be determined.
User Name: This can be located at name as well as nameAlias
objectSIDString: The windows SID for this user. This is useful to locate activity of this user across other windows systems.
createTimestamp: The create time for this user
lastLogin: Last logon timestamp for the user.
Key-users 2
The /proc/key-users lists the users with at least one key in the system. It is useful to check this file to see if there are user-ids in this file that were not found anywhere else.
A user logging off from the system does not automatically clear this file. However this is a cache file and therefore the data will be cleared at some point.
[tlasso@rhel-richmondfc proc]$ cat /proc/key-users
0: 113 112/112 79/1000000 1591/25000000
42: 4 4/4 4/200 34/20000
997: 1 1/1 1/200 9/20000
1000: 4 4/4 4/200 46/20000
357201109: 2 2/2 2/200 38/20000
357201111: 4 4/4 4/200 56/20000
In this example, user with id 357201111 had logged off the ssh session.