Friday, April 15, 2022

Lets Audit The Linux System Now!

Yes, the native "audit" is one of the best security features which is generally not used that much now-e-days. Well, if there is another well advanced security feature which does more than audit then it is good. In this blog post I wish to discuss and talk about this feature and how to get the best out of it without investing anything extra. All that is required is good administrative skills and what needs to be audited. Most importantly, this native audit feature could only track the security violations on a Linux box, however, it could not fix/resolve it. It is only an auditing system. Using this native feature one could track down any system events such as change of file attributes or shutdown events etc, and could understand which user has triggered it. Let's see how it works.


How to get started? 

This needs the audit package to be installed. On a standard/default installation mode, on most distributions this gets installed by default. Anyways, one could simply query the package using the native rpm command as shown below:

# rpm -q audit 

If this doesn't show any results then go ahead and install it by running the command "yum install audit" ( "dnf install audit" in case of RHEL8.x). 

Since audit runs as a daemon in the background, it is important to make sure that it is started and enabled. So, let's check out this now and activate/enable if not:

# systemctl is-active auditd || systemctl start auditd

# systemctl is-enabled auditd || systemctl enable auditd


Where are the configuration & logging files of auditd service stored? 

The main auditd configuration file →  /etc/audit/auditd.conf

Main Audit rules file →  /etc/audit/audit.rules 

Persistent audit rules are stored in the directory →  /etc/audit/rules.d/ 

Pre-configured rules files stored in this directory →  /usr/share/audit/sample-rules/ 

Log files are stored under the directory →  /var/log/audit (and gets rotated as defined by log rotation).

Documentation files related to auditd →  /usr/share/doc/audit/ 

In RHEL 8.x, the Audit dispatcher daemon (audisp) functionality is integrated in the Audit daemon (auditd). Configuration files of plugins for the interaction of real-time analytical programs with Audit events are located in the  /etc/audit/plugins.d/  directory by default.


How to perofrm auditing?

Given below are some of the examples to it:

[1] Audit all commands executed by root user.

In a strict environment where no command execution by root user is prohibited. However, there are still a few users who may be authorized to switch the identity as root and execute commands. So, in such cases, how to track the commands executed by root after a user who changes the identity? 

On the fly commands:

# auditctl -a exit,always -F arch=b64 -F euid=0 -S execve -k rootcmd 

# auditctl -a exit,always -F arch=b32 -F euid=0 -S execve -k rootcmd 


[2] Audit all system reboot commands executed.

Most of the time it becomes difficult to track which user has triggered the system reboot. Was the reboot made by any user or system thread? In order to understand and track such activities we could rely on audit daemon. By default, the reboot triggered by all non-root users would get tracked by auditd when enabled. We could track the system reboot messages using the 'ausearch' utility.


[3] Audit all writes or changes to the file /etc/fstab.

On the fly command:

# auditctl -w /etc/fstab -p wa -k fstab_change


[4] Audit every time a file is deleted or renamed by a system user whose ID is 1000 or larger:

# auditctl -a always,exit -S unlink -S unlinkat -S rename -S \

renameat -F auid>=1000 -F auid!=4294967295 -k deleted_renamed

*Note that the -F auid!=4294967295 option is used to exclude users whose login UID is not set.

Note: All these audit rules added are non-persistent, to make them persistent, need to add them to '/etc/audit/rules.d/audit.rules' file and restart the auditd daemon. The 'augenrules' command can also be used to create this list. 

The '/etc/audit/audit.rules' file is generated whenever the auditd service starts. Files in '/etc/audit/rules.d/' use the same auditctl command-line syntax to specify the rules. Empty lines and text following a hash sign (#) are ignored.


How to list out all added audit rules? 

Once a new rule is added to the '/etc/audit/rules.d/audit.rules' file then the auditd daemon needs to be restarted using service command, not systemctl command. Once the service is restarted, running the command "auditctl -l" would show all the newly added audit rules:


How to view audit logs or generate audit reports? 

This is achieved using "ausearch" & "aureport" commands. 

What is ausearch?

ausearch is a tool that can query the audit daemon logs for events based on different search criteria. The ausearch utility can also take input from stdin as long as the input is the raw log data. Each command line option given forms an "and" statement. For example, searching with -m and -ui means return events that have both the requested type and match the user id given. An exception is the -m  and -n options; multiple record types and nodes are allowed in a search which will return any matching node and record.


What is aureport? 

aureport is a tool that produces summary reports of the audit system logs. The aureport utility can also take input from stdin as long as the input is the raw log data. The reports have a column label at the  top to help with interpretation of the various fields. Except for the main summary report, all reports have the audit event number. You can subsequently look up the full event with ausearch -a event number. You may need to specify  start & stop times if you get multiple hits. The reports produced by aureport can be used as building blocks for more complicated analysis.

First, let's see what are the audit reports generated or gathered with keyword "fstab_changes" (one small snippet is added here for reference).

# ausearch -k fstab_changes 

time->Fri Apr 15 19:00:23 2022

type=PROCTITLE msg=audit(1650029423.025:871): proctitle=7669002F6574632F6673746162

type=PATH msg=audit(1650029423.025:871): item=0 name="/etc/fstab" inode=17004757 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0

type=CWD msg=audit(1650029423.025:871): cwd="/root"

type=SYSCALL msg=audit(1650029423.025:871): arch=c000003e syscall=188 success=yes exit=0 a0=561c050e5da0 a1=7fa1d71a41ef a2=561c05102f00 a3=1c items=1 ppid=9859 pid=9935 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=9 comm="vi" exe="/usr/bin/vi" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="fstab_changes"


In this above log output, some information is not human readable or simply can't be understood, so to make it easy, you need to use the "-i" option which converts such information into human readable. So, lets do that now and see how the ausearch output looks like now:


# ausearch -i -k fstab_changes 

type=PROCTITLE msg=audit(04/15/2022 19:00:23.025:871) : proctitle=vi /etc/fstab

type=PATH msg=audit(04/15/2022 19:00:23.025:871) : item=0 name=/etc/fstab inode=17004757 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0

type=CWD msg=audit(04/15/2022 19:00:23.025:871) : cwd=/root

type=SYSCALL msg=audit(04/15/2022 19:00:23.025:871) : arch=x86_64 syscall=setxattr success=yes exit=0 a0=0x561c050e5da0 a1=0x7fa1d71a41ef a2=0x561c05102f00 a3=0x1c items=1 ppid=9859 pid=9935 auid=redhat uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=9 comm=vi exe=/usr/bin/vi subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=fstab_changes


Notice the differences now, whatever that was displayed as non-readable become easy to understand especially the ones below:

msg=audit(04/15/2022 19:00:23.025:871)

proctitle=vi /etc/fstab

arch=x86_64 


Lets further break the output data and understand important fields/attributes of the audit message.


[1] type=PROCTITLE


The type field contains the type of the record. In this example, the PROCTITLE value specifies that this record gives the full command-line that triggered this Audit event, this is "proctitle=vi /etc/fstab".


[2] msg=audit(04/15/2022 19:00:23.025:871)


This indicates the time stamp of the audit event that gets recorded. This is in the format "time_stamp:ID". In this example, the unique ID is 871.


3- name=/etc/fstab inode=17004757 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0


This indicates unique attributes pertaining to the file '/etc/fstab', here which is inode, file permission mode which is 644 which can be checked using the 'stat' command as shown below:



4- cwd=/root

This denotes the Current Working Directory from where the command was executed. 


5- arch=x86_64

This indicates the system architecture which is x86_64 in this case.


6- success=yes exit=0

This field denotes where the command was successful or not. 


7- ppid=9859 pid=9935

As the name indicates, these are the Parent Process ID and Process ID.  Let's find out the details of the Parent Process which has called this process.



This is the process ID of the current bash shell which can also be verified using the "echo $$" command.


8- auid=redhat


This is the Audit UID of the process which had triggered this command. In this case, it says that is "redhat" which means that this is the user who has elevated the privileges into root and executed the command. This is indicated by these fields "uid=root gid=root euid=root". 


So, how is the system capable of tracking this? This is achieved using the file ‘/proc/self/loginuid’ which gets created every time a user logs into the system either via SSH or direct console login or whatever method. This file once gets created a user  logins and gets removed whenever the user logs out. If we look at the below data the contents of the file ‘/proc/self/loginuid’ is set once the user “redhat” logs into the system and doesn’t change or deleted until the user logs out. 



Finally, the “aureport” is another useful command to be used while auditing. Please look out for the man page of this command to find out more details. All the best!

No comments: