Auditd - Linux auditing system for file contents auditing and monitoring

269ac00f765508cb74b25d006dbf5bb2

Written by Dennis Nind

Last published at: February 8th, 2019

Introduction


The auditd utility can be an extremely valuable tool for monitoring what's happening on your Linux server. It is designed to integrate with the kernel and monitor system calls that the system makes.


In this guide we will learn how to check if auditd is installed, install it if it is not, check to make sure the daemon is running, create a simple audit rule, and check the logs to see what our example rule detected.


Checking for, and installing, auditd


Your cPanel server should already have a couple audit-libs* packages installed, but it will not have the actual audit package that provides the daemon that we need.


A default installation should look like this:

Code:

#rpm -qa | grep audit
audit-libs-2.8.1-3.el7_5.1.x86_64
audit-libs-python-2.8.1-3.el7_5.1.x86_64

Since it is not installed on this server, let's install it:

Code:

# yum install audit
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
 * EA4: 208.74.121.35
 * cpanel-addons-production-feed: 208.74.121.35
 * epel: fedora-epel.mirror.lstn.net
 * remi-safe: repo1.dal.innoscale.net
Resolving Dependencies
--> Running transaction check
---> Package audit.x86_64 0:2.8.1-3.el7_5.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================
 Package                               Arch                                    Version                                          Repository                               Size
=============================================================================================================================================================================
Installing:
 audit                                 x86_64                                  2.8.1-3.el7_5.1                                  updates                                 247 k

Transaction Summary
=============================================================================================================================================================================
Install  1 Package

Total download size: 247 k
Installed size: 621 k
Is this ok [y/d/N]: y
Downloading packages:
audit-2.8.1-3.el7_5.1.x86_64.rpm                                                                                                                      | 247 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : audit-2.8.1-3.el7_5.1.x86_64                                                                                                                              1/1
  Verifying  : audit-2.8.1-3.el7_5.1.x86_64                                                                                                                              1/1

Installed:
  audit.x86_64 0:2.8.1-3.el7_5.1

Complete!

Now we can see the actual audit package:

Code:

# rpm -qa | grep audit
audit-libs-2.8.1-3.el7_5.1.x86_64
audit-2.8.1-3.el7_5.1.x86_64
audit-libs-python-2.8.1-3.el7_5.1.x86_64

This will install the daemon process and it will be enabled to start on server reboot, but it will not be running:

Code:

# systemctl status auditd.service
● auditd.service - Security Auditing Service
   Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:auditd(8)
           https://github.com/linux-audit/audit-documentation

So let's start it so it will be running when we create our rules:

Code:

# systemctl start auditd.service
# systemctl status auditd.service
● auditd.service - Security Auditing Service
   Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-10-17 08:34:59 CDT; 1s ago
     Docs: man:auditd(8)
           https://github.com/linux-audit/audit-documentation
  Process: 27739 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS)
  Process: 27729 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS)
 Main PID: 27731 (auditd)
   CGroup: /system.slice/auditd.service
           └─27731 /sbin/auditd

Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: enabled 1
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: failure 1
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: pid 27731
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: rate_limit 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog_limit 8192
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 1
Oct 17 08:34:59 <HOSTNAME> systemd[1]: Started Security Auditing Service.

Adding rules


Now that we have started the service, the 'auditctl' command can be used to list/create/remove rules:

Code:

# auditctl -l
No rules

So let's create our first rule, which takes the format of:

Code:

# auditctl -w PathToFile -p Permissions -k SearchKey

Let's add a rule to watch /etc/passwd for writes(w) or attribute changes(a) and set the search key to 'passwd' to make it easy to find the entries regarding the /etc/passwd file being changed:

Code:

# auditctl -w /etc/passwd -p wa -k passwd

Now we can see the rule with auditctl and within the audit.rules file:

Code:

# auditctl -l
-w /etc/passwd -p wa -k passwd

Note: This rule will exist as long as the server is not rebooted. If you wish to make the rule permanent, you need to add it to the /etc/audit/rules.d/audit.rules file. This is how it will look by default:

Code:

# cat /etc/audit/rules.d/audit.rules
## First rule - delete all
-D

## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 8192

## Set failure mode to syslog
-f 1

To add our example rule, simply add it to the bottom of the file in the exact same way that it appears when running 'auditctl -l':

Code:

# cat /etc/audit/rules.d/audit.rules
## First rule - delete all
-D

## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 8192

## Set failure mode to syslog
-f 1

-w /etc/passwd -p wa -k passwd

Searching audit logs


The 'ausearch' command is used to search the audit logs, so we'll use the following to search for the "passwd" search key:

Code:

# ausearch -k passwd
----
time->Wed Oct 17 08:45:23 2018
type=CONFIG_CHANGE msg=audit(1539783923.295:77): auid=0 ses=51297 op=add_rule key="passwd" list=4 res=1

That entry is showing the rule was added and it is currently the only entry since /etc/passwd hasn't been modified since we created this rule.


Now let's add a cPanel account to the server so we can see how that operation modifies /etc/passwd:

Code:

# ausearch -i -k passwd
----
type=CONFIG_CHANGE msg=audit(10/17/2018 08:45:23.295:77) : auid=root ses=51297 op=add_rule key=passwd list=exit res=yes
----
type=PROCTITLE msg=audit(10/17/2018 09:04:15.572:155) : proctitle=whostmgr5 - wwwacct
type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=1 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=CWD msg=audit(10/17/2018 09:04:15.572:155) :  cwd=/usr/local/cpanel/whostmgr/docroot
type=SYSCALL msg=audit(10/17/2018 09:04:15.572:155) : arch=x86_64 syscall=open success=yes exit=10 a0=0x4ffa230 a1=O_RDWR|O_CREAT a2=0644 a3=0x7f9899e7cedd items=2 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwd
----
type=CONFIG_CHANGE msg=audit(10/17/2018 09:04:15.573:156) : auid=unset ses=unset op=updated_rules path=/etc/passwd key=passwd list=exit res=yes
----
type=PROCTITLE msg=audit(10/17/2018 09:04:15.573:157) : proctitle=whostmgr5 - wwwacct
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=4 name=/etc/passwd inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=3 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=2 name=/etc/passwd.work.d21f6ce1 inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=1 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=CWD msg=audit(10/17/2018 09:04:15.573:157) :  cwd=/usr/local/cpanel/whostmgr/docroot
type=SYSCALL msg=audit(10/17/2018 09:04:15.573:157) : arch=x86_64 syscall=rename success=yes exit=0 a0=0x580a770 a1=0x6a583f a2=0x7f989a106980 a3=0x3 items=5 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwd

The "-i" will print the actual named system call, rather than the number, making it far easier to know what system calls are taking place. The entries we're most concerned with here are the second and last entries. Within those, specifically we're looking mostly at the "syscall" and "exe" values.


The second entry shows that /etc/passwd was opened using the "open" system call ("syscall=open") to the "whostmgr5" executable file ("exe=/usr/local/cpanel/whostmgr/bin/whostmgr5")


A system call is how a program interacts with the kernel, so these are important to look at because they tell us whether a program is requesting to, for example, open a file, write to a file, read from a file, etc.


The last entry then shows that there was a "rename" call also by the same WHM process, and looking a little above that, you can see that a temporary file had been created "name=/etc/passwd.work.d21f6ce1" then it was renamed to /etc/passwd, thus replacing the old file with the new one.


Knowing that this change was done by the WHM process, one could then check cPanel's access log (/usr/local/cpanel/logs/access_log) to find the IP address and username that made the request that caused this file to be written to:

Code:

10.7.65.81 - root [10/17/2018:14:03:52 -0000] "POST /cpsess3914673904/scripts5/wwwacct HTTP/1.1" 200 0 "https://<HOSTNAME>:2087/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "s" "-" 2087

(The time difference comes from the access_log being written to in UTC and the audit logs using the system's timezone which is set to US/Central time so there's a 5 hour difference)


So we now know that on the morning of October 17th, someone at 10.7.65.81 logged into WHM as root and made a POST request to wwwacct which modified the /etc/passwd file. In this specific instance, one could then reference the /var/cpanel/accounting.log file and see the account that was created:

Code:

Wed Oct 17 09:04:16 2018:CREATE:root:root:auditdexample.tld:172.16.0.93:auditdexample

Deleting rules


Lastly, once you've gathered whatever information you're after with your audit rule, you should delete this watch rule using the same format that was used to add it, except replacing -w with -W:

Code:

# auditctl -W /etc/passwd -p wa -k passwd
# auditctl -l
No rules

The above command will remove the rule from the current running audit session. You will also need to remove the entry from /etc/audit/rules.d/audit.rules if you added it there.


It is important to delete the rules once they've served their purpose, as audit logs can potentially grow very large in size if they're left to run for extended periods of time.