Troubleshooting the Audit Service (Tasks)

This section covers various auditing error messages, preferences, and the auditing that is provided by other tools. These procedures can help you record required audit events and debug audit problems.

Troubleshooting the Audit Service (Task Map)

The following task map points to procedures for troubleshooting auditing.

How to Determine That Auditing Is Running

Auditing is enabled by default. If you believe that auditing has not been disabled, but no audit records are being sent to the active plugin, use the following procedure to isolate the issue.

Before You Begin

To modify a system file, you must be assigned the solaris.admin.edit/path-to-system-file authorization. By default, the root role has this authorization. To configure auditing, you must become an administrator who is assigned the Audit Configuration rights profile.

1. Determine that auditing is running.

   Use any of the following methods:

   • Verify the current audit condition.

     The following listing indicates that auditing is not running:

     # auditconfig -getcond
     audit condition = noaudit

     The following listing indicates that auditing is running:

     # auditconfig -getcond
     audit condition = auditing

   • Verify that the audit service is running.

     The following listing indicates that auditing is not running:

     # svcs -x auditd
     svc:/system/auditd:default (Solaris audit daemon)
      State: disabled since Sun Oct 10 10:10:10 2010
     Reason: Disabled by an administrator.
        See: http://support.oracle.com/msg/SMF-8000-05
        See: auditd(1M)
        See: audit(1M)
        See: auditconfig(1M)
        See: audit_flags(5)
        See: audit_binfile(5)
        See: audit_syslog(5)
        See: audit_remote(5)
        See: /var/svc/log/system-auditd:default.log
     Impact: This service is not running.

     The following listing indicates that the audit service is running:

     # svcs auditd
     STATE          STIME    FMRI
     online         10:10:10 svc:/system/auditd:default

   If the audit service is not running, enable it. For the procedure, see How to Enable the Audit Service.

2. Verify that at least one plugin is active.

   # audit -v
   audit: no active plugin found

   If no plugin is active, make one active.

   # auditconfig -setplugin audit_binfile active
   # audit -v
   configuration ok

3. If you created a customized audit class, verify that you assigned events to the class.

   For example, the following list of flags contains the pf class, which Oracle Solaris software did not deliver:

   # auditconfig -getflags
   active user default audit flags = pf,lo(0x0100000000000000,00x0100000000001000)
   configured user default audit flags = pf,lo(0x0100000000000000,00x0100000000001000)

   For a description of creating the pf class, see How to Add an Audit Class.

   1. Verify that the class is defined in the audit_class file.

      The audit class must be defined, and its mask must be unique.

      # grep pf /etc/security/audit_classVerify class exists
      0x0100000000000000:pf:profile
      # grep 0x0100000000000000 /etc/security/audit_classEnsure mask is unique
      0x0100000000000000:pf:profile

      Replace a mask that is not unique. If the class is not defined, define it. Otherwise, run the auditconfig -setflags command with valid values to reset the current flags.

   2. Verify that events have been assigned to the class.

      Use one of the following methods:

      # auditconfig -lsevent | egrep " pf|,pf|pf,"
      AUE_PFEXEC      116 pf execve(2) with pfexec enabled

      # auditrecord -c pf
      List of audit events assigned to pf class

      If events are not assigned to the class, assign the appropriate events to this class.

4. If the previous steps did not indicate a problem, review your email and the log files.
   1. Read the email sent to the audit_warn alias.

      The audit_warn script sends alert messages to the audit_warn email alias. In the absence of a correctly configured alias, the messages are sent to the root account.

   2. Review the log files for the audit service.

      The output from the svcs -s auditd command lists the full path to the audit logs that the audit service produces. For an example, see the listing in Step 1.

   3. Review the system log files.

      The audit_warn script writes daemon.alert messages to the /var/log/syslog file.

      The /var/adm/messages file might contain information.

5. After you locate and fix the problems, enable or restart the audit service.

   # audit -s

How to Lessen the Volume of Audit Records That Are Produced

After you have determined which events must be audited at your site, use the following suggestions to create audit files with just the information that you require.

Before You Begin

To preselect audit classes and set audit policy, you must be assigned the Audit Configuration rights profile. To modify a system file, you must be assigned the solaris.admin.edit/path-to-system-file authorization. By default, the root role has this authorization. To assign audit flags to users, roles, and rights profiles, you must assume the root role.

1. Use the default audit policy.

   Specifically, avoid adding events and audit tokens to the audit trail. The following policies grow the size of the audit trail.

   • arge policy – Adds environment variables to execv audit events. While auditing execv events can be costly, adding variables to the audit record is not costly.

   • argv policy – Adds command parameters to execv audit events. While auditing execv events can be costly, adding command parameters to the audit record is not costly.

   • public policy – If file events are being audited, adds an event to the audit trail every time an auditable event happens to a public object. File classes include fa, fc, fd, fm, fr, fw, and cl. For the definition of a public file, see Audit Terminology and Concepts.

   • path policy – Adds a path token to audit events that include an optional path token.

   • group policy – Adds a group token to audit events that include an optional newgroups token.

   • seq policy – Adds a sequence token to every audit event.

   • trail policy – Adds a trailer token to every audit event.

   • windata_down policy – On a system that is configured with Trusted Extensions, adds events when information in a labeled window is downgraded.

   • windata_up policy – On a system that is configured with Trusted Extensions, adds events when information in a labeled window is upgraded.

   • zonename policy – Adds the zone name to every audit event. If the global zone is the only configured zone, adds the string zone, global to every audit event.

   The following audit record shows the use of the ls command. The ex class is being audited and the default policy is in use:

   header,129,2,AUE_EXECVE,,mach1,2010-10-14 11:39:22.480 -07:00
   path,/usr/bin/ls
   attribute,100555,root,bin,21,320271,18446744073709551615
   subject,jdoe,root,root,root,root,2404,50036632,82 0 mach1
   return,success,0

   The following is the same record when all policies are turned on:

   header,1578,2,AUE_EXECVE,,mach1,2010-10-14 11:45:46.658 -07:00
   path,/usr/bin/ls
   attribute,100555,root,bin,21,320271,18446744073709551615
   exec_args,2,ls,/etc/security
   exec_env,49,MANPATH=/usr/share/man,USER=jdoe,GDM_KEYBOARD_LAYOUT=us,EDITOR=gedit,
     LANG=en_US.UTF-8,GDM_LANG=en_US.UTF-8,PS1=#,GDMSESSION=gnome,SESSIONTYPE=1,SHLVL=2,
     HOME=/home/jdoe,LOGNAME=jdoe,G_FILENAME_ENCODING=@locale,UTF-8, PRINTER=example-dbl,
   ...
   path,/lib/ld.so.1
   attribute,100755,root,bin,21,393073,18446744073709551615
   subject,jdoe,root,root,root,root,2424,50036632,82 0 mach1
   group,root,other,bin,sys,adm,uucp,mail,tty,lp,nuucp,daemon
   return,success,0
   zone,global
   sequence,197
   trailer,1578

2. Use the audit_syslog plugin to send some audit events to syslog.

   And do not send those audit events to the audit_binfile or audit_remote plugin. This strategy works only if you are not required to keep binary records of the audit events that you send to the syslog logs.

3. Set fewer system-wide audit flags and audit individual users.

   Reduce the amount of auditing for all users by reducing the number of audit classes that are audited system-wide.

   Use the audit_flags keyword to the roleadd, rolemod, useradd, and usermod commands to audit events for specific users and roles. For examples, see Example 28-21 and the usermod(1M) man page.

   Use the always_audit and never_audit properties of the profiles command to audit events for specific rights profiles. For information, see the profiles(1) man page.

   ---

   Note - Like other security attributes, audit flags are affected by search order. For more information, see Order of Search for Assigned Security Attributes.

   ---

4. Create your own customized audit class.

   You can create audit classes at your site. Into these classes, put only those audit events that you need to monitor. For the procedure, see How to Add an Audit Class.

   ---

   Note - For information about the effects of modifying an audit configuration file, see Audit Configuration Files and Packaging.

   ---

How to Audit All Commands by Users

As part of site security policy, some sites require audit records of all commands that are run by the root account and administrative roles. Some sites can require audit records of all commands by all users. Additionally, sites can require that the command arguments and environment be recorded.

Before You Begin

To preselect audit classes and set audit policy, you must become an administrator who is assigned the Audit Configuration rights profile. To assign audit flags to users, roles, and rights profiles, you must assume the root role.

1. Audit the lo and ex classes.

   The ex class audits all calls to the exec() and execve() functions.

   The lo class audits logins, logouts, and screen locks. The following output lists all the events in the ex and lo classes.

   % auditconfig -lsevent | grep " lo "
   AUE_login                       6152 lo login - local
   AUE_logout                      6153 lo logout
   AUE_telnet                      6154 lo login - telnet
   AUE_rlogin                      6155 lo login - rlogin
   AUE_rshd                        6158 lo rsh access
   AUE_su                          6159 lo su
   AUE_rexecd                      6162 lo rexecd
   AUE_passwd                      6163 lo passwd
   AUE_rexd                        6164 lo rexd
   AUE_ftpd                        6165 lo ftp access
   AUE_ftpd_logout                 6171 lo ftp logout
   AUE_ssh                         6172 lo login - ssh
   AUE_role_login                  6173 lo role login
   AUE_newgrp_login                6212 lo newgrp login
   AUE_admin_authenticate          6213 lo admin login
   AUE_screenlock                  6221 lo screenlock - lock
   AUE_screenunlock                6222 lo screenlock - unlock
   AUE_zlogin                      6227 lo login - zlogin
   AUE_su_logout                   6228 lo su logout
   AUE_role_logout                 6229 lo role logout
   AUE_smbd_session                6244 lo smbd(1m) session setup
   AUE_smbd_logoff                 6245 lo smbd(1m) session logoff
   AUE_ClientConnect               9101 lo client connection to x server
   AUE_ClientDisconnect            9102 lo client disconn. from x server
   % auditconfig -lsevent | egrep " ex |,ex |ex,"
   AUE_EXECVE                        23 ex,ps execve(2)

   • To audit these classes for administrative roles, modify the roles' security attributes.

     In the following example, root is a role. The site has created three roles, sysadm, auditadm, and netadm. All roles are audited for the success and failure of events in the ex and lo classes.

     # rolemod -K audit_flags=lo,ex:no root
     # rolemod -K audit_flags=lo,ex:no sysadm
     # rolemod -K audit_flags=lo,ex:no auditadm
     # rolemod -K audit_flags=lo,ex:no netadm

   • To audit these classes for all users, set the system-wide flags.

     # auditconfig -setflags lo,ex

     The output appears similar to the following:

     header,129,2,AUE_EXECVE,,mach1,2010-10-14 12:17:12.616 -07:00
     path,/usr/bin/ls
     attribute,100555,root,bin,21,320271,18446744073709551615
     subject,jdoe,root,root,root,root,2486,50036632,82 0 mach1
     return,success,0

2. To record the arguments to commands, add the argv policy.

   # auditconfig -setpolicy +argv

   The exec_args token records the command arguments:

   header,151,2,AUE_EXECVE,,mach1,2010-10-14 12:26:17.373 -07:00
   path,/usr/bin/ls
   attribute,100555,root,bin,21,320271,18446744073709551615
   exec_args,2,ls,/etc/security
   subject,jdoe,root,root,root,root,2494,50036632,82 0 mach1
   return,success,0

3. To record the environment in which the command is run, add the arge policy.

   # auditconfig -setpolicy +arge

   The exec_env token records the command environment:

   header,1460,2,AUE_EXECVE,,mach1,2010-10-14 12:29:39.679 -07:00
   path,/usr/bin/ls
   attribute,100555,root,bin,21,320271,18446744073709551615
   exec_args,2,ls,/etc/security
   exec_env,49,MANPATH=/usr/share/man,USER=jdoe,GDM_KEYBOARD_LAYOUT=us,EDITOR=gedit,
   LANG=en_US.UTF-8,GDM_LANG=en_US.UTF-8,PS1=#,GDMSESSION=gnome,SESSIONTYPE=1,SHLVL=2,
   HOME=/home/jdoe,LOGNAME=jdoe,G_FILENAME_ENCODING=@locale,UTF-8,
   PRINTER=example-dbl,...,_=/usr/bin/ls
   subject,jdoe,root,root,root,root,2502,50036632,82 0 mach1
   return,success,0

How to Find Audit Records of Changes to Specific Files

If your goal is to log file writes against a limited number of files, such as /etc/passwd and the files in the /etc/default directory, you can use the auditreduce command to locate the files.

Before You Begin

The root role can perform every task in this procedure.

If administrative rights are distributed in your organization, consider the following:

• An administrator with the Audit Configuration rights profile can run the auditconfig command.

• An administrator with the Audit Review rights profile can run the auditreduce command.

• Only the root role can assign audit flags.

For more information, see How to Use Your Assigned Administrative Rights.

1. Audit the fw class.

   Adding the class to the audit flags of a user or role generates fewer records than adding the class to the system-wide audit preselection mask. Perform one of the following steps:

   • Add the fw class to specific roles.

     # rolemod -K audit_flags=fw:no root
     # rolemod -K audit_flags=fw:no sysadm
     # rolemod -K audit_flags=fw:no auditadm
     # rolemod -K audit_flags=fw:no netadm

   • Add the fw class to the system-wide flags.

     # auditconfig -getflags
     active user default audit flags = lo(0x1000,0x1000)
     configured user default audit flags = lo(0x1000,0x1000)
     # auditconfig -setflags lo,fw
     user default audit flags = lo,fw(0x1002,0x1002)

2. Or, audit successful file-writes.

   Auditing successes generates fewer records than auditing failures and successes. Perform one of the following steps:

   • Add the +fw flag to specific roles.

     # rolemod -K audit_flags=+fw:no root
     # rolemod -K audit_flags=+fw:no sysadm
     # rolemod -K audit_flags=+fw:no auditadm
     # rolemod -K audit_flags=+fw:no netadm

   • Add the +fw flag to the system-wide flags.

     # auditconfig -getflags
     active user default audit flags = lo(0x1000,0x1000)
     configured user default audit flags = lo(0x1000,0x1000)
     # auditconfig -setflags lo,+fw
     user default audit flags = lo,+fw(0x1002,0x1000)

   • If the system-wide flags are auditing for success and for failure, audit successes only for specific users and roles.

     # auditconfig -getflags
     active user default audit flags = lo,fw(0x1002,0x1002)
     configured user default audit flags = lo,fw(0x1002,0x1002)
     # rolemod -K audit_flags=^-fw:no root
     # rolemod -K audit_flags=^-fw:no sysadm
     # rolemod -K audit_flags=^-fw:no auditadm
     # rolemod -K audit_flags=^-fw:no netadm

     The system-wide flags are still unchanged, but the preselection mask for these four roles is changed.

     # auditconfig -getflags
     active user default audit flags = lo,fw(0x1002,0x1000)
     configured user default audit flags = lo,fw(0x1002,0x1000)

3. To find the audit records for specific files, use the auditreduce command.

   # auditreduce -o file=/etc/passwd,/etc/default -O filechg

   The auditreduce command searches the audit trail for all instances of the file argument. The command creates a binary file with the suffix filechg which contains all records that include the pathnames of the files of interest. See the auditreduce(1M) man page for the syntax of the -o file=pathname option.

4. To read the filechg file, use the praudit command.

   # praudit *filechg

How to Update the Preselection Mask of Logged In Users

You want the users who are already logged in to be audited for changes to the system-wide audit preselection mask.

Before You Begin

You must become an administrator who is assigned the Audit Configuration rights profile. To terminate user sessions, you must become an administrator who is assigned the Process Management rights profile. For more information, see How to Use Your Assigned Administrative Rights.

• Update the preselection mask of users who are already logged in.

  You have two options. You can terminate the existing sessions or use the auditconfig command to update the preselection masks.

  • Terminate the users' existing sessions.

    Users can log out and log back in. Or, you in a role that is assigned the Process Management rights profile can manually terminate (kill) active sessions. The new sessions will inherit the new preselection mask. However, terminating users could be impractical.

  • Dynamically change each logged-in user's preselection mask.

    In this example, assume that you changed the system-wide audit preselection mask from lo to lo,ex.

    # auditconfig -setflags lo,ex

    1. List the regular users who are logged in and their process IDs.

       # who -a
       jdoe  - vt/2         Jan 25 07:56  4:10   1597   (:0)
       jdoe  + pts/1        Jan 25 10:10   .     1706   (:0.0)
       ...
       jdoe  + pts/2        Jan 25 11:36  3:41   1706   (:0.0)

    2. For later comparison, display each user's preselection mask.

       # auditconfig -getpinfo 1706
       audit id = jdoe(1234)
       process preselection mask = lo(0x1000,0x1000)
       terminal id (maj,min,host) = 9426,65559,mach1(192.168.123.234)
       audit session id = 103203403

    3. Modify the appropriate preselection mask by running one or more of the following commands:

       # auditconfig -setpmask 1706 lo,ex /* for this process */ 
       # auditconfig -setumask jdoe lo,ex /* for this user */ 
       # auditconfig -setsmask 103203403 lo,ex /* for this session */

    4. Verify that the preselection mask for the user has changed.

       For example, check a process that existed before you changed the mask.

       # auditconfig -getpinfo 1706
       audit id = jdoe(1234)
       process preselection mask = ex,lo(0x40001000,0x40001000) 
       terminal id (maj,min,host) = 9426,65559,mach1(192.168.123.234)
       audit session id = 103203403

How to Prevent the Auditing of Specific Events

For maintenance purposes, sometimes a site wants to prevent events from being audited.

Before You Begin

You must assume the root role. For more information, see How to Use Your Assigned Administrative Rights.

1. Change the class of the event to the no class.

   ---

   Note - For information about the effects of modifying an audit configuration file, see Audit Configuration Files and Packaging.

   ---

   For example, events 26 and 27 belong to the pm class.

   ## audit_event file
   ...
   25:AUE_VFORK:vfork(2):ps
   26:AUE_SETGROUPS:setgroups(2):pm
   27:AUE_SETPGRP:setpgrp(2):pm
   28:AUE_SWAPON:swapon(2):no
   ...

   Change these events to the no class.

   ## audit_event file
   ...
   25:AUE_VFORK:vfork(2):ps
   26:AUE_SETGROUPS:setgroups(2):no
   27:AUE_SETPGRP:setpgrp(2):no
   28:AUE_SWAPON:swapon(2):no
   ...

   If the pm class is currently being audited, existing sessions will still audit events 26 and 27. To stop these events from being audited, you must update the users' preselection masks by following the instructions in How to Update the Preselection Mask of Logged In Users.

   ---

   Caution - Never comment out events in the audit_event file. This file is used by the praudit command to read binary audit files. Archived audit files might contain events that are listed in the file.

   ---

2. Refresh the kernel events.

   # auditconfig -conf
   Configured 283 kernel events.

How to Limit the Size of Binary Audit Files

Binary audit files grow without limit. For ease of archiving and searching, you might want to limit the size. You can also create smaller binary files from the original file.

Before You Begin

You must become an administrator who is assigned the Audit Configuration rights profile to set the p_fsize attribute. You must become an administrator who is assigned the Audit Review rights profile to use the auditreduce command. For more information, see How to Use Your Assigned Administrative Rights.

1. Use the p_fsize attribute to limit the size of individual binary audit files.

   For a description of the p_fsize attribute, see the OBJECT ATTRIBUTES section of the audit_binfile(5) man page.

   For an example, see Example 28-14.

2. Use the auditreduce command to select records and write those records to a smaller file for further analysis.

   The auditreduce -lowercase options find specific records.

   The auditreduce -Uppercase options write your selections to a file. For more information, see the auditreduce(1M) man page. See also Managing Audit Records on Local Systems (Tasks).

How to Compress Audit Files on a Dedicated File System

Audit files can grow large. You can set an upper limit to the size of a file, as shown in Example 28-14. In this procedure, you use compression to reduce the size.

Before You Begin

You must become an administrator who is assigned the ZFS File System Management and ZFS Storage Management rights profiles. The latter profile enables you to create storage pools. For more information, see How to Use Your Assigned Administrative Rights.

1. Dedicate a ZFS file system for audit files.

   For the procedure, see How to Create ZFS File Systems for Audit Files.

2. Compress the ZFS storage pool by using one of the following options.

   With both options, the audit file system is compressed. After the audit service is refreshed, the compression ratio is displayed.

   To set compression, use the zfs set compression=on dataset command. In the following examples, the ZFS pool auditp/auditf is the dataset.

   • Use the default compression algorithm.

     # zfs set compression=on auditp/auditf
     # audit -s
     # zfs get compressratio auditp/auditf
     NAME           PROPERTY       VALUE  SOURCE
     auditp/auditf  compressratio  4.54x  -

   • Use a higher compression algorithm.

     # zfs set compression=gzip-9 auditp/auditf
     # zfs get compression auditp/auditf
     NAME           PROPERTY     VALUE     SOURCE
     auditp/auditf  compression  gzip-9    local
     # audit -s
     # zfs get compressratio auditp/auditf
     NAME           PROPERTY       VALUE  SOURCE
     auditp/auditf  compressratio  16.89x  -

     The gzip-9 compression algorithm results in files that occupy one-third less space than the default compression algorithm, lzjb. For more information, see Chapter 5, Managing Oracle Solaris ZFS File Systems, in Oracle Solaris 11.1 Administration: ZFS File Systems.

How to Audit Logins From Other Operating Systems

The Oracle Solaris OS can audit all logins, independent of source.

Before You Begin

You must become an administrator who is assigned the Audit Configuration rights profile. For more information, see How to Use Your Assigned Administrative Rights.

1. Audit the lo class for attributable events and non-attributable events.

   This class audits logins, logouts, and screen locks. These classes are audited by default.

   # auditconfig -getflags
   active user default audit flags = lo(0x1000,0x1000)
   configured user default audit flags = lo(0x1000,0x1000)
   # auditconfig -getnaflags
   active non-attributable audit flags = lo(0x1000,0x1000)
   configured non-attributable audit flags = lo(0x1000,0x1000)

2. If the values have been changed, add the lo flag.

   # auditconfig -getflags
   active user default audit flags = as,st(0x20800,0x20800)
   configured user default audit flags = as,st(0x20800,0x20800)
   # auditconfig -setflags lo,as,st
   user default audit flags = as,lo,st(0x21800,0x21800)
   # auditconfig -getnaflags
   active non-attributable audit flags = na(0x400,0x400)
   configured non-attributable audit flags = na(0x400,0x400)
   # auditconfig -setnaflags lo,na
   non-attributable audit flags = lo,na(0x1400,0x1400)

   ---

   Note - To audit ssh logins, your system must be running the ssh daemon from Oracle Solaris. This daemon is modified for the audit service on an Oracle Solaris system. For more information, see Secure Shell and the OpenSSH Project.

   ---

How to Audit FTP and SFTP File Transfers

The FTP service creates logs of its file transfers. The SFTP service, which runs under the ssh protocol, can be audited by preselecting the ft audit class. Logins to both services can be audited.

Before You Begin

You must become an administrator who is assigned the Audit Configuration rights profile. For more information, see How to Use Your Assigned Administrative Rights.

1. To log commands and file transfers of the FTP service, see the proftpd(8) man page.

   For the available logging options, read ProFTPD Logging.

2. To log sftp access and file transfers, audit the ft class.

   The ft class includes the following SFTP transactions:

   % auditrecord -c ft
   file transfer: chmod ...
   file transfer: chown ...
   file transfer: get ...
   file transfer: mkdir ...
   file transfer: put ...
   file transfer: remove ...
   file transfer: rename ...
   file transfer: rmdir ...
   file transfer: session start ...
   file transfer: session end ...
   file transfer: symlink ...
   file transfer: utimes

3. To record access to the FTP server, audit the lo class.

   As the following output indicates, logging in to and out of the proftpd daemon generates audit records.

   % auditrecord -c lo | more
   ...
   FTP server login
     program     proftpd              See in.ftpd(1M)
     event ID    6165                 AUE_ftpd
     class       lo                   (0x0000000000001000)
         header
         subject
         [text]                       error message
         return
   
   FTP server logout
     program     proftpd              See in.ftpd(1M)
     event ID    6171                 AUE_ftpd_logout
     class       lo                   (0x0000000000001000)
         header
         subject
         return
   ...