sysinfo Provider - Oracle Solaris 11.1 Dynamic Tracing Guide

`sysinfo` Provider

The sysinfo provider makes available probes that correspond to kernel statistics classified by the name sys. Because these statistics provide the input for system monitoring utilities like mpstat(1M), the sysinfo provider enables quick exploration of observed aberrant behavior.

Probes

The sysinfo provider makes available probes that correspond to the fields in the sys named kernel statistic: a probe provided by sysinfo fires immediately before the corresponding sys value is incremented. The following example shows how to display both the names and the current values of the sys named kernel statistic using the kstat(1M) command.

$ kstat -n sys
module: cpu                             instance: 0
name:   sys                             class:    misc
        bawrite                         123
        bread                           2899
        bwrite                          17995
        cpu_ticks_idle                  73743866
        cpu_ticks_kernel                2096277
        cpu_ticks_user                  1010122
        cpu_ticks_wait                  46413
...

The sysinfo probes are described in the following table.

`bawrite`	Probe that fires whenever a buffer is about to be asynchronously written out to a device.
`bread`	Probe that fires whenever a buffer is physically read from a device. bread fires after the buffer has been requested from the device, but before blocking pending its completion.
`bwrite`	Probe that fires whenever a buffer is about to be written out to a device, whether synchronously or asynchronously.
`cpu_ticks_idle`	Probe that fires when the periodic system clock has made the determination that a CPU is idle. Note that this probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The `cpu_t` argument (`arg2`) indicates the CPU that has been deemed idle. See Arguments for details.
`cpu_ticks_kernel`	Probe that fires when the periodic system clock has made the determination that a CPU is executing in the kernel. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The `cpu_t` argument (`arg2`) indicates the CPU that has been deemed to be executing in the kernel. See Arguments for details.
`cpu_ticks_user`	Probe that fires when the periodic system clock has made the determination that a CPU is executing in user mode. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The `cpu_t` argument (`arg2`) indicates the CPU that has been deemed to be running in user-mode. See Arguments for details.
`cpu_ticks_wait`	Probe that fires when the periodic system clock has made the determination that a CPU is otherwise idle, but some threads are waiting for I/O on the CPU. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The `cpu_t` argument (`arg2`) indicates the CPU that has been deemed waiting on I/O. See Arguments for details.
`idlethread`	Probe that fires whenever a CPU enters the idle loop.
`intrblk`	Probe that fires whenever an interrupt thread blocks.
`inv_swtch`	Probe that fires whenever a running thread is forced to involuntarily give up the CPU.
`lread`	Probe that fires whenever a buffer is logically read from a device.
`lwrite`	Probe that fires whenever a buffer is logically written to a device.
`modload`	Probe that fires whenever a kernel module is loaded.
`modunload`	Probe that fires whenever a kernel module is unloaded.
`msg`	Probe that fires whenever a msgsnd(2) or msgrcv(2) system call is made, but before the message queue operations have been performed.
`mutex_adenters`	Probe that fires whenever an attempt is made to acquire an owned adaptive lock. If this probe fires, one of the `lockstat` provider's `adaptive-block` or `adaptive-spin` probes will also fire. See `lockstat` Provider for details.
`namei`	Probe that fires whenever a name lookup is attempted in the filesystem.
`nthreads`	Probe that fires whenever a thread is created.
`phread`	Probe that fires whenever a raw I/O read is about to be performed.
`phwrite`	Probe that fires whenever a raw I/O write is about to be performed.
`procovf`	Probe that fires whenever a new process cannot be created because the system is out of process table entries.
`pswitch`	Probe that fires whenever a CPU switches from executing one thread to executing another.
`readch`	Probe that fires after each successful read, but before control is returned to the thread performing the read. A read may occur through the read(2), readv(2) or pread(2) system calls. `arg0` contains the number of bytes that were successfully read.
`rw_rdfails`	Probe that fires whenever an attempt is made to read-lock a readers/writer when the lock is either held by a writer, or desired by a writer. If this probe fires, the `lockstat` provider's `rw-block` probe will also fire. See `lockstat` Provider for details.
`rw_wrfails`	Probe that fires whenever an attempt is made to write-lock a readers/writer lock when the lock is held either by some number of readers or by another writer. If this probe fires, the `lockstat` provider's `rw-block` probe will also fire. See `lockstat` Provider for details.
`sema`	Probe that fires whenever a semop(2) system call is made, but before any semaphore operations have been performed.
`sysexec`	Probe that fires whenever an exec(2) system call is made.
`sysfork`	Probe that fires whenever a fork(2) system call is made.
`sysread`	Probe that fires whenever a read(2), readv(2), or pread(2) system call is made.
`sysvfork`	Probe that fires whenever a vfork(2) system call is made.
`syswrite`	Probe that fires whenever a write(2), writev(2), or pwrite(2) system call is made.
`trap`	Probe that fires whenever a processor trap occurs. Note that some processors, in particular UltraSPARC variants, handle some light-weight traps through a mechanism that does not cause this probe to fire.
`ufsdirblk`	Probe that fires whenever a directory block is read from the UFS file system. See ufs(7FS) for details on UFS.
`ufsiget`	Probe that fires whenever an inode is retrieved. See ufs(7FS) for details on UFS.
`ufsinopage`.	Probe that fires after an in-core inode without any associated data pages has been made available for reuse. See ufs(7FS) for details on UFS.
`ufsipage`	Probe that fires after an in-core inode with associated data pages has been made available for reuse. This probe fires after the associated data pages have been flushed to disk. See ufs(7FS) for details on UFS.
`wait_ticks_io`	Probe that fires when the periodic system clock has made the determination that a CPU is otherwise idle but some threads are waiting for I/O on the CPU. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The `cpu_t` argument (`arg2`) indicates the CPU that is described as waiting for I/O. See Arguments for details on `arg2`. No semantic difference between `wait_ticks_io` and `cpu_ticks_wait`; `wait_ticks_io` exists solely for historical reasons.
`writech`	Probe that fires after each successful write, but before control is returned to the thread performing the write. A write may occur through the write(2), writev(2) or pwrite(2) system calls. `arg0` contains the number of bytes that were successfully written
`xcalls`	Probe that fires whenever a cross-call is about to be made. A cross-call is the operating system's mechanism for one CPU to request immediate work of another CPU.

Arguments

The arguments to sysinfo probes are as follows:

`arg0`	The value by which the statistic is to be incremented. For most probes, this argument is always 1, but for some probes this argument may take other values.
`arg1`	A pointer to the current value of the statistic to be incremented. This value is a 64–bit quantity that will be incremented by the value in arg0. Dereferencing this pointer enables consumers to determine the current count of the statistic corresponding to the probe.
`arg2`	A pointer to the `cpu_t` structure that corresponds to the CPU on which the statistic is to be incremented. This structure is defined in `<sys/cpuvar.h>`, but it is part of the kernel implementation and should be considered Private.

The value of arg0 is 1 for most sysinfo probes. However, the readch and writech probes set arg0 to the number of bytes read or written, respectively. This features permits you to determine the size of reads by executable name, as shown in the following example:

# dtrace -n readch'{@[execname] = quantize(arg0)}'
dtrace: description 'readch' matched 4 probes
^C
  xclock                                            
           value  ------------- Distribution ------------- count    
              16 |                                         0        
              32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
              64 |                                         0        

  acroread                                          
           value  ------------- Distribution ------------- count    
              16 |                                         0        
              32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3        
              64 |                                         0        

  FvwmAuto                                          
           value  ------------- Distribution ------------- count    
               2 |                                         0        
               4 |@@@@@@@@@@@@@                            13       
               8 |@@@@@@@@@@@@@@@@@@@@@                    21       
              16 |@@@@@                                    5        
              32 |                                         0        

  xterm                                             
           value  ------------- Distribution ------------- count    
              16 |                                         0        
              32 |@@@@@@@@@@@@@@@@@@@@@@@@                 19       
              64 |@@@@@@@@@                                7        
             128 |@@@@@@                                   5        
             256 |                                         0        

  fvwm2                                             
           value  ------------- Distribution ------------- count    
              -1 |                                         0        
               0 |@@@@@@@@@                                186      
               1 |                                         0        
               2 |                                         0        
               4 |@@                                       51       
               8 |                                         17       
              16 |                                         0        
              32 |@@@@@@@@@@@@@@@@@@@@@@@@@@               503      
              64 |                                         9        
             128 |                                         0        

  Xsun                                              
           value  ------------- Distribution ------------- count    
              -1 |                                         0        
               0 |@@@@@@@@@@@                              269      
               1 |                                         0        
               2 |                                         0        
               4 |                                         2        
               8 |@                                        31       
              16 |@@@@@                                    128      
              32 |@@@@@@@                                  171      
              64 |@                                        33       
             128 |@@@                                      85       
             256 |@                                        24       
             512 |                                         8        
            1024 |                                         21       
            2048 |@                                        26       
            4096 |                                         21       
            8192 |@@@@                                     94       
           16384 |                                         0

The sysinfo provider sets arg2 to be a pointer to a cpu_t, a structure internal to the kernel implementation. Most sysinfo probes fire on the CPU on which the statistic is being incremented, but some probes do not. The exceptional probes include cpu_ticks_idle, cpu_ticks_kernel, cpu_ticks_user and cpu_ticks_wait, which always fire on the CPU executing the system clock. Use the cpu_id member of the cpu_t structure to determine the CPU of interest. The following D script runs for about ten seconds and gives a quick snapshot of relative CPU behavior on a statistic-by-statistic basis:

cpu_ticks_*
{
        @[probename] = lquantize(((cpu_t *)arg2)->cpu_id, 0, 1024, 1);
}

tick-1sec
/x++ >= 10/
{
        exit(0);
}

Running the above script results in output similar to the following example:

# dtrace -s ./tick.d
dtrace: script './tick.d' matched 5 probes
CPU     ID                    FUNCTION:NAME
 22  37588                       :tick-1sec 

  cpu_ticks_user                                    
           value  ------------- Distribution ------------- count    
              11 |                                         0        
              12 |@@@@@@@@                                 14       
              13 |@@@@                                     7        
              14 |@                                        3        
              15 |@                                        2        
              16 |@@                                       4        
              17 |@@@@@@                                   10       
              18 |                                         0        
              19 |@                                        2        
              20 |@@@                                      6        
              21 |@@@                                      5        
              22 |                                         1        
              23 |@@@@@@                                   10       
              24 |                                         0        

  cpu_ticks_wait                                    
           value  ------------- Distribution ------------- count    
              11 |                                         0        
              12 |@@@@@@@@@@@@@                            241      
              13 |@@@@@@@@@@@@@                            236      
              14 |                                         16       
              15 |@@@@@@@                                  132      
              16 |                                         11       
              17 |                                         10       
              18 |                                         7        
              19 |@                                        18       
              20 |                                         4        
              21 |                                         16       
              22 |                                         13       
              23 |                                         10       
              24 |                                         0        

  cpu_ticks_kernel                                  
           value  ------------- Distribution ------------- count    
              11 |                                         0        
              12 |@@@@@@@@                                 234      
              13 |@@@@@                                    159      
              14 |@@@                                      104      
              15 |@@@@                                     131      
              16 |@@                                       66       
              17 |@                                        40       
              18 |@                                        51       
              19 |@                                        36       
              20 |@@                                       56       
              21 |@                                        42       
              22 |@@@                                      96       
              23 |@@                                       57       
              24 |                                         0        

  cpu_ticks_idle                                    
           value  ------------- Distribution ------------- count    
              11 |                                         0        
              12 |@@                                       534      
              13 |@@                                       621      
              14 |@@@                                      900      
              15 |@@                                       758      
              16 |@@@                                      942      
              17 |@@@                                      963      
              18 |@@@                                      965      
              19 |@@@                                      967      
              20 |@@@                                      957      
              21 |@@@                                      960      
              22 |@@@                                      913      
              23 |@@@                                      946      
              24 |                                         0

Example

Examine the following output from mpstat(1M):

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
 12   90  22 5760   422  299  435   26   71  116   11  1372    5  19  17  60
 13   46  18 4585   193  162  431   25   69  117   12  1039    3  17  14  66
 14   33  13 3186   405  381  397   21   58  105   10   770    2  17  11  70
 15   34  19 4769   109   78  417   23   57  115   13   962    3  14  14  69
 16   74  16 4421   437  406  448   29   77  111    8  1020    4  23  14  59
 17   51  15 4493   139  110  378   23   62  109    9   928    4  18  14  65
 18   41  14 4204   494  468  360   23   56  102    9   849    4  17  12  68
 19   37  14 4229   115   87  363   22   50  106   10   845    3  15  14  67
 20   78  17 5170   200  169  456   26   69  108    9  1119    5  21  25  49
 21   53  16 4817    78   51  394   22   56  106    9   978    4  17  22  57
 22   32  13 3474   486  463  347   22   48  106    9   769    3  17  17  63
 23   43  15 4572    59   34  361   21   46  102   10   947    4  15  22  59

From the above output, you might conclude that the xcal field seems too high, especially given the relative idleness of the system. mpstat determines the value in the xcal field by examining the xcalls field of thesys kernel statistic. This aberration can therefore be explored easily by enabling the xcalls sysinfo probe, as shown in the following example:

# dtrace -n xcalls'{@[execname] = count()}'
dtrace: description 'xcalls' matched 4 probes
^C
  dtterm                                                            1
  nsrd                                                              1
  in.mpathd                                                         2
  top                                                               3
  lockd                                                             4
  java_vm                                                          10
  ksh                                                              19
  iCald.pl6+RPATH                                                  28
  nwadmin                                                          30
  fsflush                                                          34
  nsrindexd                                                        45
  in.rlogind                                                       56
  in.routed                                                       100
  dtrace                                                          153
  rpc.rstatd                                                      246
  imapd                                                           377
  sched                                                           431
  nfsd                                                           1227
  find                                                           3767

The output shows where to look for the source of the cross-calls. Some number of find(1) processes are causing the majority of the cross-calls. The following D script can be used to understand the problem in further detail:

syscall:::entry
/execname == "find"/
{
        self->syscall = probefunc;
        self->insys = 1;
}

sysinfo:::xcalls
/execname == "find"/
{
        @[self->insys ? self->syscall : "<none>"] = count();
}

syscall:::return
/self->insys/
{
        self->insys = 0;
        self->syscall = NULL;
}

This script uses the syscall provider to attribute cross-calls from find to a particular system call. Some cross-calls, such as those resulting from page faults, might not emanate from system calls. The script prints “<none>” in these cases. Running the script results in output similar to the following example:

# dtrace -s ./find.d
 dtrace: script './find.d' matched 444 probes
^C
  <none>                                                            2
  lstat64                                                        2433
  getdents64                                                    14873

This output indicates that the majority of cross-calls induced by find are in turn induced by getdents(2) system calls. Further exploration would depend on the direction you want to explore. If you want to understand why find processes are making calls to getdents, you could write a D script to aggregate on ustack when find induces a cross-call. If you want to understand why calls to getdents are inducing cross-calls, you could write a D script to aggregate on stack when find induces a cross-call. Whatever your next step, the presence of the xcalls probe has enabled you to quickly discover the root cause of the unusual monitoring output.

Stability

The sysinfo provider uses DTrace's stability mechanism to describe its stabilities, as shown in the following table. For more information about the stability mechanism, see Chapter 18, Stability.

Element	Name stability	Data stability	Dependency class
Provider	Evolving	Evolving	ISA
Module	Private	Private	Unknown
Function	Private	Private	Unknown
Name	Evolving	Evolving	ISA
Arguments	Private	Private	ISA

Skip Navigation Links
Exit Print View
	Oracle Solaris 11.1 Dynamic Tracing Guide Oracle Solaris 11.1 Information Library