Getting Started: Creating a Sample Crash Dump

This section shows you how to obtain a sample crash dump, and how to invoke MDB in order to examine it.

Setting kmem_flags

The kernel memory allocator contains many advanced debugging features, but these are not enabled by default because they can cause performance degradation. In order to follow the examples in this guide, you should turn on these features. You should enable these features only on a test system, as they can cause performance degradation or expose latent problems.

The allocator's debugging functionality is controlled by the kmem_flags tunable. To get started, make sure kmem_flags is set properly:

# mdb -k
> kmem_flags/X
kmem_flags:
kmem_flags:     f

If kmem_flags is not set to f, you should add the following line to the /etc/system file:

set kmem_flags=0xf

The reboot the system. When the system reboots, confirm that kmem_flags is set to f. Remember to remove your /etc/system modifications before returning this system to production use.

Forcing a Crash Dump

The next step is to make sure crash dumps are properly configured. First, confirm that dumpadm is configured to save kernel crash dumps and that savecore is enabled. See dumpadm(1M) for more information on crash dump parameters.

# dumpadm
              Dump content: kernel pages
               Dump device: /dev/dsk/c0t0d0s1 (swap)
        Savecore directory: /var/crash/
          Savecore enabled: yes
           Save compressed: on

Next, reboot the system using the -d flag to reboot(1M), which forces the kernel to panic and save a crash dump.

# reboot -d
Sep 28 17:51:18 testsystem reboot: rebooted by root

panic[cpu0]/thread=70aacde0: forced crash dump initiated at user request

401fbb10 genunix:uadmin+55c (1, 1, 0, 6d700000, 5, 0)
  %l0-7: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
         00000000
...

When the system reboots, savecore runs automatically to preserve the crash dump in a file. When finished, a message is printed on the system console:

Sep 17 10:47:23 testsystem savecore: Decompress the crash dump with
Sep 17 10:47:23 testsystem 'savecore -vf /var/crash/vmdump.0'

If the message does not appear right away, check to whether savecore(1M) is still running:

$ pgrep savecore
864
$ cd /var/crash/
$ ls
bounds    vmdump.0

The vmdump.n file is a compressed version of vmcore.n plus unix.n.

If your dump directory contains no dump files , then that partition might be out of space. You can free up space and run savecore(1M) manually as root to subsequently save the dump.

If your dump directory contains multiple crash dumps, the one you just created is the unix.n and vmcore.n pair or vmdump.n file with the most recent modification time.

Saving a Crash Dump

When the system panics, or when you enter reboot -d, the following kinds of messages appear on the console:

Sep 17 10:47:23 testsystem savecore: Decompress the crash dump with 
Sep 17 10:47:23 testsystem 'savecore -vf /var/crash/vmdump.0'

Enter the following command:

root@testsystem # savecore -vf /var/crash/vmdump.0
savecore: System dump time: Thu Sep 17 10:43:20 2009

savecore: saving system crash dump in /var/crash/{unix,vmcore}.0
Constructing namelist /var/crash/unix.0
Constructing corefile /var/crash/vmcore.0
 1:29 100% done: 825215 of 825215 pages saved
1:30 dump decompress is done

Now you can use mdb:

root@testsystem# mdb /var/crash/{unix,vmcore}.0
Loading modules: [ unix genunix specfs dtrace zfs scsi_vhci sd mpt px mac ldc sockfs
ip hook neti sctp arp usba stmf qlc fctl nca lofs idm logindmux ptm ufs md cpc sppp
random smbsrv nfs crypto mdesc nsctl sdbc sv rdc fcp fcip ii nsmb ]
>

You can copy the vmdump.n file to another system for analysis. You can use savecore(1M) either locally or remotely to uncompress the dump file.

Use the dumpadm(1M) command to control the particular paths of the dump device and the savecore directory.

You can use the file(1) command to quickly examine files in the directory:

$ cd /var/crash/
$ file *
bounds:         ascii text
unix.0:         ELF 64-bit MSB executable SPARCV9 Version 1, UltraSPARC3 Extensions 
Required, statically linked, not stripped, no debugging information available
vmcore.0:       SunOS 5.11 Generic 64-bit SPARC crash dump from 'testsystem'
vmdump.0:       SunOS 5.11 Generic 64-bit SPARC compressed crash dump from 'testsystem'

Starting MDB

Now, run mdb on the crash dump you created, and check its status:

$ mdb unix.1 vmcore.1
Loading modules: [ unix krtld genunix ip nfs ipc ]
> ::status
debugging crash dump vmcore.1 (32-bit) from testsystem
operating system: 5.10 Generic (sun4u)
panic message: forced crash dump initiated at user request

In the examples presented in this guide, a crash dump from a 32-bit kernel is used. All of the techniques presented here are applicable to a 64-bit kernel, and care has been taken to distinguish pointers (sized differently on 32- and 64-bit systems) from fixed-sized quantities, which are invariant with respect to the kernel data model.

An UltraSPARC workstation was used to generate the example presented. Your results can vary depending on the architecture and model of system you use.