Debugging Tools

This section describes two debuggers that can be applied to device drivers. Both debuggers are described in detail in the Solaris Modular Debugger Guide.

The kmdb(1) kernel debugger provides typical runtime debugger facilities, such as breakpoints, watch points, and single-stepping. The kmdb debugger supersedes kadb, which was available in previous releases. The commands that were previously available from kadb are used in kmdb, in addition to new functionality. Where kadb could only be loaded at boot time, kmdb can be loaded at any time. The kmdb debugger is preferred for live, interactive debugging due to its execution controls.
The mdb(1) modular debugger is more limited than kmdb as a real-time debugger, but mdb has rich facilities for postmortem debugging.

The kmdb and mdb debuggers mostly share the same user interface. Many debugging techniques therefore can be applied with the same commands in both tools. Both debuggers support macros, dcmds, and dmods. A dcmd (pronounced dee-command) is a routine in the debugger that can access any of the properties of the current target program. A dcmd can be dynamically loaded at runtime. A dmod, which is short for debugger module, is a package of dcmds that can be loaded to provide non-standard behavior.

Both mdb and kmdb are backward-compatible with legacy debuggers such as adb and kadb. The mdb debugger can execute all of the macros that are available to kmdb as well as any legacy user-defined macros for adb. See the Solaris Modular Debugger Guide for information about where to find standard macro sets.

Postmortem Debugging

Postmortem analysis offers numerous advantages to driver developers. More than one developer can examine a problem in parallel. Multiple instances of the debugger can be used simultaneously on a single crash dump. The analysis can be performed offline so that the crashed system can be returned to service, if possible. Postmortem analysis enables the use of user-developed debugger functionality in the form of dmods. Dmods can bundle functionality that would be too memory-intensive for real-time debuggers, such as kmdb.

When a system panics while kmdb is loaded, control is passed to the debugger for immediate investigation. If kmdb does not seem appropriate for analyzing the current problem, a good strategy is to use :c to continue execution and save the crash dump. When the system reboots, you can perform postmortem analysis with mdb on the saved crash dump. This process is analogous to debugging an application crash from a process core file.

Note - In earlier versions of the Solaris operating system, adb(1) was the recommended tool for postmortem analysis. In the current Solaris operating system, mdb(1) is the recommended tool for postmortem analysis. The mdb() feature set surpasses the set of commands from the legacy crash(1M) utility. The crash utility is no longer available in the Solaris operating system.

Using the `kmdb` Kernel Debugger

The kmdb debugger is an interactive kernel debugger that provides the following capabilities:

Control of kernel execution
Inspection of the kernel state
Live modifications to the code

This section assumes that you are already familiar with the kmdb debugger. The focus in this section is on kmdb capabilities that are useful in device driver design. To learn how to use kmdb in detail, refer to the kmdb(1) man page and to the Solaris Modular Debugger Guide. If you are familiar with kadb, refer to the kadb(1M) man page for the major differences between kadb and kmdb.

The kmdb debugger can be loaded and unloaded at will. Instructions for loading and unloading kmdb are in the Solaris Modular Debugger Guide. For safety and convenience, booting with an alternate kernel is highly encouraged. The boot process is slightly different between the SPARC platform and the x86 platform, as described in this section.

Note - By default, kmdb uses the CPU ID as the prompt when kmdb is running. In the examples in this chapter [0] is used as the prompt unless otherwise noted.

Booting `kmdb` With an Alternate Kernel on the SPARC Platform

Use either of the following commands to boot a SPARC system with both kmdb and an alternate kernel:

boot kmdb -D kernel.test/sparcv9/unix 
boot kernel.test/sparcv9/unix -k

Booting `kmdb` With an Alternate Kernel on the x86 Platform

Use either of the following commands to boot an x86 system with both kmdb and an alternate kernel:

b kmdb -D kernel.test/unix 
b kernel.test/unix -k

Setting Breakpoints in `kmdb`

Use the bp command to set a breakpoint, as shown in the following example.

Example 22-7 Setting Standard Breakpoints in `kmdb`

[0]> myModule`myBreakpointLocation::bp

If the target module has not been loaded, then an error message that indicates this condition is displayed, and the breakpoint is not created. In this case you can use a deferred breakpoint. A deferred breakpoint activates automatically when the specified module is loaded. Set a deferred breakpoint by specifying the target location after the bp command. The following example demonstrates a deferred breakpoint.

Example 22-8 Setting Deferred Breakpoints in `kmdb`

[0]>::bp myModule`myBreakpointLocation

For more information on using breakpoints, see the Solaris Modular Debugger Guide. You can also get help by typing either of the following two lines:

> ::help bp
> ::bp dcmd

`kmdb` Macros for Driver Developers

The kmdb(1M) debugger supports macros that can be used to display kernel data structures. Use $M to display kmdb macros. Macros are used in the form:

[ address ] $<macroname

Note - Neither the information displayed by these macros nor the format in which the information is displayed, constitutes an interface. Therefore, the information and format can change at any time.

The kmdb macros in the following table are particularly useful to developers of device drivers. For convenience, legacy macro names are shown where applicable.

Table 22-1 `kmdb` Macros

Dcmd	Legacy Macro	Description
`::devinfo`	`devinfo` `devinfo_brief` `devinfo.prop`	Print a summary of a device node
`::walk devinfo_parents`	`devinfo.parent`	Walk the ancestors of a device node
`::walk devinfo_sibling`	`devinfo.sibling`	Walk the siblings of a device node
`::minornodes`	`devinfo.minor`	Print the minor nodes that correspond to the given device node
`::major2name`		Print the name of a device that is bound to a given device node.
`::devbindings`		Print the device nodes that are bound to a given device node or major number.

The ::devinfo dcmd displays a node state that can have one of the following values:

DS_ATTACHED: The driver's attach(9E) routine returned successfully.
DS_BOUND: The node is bound to a driver, but the driver's probe(9E) routine has not yet been called.
DS_INITIALIZED: The parent nexus has assigned a bus address for the driver. The implementation-specific initializations have been completed. The driver's probe(9E) routine has not yet been called at this point.
DS_LINKED: The device node has been linked into the kernel's device tree, but the system has not yet found a driver for this node.
DS_PROBED: The driver's probe(9E) routine returned successfully.
DS_READY: The device is fully configured.

Using the `mdb` Modular Debugger

The mdb(1) modular debugger can be applied to the following types of files:

Live operating system components
Operating system crash dumps
User processes
User process core dumps
Object files

The mdb debugger provides sophisticated debugging support for analyzing kernel problems. This section provides an overview of mdb features. For a complete discussion of mdb, refer to the Solaris Modular Debugger Guide.

Although mdb can be used to alter live kernel state, mdb lacks the kernel execution control that is provided by kmdb. As a result kmdb is preferred for runtime debugging. The mdb debugger is used more for static situations.

Note - The prompt for mdb is >.

Getting Started With the Modular Debugger

The mdb debugger provides an extensive programming API for implementing debugger modules so that driver developers can implement custom debugging support. The mdb debugger also provides many usability features, such as command-line editing, command history, an output pager, and online help.

Note - The adb macros should no longer be used. That functionality has largely been superseded by the dcmds in mdb.

The mdb debugger provides a rich set of modules and dcmds. With these tools, you can debug the Solaris kernel, any associated modules, and device drivers. These facilities enable you to perform tasks such as:

Formulate complex debugging queries
Locate all the memory allocated by a particular thread
Print a visual picture of a kernel STREAM
Determine what type of structure a particular address refers to
Locate leaked memory blocks in the kernel
Analyze memory to locate stack traces
Assemble dcmds into modules called dmods for creating customized operations

To get started, switch to the crash directory and type mdb, specifying a system crash dump, as illustrated in the following example.

Example 22-9 Invoking `mdb` on a Crash Dump

% cd /var/crash/testsystem
% ls
bounds     unix.0    vmcore.0
% mdb unix.0 vmcore.0
Loading modules: [ unix krtld genunix ufs_log ip usba s1394 cpc nfs ]
> ::status
debugging crash dump vmcore.0 (64-bit) from testsystem
operating system: 5.10 Generic (sun4u)
panic message: zero
dump content: kernel pages only

When mdb responds with the > prompt, you can run commands.

To examine the running kernel on a live system, run mdb from the system prompt as follows.

Example 22-10 Invoking `mdb` on a Running Kernel

# mdb -k
Loading modules: [ unix krtld genunix ufs_log ip usba s1394 ptm cpc ipc nfs ]
> ::status
debugging live kernel (64-bit) on testsystem
operating system: 5.10 Generic (sun4u)

Useful Debugging Tasks With `kmdb` and `mdb`

This section provides examples of useful debugging tasks. The tasks in this section can be performed with either mdb or kmdb unless specifically noted. This section assumes a basic knowledge of the use of kmdb and mdb. Note that the information presented here is dependent on the type of system used. A Sun Blade^TM 100 workstation running the 64-bit kernel was used to produce these examples.

Caution - Because irreversible destruction of data can result from modifying data in kernel structures, you should exercise extreme caution. Do not modify or rely on data in structures that are not part of the Solaris DDI. See the Intro(9S) man page for information on structures that are part of the Solaris DDI.

Exploring System Registers With `kmdb`

The kmdb debugger can display machine registers as a group or individually. To display all registers as a group, use $r as shown in the following example.

Example 22-11 Reading All Registers on a SPARC Processor With `kmdb`

[0]: $r

g0    0                                 l0      0
g1    100130a4      debug_enter         l1      edd00028
g2    10411c00      tsbmiss_area+0xe00  l2      10449c90
g3    10442000      ti_statetbl+0x1ba   l3      1b
g4    3000061a004                       l4      10474400     ecc_syndrome_tab+0x80
g5    0                                 l5      3b9aca00
g6    0                                 l6      0
g7    2a10001fd40                       l7      0
o0    0                                 i0      0
o1    c                                 i1      10449e50
o2    20                                i2      0
o3    300006b2d08                       i3      10
o4    0                                 i4      0
o5    0                                 i5      b0
sp    2a10001b451                       fp      2a10001b521
o7    1001311c      debug_enter+0x78    i7      1034bb24     zsa_xsint+0x2c4
y     0
tstate: 1604  (ccr=0x0, asi=0x0, pstate=0x16, cwp=0x4)
pstate: ag:0 ie:1 priv:1 am:0 pef:1 mm:0 tle:0 cle:0 mg:0 ig:0
winreg: cur:4 other:0 clean:7 cansave:1 canrest:5 wstate:14
tba   0x10000000
pc    edd000d8 edd000d8:        ta      %icc,%g0 + 125
npc   edd000dc edd000dc:        nop

The debugger exports each register value to a variable with the same name as the register. If you read the variable, the current value of the register is returned. If you write to the variable, the value of the associated machine register is changed. The following example changes the value of the %o0 register from 0 to 1 on an x86 machine.

Example 22-12 Reading and Writing Registers on an x86 Machine With `kmdb`

[0]> &lt;eax=K
        c1e6e0f0
[0]> 0>eax
[0]> &lt;eax=K
        0
[0]>  c1e6e0f0>eax

If you need to inspect the registers of a different processor, you can use the ::cpuregs dcmd. The ID of the processor to be examined can be supplied as either the address to the dcmd or as the value of the -c option, as shown in the following example.

Example 22-13 Inspecting the Registers of a Different Processor

[0]> 0::cpuregs
   %cs = 0x0158            %eax = 0xc1e6e0f0 kmdbmod`kaif_dvec
   %ds = 0x0160            %ebx = 0x00000000

The following example switches from processor 0 to processor 3 on a SPARC machine. The %g3 register is inspected and then cleared. To confirm the new value, %g3 is read again.

Example 22-14 Retrieving the Value of an Individual Register From a Specified Processor

[0]> 3::switch
[3]> <g3=K
        24
[3]> 0>g3
[3]> <g3
        0

Detecting Kernel Memory Leaks

The ::findleaks dcmd provides powerful, efficient detection of memory leaks in kernel crash dumps. The full set of kernel-memory debugging features must be enabled for ::findleaks to be effective. For more information, see Setting kmem_flags Debugging Flags. Run ::findleaks during driver development and testing to detect code that leaks memory, thus wasting kernel resources. See Chapter 9, Debugging With the Kernel Memory Allocator, in Solaris Modular Debugger Guide for a complete discussion of ::findleaks.

Note - Code that leaks kernel memory can render the system vulnerable to denial-of-service attacks.

Writing Debugger Commands With `mdb`

The mdb debugger provides a powerful API for implementing debugger facilities that you customize to debug your driver. The Solaris Modular Debugger Guide explains the programming API in detail.

The SUNWmdbdm package installs sample mdb source code in the directory /usr/demo/mdb. You can use mdb to automate lengthy debugging chores or help to validate that your driver is behaving properly. You can also package your mdb debugging modules with your driver product. With packaging, these facilities are available to service personnel at a customer site.

Obtaining Kernel Data Structure Information

The Solaris kernel provides data type information in structures that can be inspected with either kmdb or mdb.

Note - The kmdb and mdb dcmds can be used only with objects that contain compressed symbolic debugging information that has been designed for use with mdb. This information is currently available only for certain Solaris kernel modules. The SUNWzlib package must be installed to process the symbolic debugging information.

The following example demonstrates how to display the data in the scsi_pkt structure.

Example 22-15 Displaying Kernel Data Structures With a Debugger

> 7079ceb0::print -t 'struct scsi_pkt'
{
    opaque_t pkt_ha_private = 0x7079ce20
    struct scsi_address pkt_address = {
        struct scsi_hba_tran *a_hba_tran = 0x70175e68
        ushort_t a_target = 0x6
        uchar_t a_lun = 0
        uchar_t a_sublun = 0
    }
    opaque_t pkt_private = 0x708db4d0
    int (*)() *pkt_comp = sd_intr
    uint_t pkt_flags = 0
    int pkt_time = 0x78
    uchar_t *pkt_scbp = 0x7079ce74
    uchar_t *pkt_cdbp = 0x7079ce64
    ssize_t pkt_resid = 0
    uint_t pkt_state = 0x37
    uint_t pkt_statistics = 0
    uchar_t pkt_reason = 0
}

The size of a data structure can be useful in debugging. Use the ::sizeof dcmd to obtain the size of a structure, as shown in the following example.

Example 22-16 Displaying the Size of a Kernel Data Structure

> ::sizeof struct scsi_pkt
sizeof (struct scsi_pkt) = 0x58

The address of a specific member within a structure is also useful in debugging. Several methods are available for determining a member's address.

Use the ::offsetof dcmd to obtain the offset for a given member of a structure, as in the following example.

Example 22-17 Displaying the Offset to a Kernel Data Structure

> ::offsetof struct scsi_pkt pkt_state
offsetof (struct pkt_state) = 0x48

Use the ::print dcmd with the -a option to display the addresses of all members of a structure, as in the following example.

Example 22-18 Displaying the Relative Addresses of a Kernel Data Structure

> ::print -a struct scsi_pkt
{
    0 pkt_ha_private
    8 pkt_address {
    ...
    }
    18 pkt_private
    ...
}

If an address is specified with ::print in conjunction with the -a option, the absolute address for each member is displayed.

Example 22-19 Displaying the Absolute Addresses of a Kernel Data Structure

> 10000000::print -a struct scsi_pkt
{
    10000000 pkt_ha_private
    10000008 pkt_address {
    ...
    }
    10000018 pkt_private
    ...
}

The ::print, ::sizeof and ::offsetof dcmds enable you to debug problems when your driver interacts with the Solaris kernel.

Caution - This facility provides access to raw kernel data structures. You can examine any structure whether or not that structure appears as part of the DDI. Therefore, you should refrain from relying on any data structure that is not explicitly part of the DDI.

Note - These dcmds should be used only with objects that contain compressed symbolic debugging information that has been designed for use with mdb. Symbolic debugging information is currently available for certain Solaris kernel modules only. The SUNWzlib (32-bit) or SUNWzlibx (64-bit) decompression software must be installed to process the symbolic debugging information. The kmdb debugger can process symbolic type data with or without the SUNWzlib or SUNWzlibx packages.

Obtaining Device Tree Information

The mdb debugger provides the ::prtconf dcmd for displaying the kernel device tree. The output of the ::prtconf dcmd is similar to the output of the prtconf(1M) command.

Example 22-20 Using the `::prtconf` Dcmd

> ::prtconf
300015d3e08      SUNW,Sun-Blade-100
    300015d3c28      packages (driver not attached)
        300015d3868      SUNW,builtin-drivers (driver not attached)
        300015d3688      deblocker (driver not attached)
        300015d34a8      disk-label (driver not attached)
        300015d32c8      terminal-emulator (driver not attached)
        300015d30e8      obp-tftp (driver not attached)
        300015d2f08      dropins (driver not attached)
        300015d2d28      kbd-translator (driver not attached)
        300015d2b48      ufs-file-system (driver not attached)
    300015d3a48      chosen (driver not attached)
    300015d2968      openprom (driver not attached)

You can display the node by using a macro, such as the ::devinfo dcmd, as shown in the following example.

Example 22-21 Displaying Device Information for an Individual Node

> 300015d3e08::devinfo
300015d3e08      SUNW,Sun-Blade-100
        System properties at 0x300015abdc0:
            name='relative-addressing' type=int items=1
                value=00000001
            name='MMU_PAGEOFFSET' type=int items=1
                value=00001fff
            name='MMU_PAGESIZE' type=int items=1
                value=00002000
            name='PAGESIZE' type=int items=1
                value=00002000
        Driver properties at 0x300015abe00:
            name='pm-hardware-state' type=string items=1
                value='no-suspend-resume'

Use ::prtconf to see where your driver has attached in the device tree, and to display device properties. You can also specify the verbose (-v) flag to ::prtconf to display the properties for each device node, as follows.

Example 22-22 Using the `::prtconf` Dcmd in Verbose Mode

> ::prtconf -v
DEVINFO          NAME
300015d3e08      SUNW,Sun-Blade-100
        System properties at 0x300015abdc0:
            name='relative-addressing' type=int items=1
                value=00000001
            name='MMU_PAGEOFFSET' type=int items=1
                value=00001fff
            name='MMU_PAGESIZE' type=int items=1
                value=00002000
            name='PAGESIZE' type=int items=1
                value=00002000
        Driver properties at 0x300015abe00:
            name='pm-hardware-state' type=string items=1
                value='no-suspend-resume'
        ...
        300015ce798      pci10b9,5229, instance #0
                Driver properties at 0x300015ab980:
                    name='target2-dcd-options' type=any items=4
                        value=00.00.00.a4
                    name='target1-dcd-options' type=any items=4
                        value=00.00.00.a2
                    name='target0-dcd-options' type=any items=4
                        value=00.00.00.a4

Another way to locate instances of your driver is the ::devbindings dcmd. Given a driver name, the command displays a list of all instances of the named driver as demonstrated in the following example.

Example 22-23 Using the `::devbindings` Dcmd to Locate Driver Instances

> ::devbindings dad
300015ce3d8      ide-disk (driver not attached)
300015c9a60      dad, instance #0
        System properties at 0x300015ab400:
            name='lun' type=int items=1
                value=00000000
            name='target' type=int items=1
                value=00000000
            name='class_prop' type=string items=1
                value='ata'
            name='type' type=string items=1
                value='ata'
            name='class' type=string items=1
                value='dada'
...
300015c9880      dad, instance #1
        System properties at 0x300015ab080:
            name='lun' type=int items=1
                value=00000000
            name='target' type=int items=1
                value=00000002
            name='class_prop' type=string items=1
                value='ata'
            name='type' type=string items=1
                value='ata'
            name='class' type=string items=1
                value='dada'

Retrieving Driver Soft State Information

A common problem when debugging a driver is retrieving the soft state for a particular driver instance. The soft state is allocated with the ddi_soft_state_zalloc(9F) routine. The driver can obtain the soft state through ddi_get_soft_state(9F). The name of the soft state pointer is the first argument to ddi_soft_state_init(9F)). With the name, you can use mdb to retrieve the soft state for a particular driver instance through the ::softstate dcmd:

> *bst_state::softstate 0x3
702b7578

In this case, ::softstate is used to fetch the soft state for instance 3 of the bst sample driver. This pointer references a bst_soft structure that is used by the driver to track state for this instance.

Modifying Kernel Variables

You can use both kmdb and mdb to modify kernel variables or other kernel state. Kernel state modification with mdb should be done with care, because mdb does not stop the kernel before making modifications. Groups of modifications can be made atomically by using kmdb, because kmdb stops the kernel before allowing access by the user. The mdb debugger is capable of making single atomic modifications only.

Be sure to use the proper format specifier to perform the modification. The formats are:

w – Writes the lowest two bytes of the value of each expression to the target beginning at the location specified by dot
W – Writes the lowest 4 bytes of the value of each expression to the target beginning at the location specified by dot
Z – Write the complete 8 bytes of the value of each expression to the target beginning at the location specified by dot

Use the ::sizeof dcmd to determine the size of the variable to be modified.

The following example overwrites the value of moddebug with the value 0x80000000.

Example 22-24 Modifying a Kernel Variable With a Debugger

> moddebug/W 0x80000000
    moddebug:       0 = 0x80000000

Debugging Tools

Postmortem Debugging

Using the kmdb Kernel Debugger

Booting kmdb With an Alternate Kernel on the SPARC Platform

Booting kmdb With an Alternate Kernel on the x86 Platform

Setting Breakpoints in kmdb

Example 22-7 Setting Standard Breakpoints in kmdb

Example 22-8 Setting Deferred Breakpoints in kmdb

kmdb Macros for Driver Developers

Table 22-1 kmdb Macros

Using the mdb Modular Debugger