| Skip Navigation Links | |
| Exit Print View | |
|
man pages section 1: User Commands Oracle Solaris 11.1 Information Library |
| Skip Navigation Links | |
| Exit Print View | |
|
|
man pages section 1: User Commands Oracle Solaris 11.1 Information Library |
- modular debugger
mdb [-fkmuwyAFKMSUW] [±o option] [-p pid] [-s distance]
[-I path] [-L path] [-P prompt] [-R root]
[-V dis-version] [object [core] | core | suffix]The mdb utility is an extensible utility for low-level debugging and editing of the live operating system, operating system crash dumps, user processes, user process core dumps, and object files. For a more detailed description of mdb features, refer to the manual, Oracle Solaris Modular Debugger Guide.
Debugging is the process of analyzing the execution and state of a software program in order to remove defects. Traditional debugging tools provide facilities for execution control so that programmers can re-execute programs in a controlled environment and display the current state of program data or evaluate expressions in the source language used to develop the program.
Unfortunately, these techniques are often inappropriate for debugging complex software systems such as an operating system, where bugs might not be reproducible and program state is massive and distributed, for programs that are highly optimized, have had their debug information removed, or are themselves low-level debugging tools, or for customer situations where the developer can only access post-mortem information.
mdb provides a completely customizable environment for debugging these programs and scenarios, including a dynamic module facility that programmers can use to implement their own debugging commands to perform program-specific analysis. Each mdb module can be used to examine the program in several different contexts, including live and post-mortem.
The target is the program being inspected by the debugger. mdb currently provides support for the following types of targets: user processes, user process core files, the live operating system (via /dev/kmem and /dev/ksyms), operating system crash dumps, user process images recorded inside an operating system crash dump, ELF object files, and raw binary files. Each target exports a standard set of properties, including one or more address spaces, one or more symbol tables, a set of load objects, and a set of threads that can be examined using the debugger commands described below.
A debugger command, or dcmd (pronounced dee-command) in mdb terminology, is a routine in the debugger that can access any of the properties of the current target. mdb parses commands from standard input, and then executes the corresponding dcmds. Each dcmd can also accept a list of string or numerical arguments, as shown in the syntax description below. mdb contains a set of built-in dcmds, described below, that are always available. You can also extend the capabilities of mdb itself by writing your own dcmds, as described in the Oracle Solaris Modular Debugger Guide.
A walker is a set of routines that describe how to walk, or iterate, through the elements of a particular program data structure. A walker encapsulates the data structure's implementation from dcmds and from mdb itself. You can use walkers interactively, or use them as a primitive to build other dcmds or walkers. As with dcmds, you can extend mdb by implementing your own walkers as part of a debugger module.
A debugger module, or dmod (pronounced dee-mod), is a dynamically loaded library containing a set of dcmds and walkers. During initialization, mdb attempts to load dmods corresponding to the load objects present in the target. You can subsequently load or unload dmods at any time while running mdb. mdb ships with a set of standard dmods for debugging the Solaris kernel. The Oracle Solaris Modular Debugger Guide contains more information on developing your own debugger modules.
A macro file is a text file containing a set of commands to execute. Macro files are typically used to automate the process of displaying a simple data structure. mdb provides complete backward compatibility for the execution of macro files written for adb(1), and the Solaris installation includes a set of macro files for debugging the Solaris kernel that can be used with either tool.
The debugger processes commands from standard input. If standard input is a terminal, mdb provides terminal editing capabilities. mdb can also process commands from macro files and from dcmd pipelines, described below. The language syntax is designed around the concept of computing the value of an expression (typically a memory address in the target), and then applying a dcmd to that address. The current address location is referred to as dot, and its value is referenced using ``.''.
A metacharacter is one of the following characters:
[ ] | ! / \ ? = > $ : ;
NEWLINE SPACE TABA blank is a TAB or a SPACE. A word is a sequence of characters separated by one or more non-quoted metacharacters. Some of the metacharacters only function as delimiters in certain contexts, as described below. An identifier is a sequence of letters, digits, underscores, periods, or backquotes beginning with a letter, underscore, or period. Identifiers are used as the names of symbols, variables, dcmds, and walkers. Commands are delimited by a NEWLINE or semicolon ( ; ).
A dcmd is denoted by one of the following words or metacharacters:
/ \ ? = > $character :character ::identifier
dcmds named by metacharacters or prefixed by a single $ or : are provided as built-in operators, and implement complete compatibility with the command set of the legacy adb(1) utility. Once a dcmd has been parsed, the /, \, ?, =, >, $, and : characters are no longer recognized as metacharacters until the termination of the argument list.
A simple-command is a dcmd followed by a sequence of zero or more blank-separated words. The words are passed as arguments to the invoked dcmd, except as specified under Quoting and Arithmetic Expansion below. Each dcmd returns an exit status that indicates it was either successful, failed, or was invoked with invalid arguments.
A pipeline is a sequence of one or more simple commands separated by |. Unlike the shell, dcmds in mdb pipelines are not executed as separate processes. After the pipeline has been parsed, each dcmd is invoked in order from left to right. Each dcmd's output is processed and stored as described under dcmd Pipelines below. Once the left-hand dcmd is complete, its processed output is used as input for the next dcmd in the pipeline. If any dcmd does not return a successful exit status, the pipeline is aborted.
An expression is a sequence of words that is evaluated to compute a 64-bit unsigned integer value. The words are evaluated using the rules described under Arithmetic Expansion below.
A command is one of the following:
A simple-command or pipeline can be optionally suffixed with the ! character, indicating that the debugger should open a pipe(2) and send the standard output of the last dcmd in the mdb pipeline to an external process created by executing $SHELL -c followed by the string formed by concatenating the words after the ! character. For more details, refer to Shell Escapes below.
A simple-command or pipeline can be prefixed with an expression. Before execution of the pipeline, the value of dot (the variable denoted by ``.'') is set to the value of the expression.
A simple-command or pipeline can be prefixed with two expressions. The first is evaluated to determine the new value of dot, and the second is evaluated to determine a repeat count for the first dcmd in the pipeline. This dcmd is executed count times before the next dcmd in the pipeline is executed. The repeat count only applies to the first dcmd in the pipeline.
If the initial expression is omitted, dot is not modified but the first dcmd in the pipeline is repeated according to the value of the expression.
A command can consist only of an arithmetic expression. The expression is evaluated and the dot variable is set to its value, and then the previous dcmd and arguments are executed using the new value of dot.
A command can consist only of a dot expression and repeat count expression. After dot is set to the value of the first expression, the previous dcmd and arguments are repeatedly executed the number of times specified by the value of the second expression.
If the initial expression is omitted, dot is not modified but the previous dcmd and arguments are repeatedly executed the number of times specified by the value of the count expression.
If the command begins with the ! character, no dcmds are executed and the debugger simply executes $SHELL -c followed by the string formed by concatenating the words after the ! character.
A word beginning with // causes that word and all the subsequent characters up to a NEWLINE to be ignored.
Arithmetic expansion is performed when an mdb command is preceded by an optional expression representing a start address, or a start address and a repeat count. Arithmetic expansion can also be performed to compute a numerical argument for a dcmd. An arithmetic expression can appear in an argument list enclosed in square brackets preceded by a dollar sign ($[ expression ]), and is replaced by the value of the expression.
Expressions can contain any of the following special words:
The specified integer value. Integer values can be prefixed with 0i or 0I to indicate binary values, 0o or 0O to indicate octal values, 0t or 0T to indicate decimal values, and 0x or 0X to indicate hexadecimal values (the default).
The specified decimal floating point value, converted to its IEEE double-precision floating point representation.
The integer value computed by converting each character to a byte equal to its ASCII value. Up to eight characters can be specified in a character constant. Characters are packed into the integer in reverse order (right-to-left) beginning at the least significant byte.
The value of the variable named by identifier.
The value of the symbol named by identifier.
The value of expression.
The value of dot.
The most recent value of dot used to execute a dcmd.
The value of dot incremented by the current increment.
The value of dot decremented by the current increment.
The increment is a global variable that stores the total bytes read by the last formatting dcmd. For more information on the increment, refer to the discussion of Formatting dcmds below.
Unary operators are right associative and have higher precedence than binary operators. The unary operators are:
Logical negation.
Bitwise complement.
Integer negation.
The value of a pointer-sized quantity at the object file location corresponding to virtual address expression in the target's virtual address space.
The value of a char, short, int, or long-sized quantity at the object file location corresponding to virtual address expression in the target's virtual address space.
The value of a one, two, four, or eight-byte quantity at the object file location corresponding to virtual address expression in the target's virtual address space.
The value of a pointer-sized quantity at virtual address expression in the target's virtual address space.
The value of a char, short, int, or long-sized quantity at virtual address expression in the target's virtual address space.
The value of a one, two, four, or eight-byte quantity at virtual address expression in the target's virtual address space.
Binary operators are left associative and have lower precedence than unary operators. The binary operators, in order of precedence from highest to lowest, are:
Integer multiplication.
Integer division.
Left-hand side rounded up to next multiple of right-hand side.
Integer addition.
Integer subtraction.
Bitwise shift left.
Bitwise shift right.
Logical equality.
Logical inequality.
Bitwise AND.
Bitwise exclusive OR.
Bitwise inclusive OR.
Each metacharacter described above (see Syntax) terminates a word unless quoted. Characters can be quoted (forcing mdb to interpret each character as itself without any special significance) by enclosing them in a pair of single (' ') or double (“ ”) quote marks. A single quote cannot appear within single quotes. Inside double quotes, mdb recognizes the C programming language character escape sequences.
The ! character can be used to create a pipeline between an mdb command and the user's shell. If the $SHELL environment variable is set, mdb forks and execs this program for shell escapes; otherwise /bin/sh is used. The shell is invoked with the -c option followed by a string formed by concatenating the words after the ! character. The ! character takes precedence over all other metacharacters, except semicolon (;) and NEWLINE. Once a shell escape is detected, the remaining characters up to the next semicolon or NEWLINE are passed as is to the shell. The output of shell commands can not be piped to mdb dcmds. Commands executed by a shell escape have their output sent directly to the terminal, not to mdb.
A variable is a variable name, a corresponding integer value, and a set of attributes. A variable name is a sequence of letters, digits, underscores, or periods. A variable can be assigned a value using the > dcmd or ::typeset dcmd, and its attributes can be manipulated using the ::typeset dcmd. Each variable's value is represented as a 64-bit unsigned integer. A variable can have one or more of the following attributes: read-only (cannot be modified by the user), persistent (cannot be unset by the user), and tagged (user-defined indicator).
The following variables are defined as persistent:
The most recent value printed using the /, \, ?, or = dcmd.
The most recent count used with the $< dcmd.
The virtual address of the base of the data section.
The size of the data section in bytes.
The virtual address of the entry point.
The initial bytes (magic number) of the target's primary object file, or zero if no object file has been read yet.
The size of the text section in bytes.
The count of the number of times the matched software event specifier has been matched. See Event Callbacks, below.
The thread identifier of the current representative thread. The value of the identifier depends on the threading model used by the current target. See Thread Support, below.
In addition, the mdb kernel and process targets export the current values of the representative thread's register set as named variables. The names of these variables depend on the target's platform and instruction set architecture.
As explained in the Syntax description above, a symbol identifier present in an expression context evaluates to the value of this symbol. The value typically denotes the virtual address of the storage associated with the symbol in the target's virtual address space. A target can support multiple symbol tables including, but not limited to, a primary executable symbol table, a primary dynamic symbol table, a run-time link-editor symbol table, and standard and dynamic symbol tables for each of a number of load objects (such as shared libraries in a user process, or kernel modules in the Solaris kernel). The target typically searches the primary executable's symbol tables first, and then one or more of the other symbol tables. Notice that ELF symbol tables only contain entries for external, global, and static symbols; automatic symbols do not appear in the symbol tables processed by mdb.
Additionally, mdb provides a private user-defined symbol table that is searched prior to any of the target symbol tables. The private symbol table is initially empty, and can be manipulated using the ::nmadd and ::nmdel dcmds. The ::nm -P option can be used to display the contents of the private symbol table. The private symbol table allows the user to create symbol definitions for program functions or data that were either missing from the original program or stripped out. These definitions are then used whenever mdb converts a symbolic name to an address, or an address to the nearest symbol.
As targets contain multiple symbol tables, and each symbol table can include symbols from multiple object files, different symbols with the same name can exist. mdb uses the backquote (`) character as a symbol name scoping operator to allow the programmer to obtain the value of the desired symbol in this situation. The programmer can specify the scope used to resolve a symbol name as either: object`name, or file`name, or object`file`name. The object identifier refers to the name of a load object. The file identifier refers to the basename of a source file that has a symbol of type STT_FILE in the specified object's symbol table. The object identifier's interpretation depends on the target type.
The mdb kernel target expects object to specify the basename of a loaded kernel module. For example, the symbol name
specfs`_init
evaluates to the value of the _init symbol in the specfs kernel module.
The mdb process target expects object to specify the name of the executable or of a loaded shared library. It can take any of the following forms:
An exact match (that is, a full pathname): /usr/lib/libc.so.1
An exact basename match: libc.so.1
An initial basename match up to a ``.'' suffix: libc.so or libc
The literal string a.out is accepted as an alias for the executable.
The process target also accepts any of the four forms described above preceded by an optional link-map id (lmid). The lmid prefix is specified by an initial “LM” followed by the link-map id in hexadecimal followed by an additional backquote. For example, the symbol name
LM0`libc.so.1`_init
evaluates to the value of the _init symbol in the libc.so.1 library that is loaded on link-map 0 (LM_ID_BASE). The link-map specifier can be necessary to resolve symbol naming conflicts in the event that the same library is loaded on more than one link map. For more information on link maps, refer to the Linker and Libraries Guide and dlopen(3C). Link-map identifiers are displayed when symbols are printed according to the setting of the showlmid option, as described under OPTIONS.
In the case of a naming conflict between symbols and hexadecimal integer values, mdb attempts to evaluate an ambiguous token as a symbol first, before evaluating it as an integer value. For example, the token f can either refer to the decimal integer value 15 specified in hexadecimal (the default base), or to a global variable named f in the target's symbol table. If a symbol with an ambiguous name is present, the integer value can be specified by using an explicit 0x or 0X prefix.
As described earlier, each mdb dmod provides a set of dcmds and walkers. dcmds and walkers are tracked in two distinct, global namespaces. mdb also keeps track of a dcmd and walker namespace associated with each dmod. Identically named dcmds or walkers within a given dmod are not allowed: a dmod with this type of naming conflict fails to load. Name conflicts between dcmds or walkers from different dmods are allowed in the global namespace. In the case of a conflict, the first dcmd or walker with that particular name to be loaded is given precedence in the global namespace. Alternate definitions are kept in a list in load order. The backquote character (`) can be used in a dcmd or walker name as a scoping operator to select an alternate definition. For example, if dmods m1 and m2 each provide a dcmd d, and m1 is loaded prior to m2, then:
Executes m1's definition of d.
Executes m1's definition of d.
Executes m2's definition of d.
If module m1 were now unloaded, the next dcmd on the global definition list (m2`d) would be promoted to global visibility. The current definition of a dcmd or walker can be determined using the ::which dcmd, described below. The global definition list can be displayed using the ::which -v option.
dcmds can be composed into a pipeline using the | operator. The purpose of a pipeline is to pass a list of values, typically virtual addresses, from one dcmd or walker to another. Pipeline stages might be used to map a pointer from one type of data structure to a pointer to a corresponding data structure, to sort a list of addresses, or to select the addresses of structures with certain properties.
mdb executes each dcmd in the pipeline in order from left to right. The leftmost dcmd is executed using the current value of dot, or using the value specified by an explicit expression at the start of the command. When a | operator is encountered, mdb creates a pipe (a shared buffer) between the output of the dcmd to its left and the mdb parser, and an empty list of values. As the dcmd executes, its standard output is placed in the pipe and then consumed and evaluated by the parser, as if mdb were reading this data from standard input. Each line must consist of an arithmetic expression terminated by a NEWLINE or semicolon (;). The value of the expression is appended to the list of values associated with the pipe. If a syntax error is detected, the pipeline is aborted.
When the dcmd to the left of a | operator completes, the list of values associated with the pipe is then used to invoke the dcmd to the right of the | operator. For each value in the list, dot is set to this value and the right-hand dcmd is executed. Only the rightmost dcmd in the pipeline has its output printed to standard output. If any dcmd in the pipeline produces output to standard error, these messages are printed directly to standard error and are not processed as part of the pipeline.
The debugger ignores the PIPE and QUIT signals. The INT signal aborts the command that is currently executing. The debugger intercepts and provides special handling for the ILL, TRAP, EMT, FPE, BUS, and SEGV signals. If any of these signals are generated asynchronously (that is, delivered from another process using kill(2)), mdb restores the signal to its default disposition and dump core. However, if any of these signals are generated synchronously by the debugger process itself and a dcmd from an externally loaded dmod is currently executing, and standard input is a terminal, mdb provides a menu of choices allowing the user to force a core dump, quit without producing a core dump, stop for attach by a debugger, or attempt to resume. The resume option aborts all active commands and unload the dmod whose dcmd was active at the time the fault occurred. It can then be subsequently re-loaded by the user. The resume option provides limited protection against buggy dcmds. Refer to WARNINGS, Use of the Error Recovery Mechanism, below for information about the risks associated with the resume option.
The text of the last HISTSIZE (default 128) commands entered from a terminal device are saved in memory. The in-line editing facility, described next, provides key mappings for searching and fetching elements from the history list.
If standard input is a terminal device, mdb provides some simple emacs-style facilities for editing the command line. The search, previous, and next commands in edit mode provide access to the history list. Only strings, not patterns, are matched when searching. In the table below, the notation for control characters is caret (^) followed by a character shown in upper case. The notation for escape sequences is M- followed by a character. For example, M-f (pronounced meta-eff) is entered by depressing ESC followed by 'f', or by depressing Meta followed by 'f' on keyboards that support a Meta key. A command line is committed and executed using RETURN or NEWLINE. The edit commands are:
Move cursor forward (right) one character.
Move cursor forward one word.
Move cursor backward (left) one character.
Move cursor backward one word.
Move cursor to start of line.
Move cursor to end of line.
Delete current character, if the current line is not empty. If the current line is empty, ^D denotes EOF and the debugger exits.
(Meta-backspace) Delete previous word.
Delete from the cursor to the end of the line.
Clear the screen and reprint the current line.
Transpose current character with next character.
Fetch the next command from the history. Each time ^N is entered, the next command forward in time is retrieved.
Fetch the previous command from the history. Each time ^P is entered, the next command backward in time is retrieved.
Search backward in the history for a previous command line containing string. The string should be terminated by a RETURN or NEWLINE. If string is omitted, the previous history element containing the most recent string is retrieved.
The editing mode also interprets the following user-defined sequences as editing commands. User defined sequences can be read or modified using the stty(1) command.
User defined erase character (usually ^H or ^?). Delete previous character.
User defined interrupt character (usually ^C). Abort the current command and print a new prompt.
User defined kill character (usually ^U). Kill the entire current command line.
User defined quit character (usually ^\). Quit the debugger.
User defined suspend character (usually ^Z). Suspend the debugger.
User defined word erase character (usually ^W). Erase the preceding word.
On keyboards that support an extended keypad with arrow keys, mdb interprets these keystrokes as editing commands:
Fetch the previous command from the history (same as ^P).
Fetch the next command from the history (same as ^N).
Move cursor backward one character (same as ^B).
Move cursor forward one character (same as ^F).
mdb provides a built-in output pager. The output pager is enabled if the debugger's standard output is a terminal device. Each time a command is executed, mdb pauses after one screenful of output is produced and displays a pager prompt:
>> More [<space>, <cr>, q, n, c, a] ?
The following key sequences are recognized by the pager:
Display the next screenful of output.
Abort the current top-level command and return to the prompt.
Continue displaying output without pausing at each screenful until the current top-level command is complete.
Display the next line of output.
Quit (abort) the current dcmd only.
The /, \, ?, and = metacharacters are used to denote the special output formatting dcmds. Each of these dcmds accepts an argument list consisting of one or more format characters, repeat counts, or quoted strings. A format character is one of the ASCII characters shown in the table below. Format characters are used to read and format data from the target. A repeat count is a positive integer preceding the format character that is always interpreted in base 10 (decimal). A repeat count can also be specified as an expression enclosed in square brackets preceded by a dollar sign ($[ ]). A string argument must be enclosed in double-quotes (“ ”). No blanks are necessary between format arguments.
The formatting dcmds are:
Display data from the target's virtual address space starting at the virtual address specified by dot.
Display data from the target's physical address space starting at the physical address specified by dot.
Display data from the target's primary object file starting at the object file location corresponding to the virtual address specified by dot.
Display the value of dot itself in each of the specified data formats. The = dcmd is therefore useful for converting between bases and performing arithmetic.
In addition to dot, mdb keeps track of another global value called the increment. The increment represents the distance between dot and the address following all the data read by the last formatting dcmd. For example, if a formatting dcmd is executed with dot equal to address A, and displays a 4-byte integer, then after this dcmd completes, dot is still A, but the increment is set to 4. The + character (described under Arithmetic Expansion above) would now evaluate to the value A + 4, and could be used to reset dot to the address of the next data object for a subsequent dcmd.
Most format characters increase the value of the increment by the number of bytes corresponding to the size of the data format, shown in the table. The table of format characters can be displayed from within mdb using the ::formats dcmd. The format characters are:
|
The /, \, and ? formatting dcmds can also be used to write to the target's virtual address space, physical address space, or object file by specifying one of the following modifiers as the first format character, and then specifying a list of words that are either immediate values or expressions enclosed in square brackets preceded by a dollar sign ($[ ]).
The write modifiers are:
Write the lowest byte of the value of each expression to the target beginning at the location specified by dot.
Write the lowest two bytes of the value of each expression to the target beginning at the location specified by dot.
Write the lowest 4 bytes of the value of each expression to the target beginning at the location specified by dot.
Write the complete 8 bytes of the value of each expression to the target beginning at the location specified by dot.
The /, \, and ? formatting dcmds can also be used to search for a particular integer value in the target's virtual address space, physical address space, and object file, respectively, by specifying one of the following modifiers as the first format character, and then specifying a value and optional mask. The value and mask are each specified as either immediate values or expressions enclosed in square brackets preceded by a dollar sign. If only a value is specified, mdb reads integers of the appropriate size and stops at the address containing the matching value. If a value V and mask M are specified, mdb reads integers of the appropriate size and stops at the address containing a value X where (X & M) == V. At the completion of the dcmd, dot is updated to the address containing the match. If no match is found, dot is left at the last address that was read.
The search modifiers are:
|
Notice that for both user and kernel targets, an address space is typically composed of a set of discontiguous segments. It is not legal to read from an address that does not have a corresponding segment. If a search reaches a segment boundary without finding a match, it aborts when the read past the end of the segment boundary fails.
mdb provides facilities for controlling and tracing the execution of a live running program. Currently, only the user process target provides support for execution control. mdb provides a simple model of execution control: a target process can be started from within the debugger using ::run, or mdb can attach to an existing process using :A, ::attach, or the -p command-line option, as described below. A list of traced software events can be specified by the user. Each time a traced event occurs in the target process, all threads in the target stop, the thread that triggered the event is chosen as the representative thread, and control returns to the debugger. Once the target program is set running, control can be asynchronously returned to the debugger by typing the user-defined interrupt character (typically ^C).
A software event is a state transition in the target program that is observed by the debugger. For example, the debugger can observe the transition of a program counter register to a value of interest (a breakpoint) or the delivery of a particular signal.
A software event specifier is a description of a class of software events that is used by the debugger to instrument the target program in order to observe these events. The ::events dcmd is used to list the software event specifiers. A set of standard properties is associated with each event specifier, as described under ::events, below.
The debugger can observe a variety of different software events, including breakpoints, watchpoints, signals, machine faults, and system calls. New specifiers can be created using ::bp, ::fltbp, ::sigbp, ::sysbp, or ::wp. Each specifier has an associated callback (an mdb command string to execute as if it had been typed at the command prompt) and a set of properties, as described below. Any number of specifiers for the same event can be created, each with different callbacks and properties. The current list of traced events and the properties of the corresponding event specifiers can be displayed using the ::events dcmd. The event specifier properties are defined as part of the description of the ::events and ::evset dcmds, below.
The execution control built-in dcmds, described below, are always available, but issues an error message indicating they are not supported if applied to a target that does not support execution control. For more information about the interaction of exec, attach, release, and job control with debugger execution control, refer to NOTES, below.
The ::evset dcmd and event tracing dcmds allow you to associate an event callback (using the -c option) with each event specifier. The event callbacks are strings that represent mdb commands to execute when the corresponding event occurs in the target. These commands are executed as if they had been typed at the command prompt. Before executing each callback, the dot variable is set to the value of the representative thread's program counter and the “hits” variable is set to the number of times this specifier has been matched, including the current match.
If the event callbacks themselves contain one or more commands to continue the target (for example, ::cont or ::step), these commands do not immediately continue the target and wait for it to stop again. Instead, inside of an event callback, the continue dcmds note that a continue operation is now pending, and then return immediately. Therefore, if multiple dcmds are included in an event callback, the step or continue dcmd should be the last command specified. Following the execution of all event callbacks, the target immediately resumes execution if all matching event callbacks requested a continue. If conflicting continue operations are requested, the operation with the highest precedence determines what type of continue occurs. The order of precedence from highest to lowest is: step, step-over (next), step-out, continue.
mdb provides facilities to examine the stacks and registers of each thread associated with the target. The persistent “thread” variable contains the current representative thread identifier. The format of the thread identifier depends on the target. The ::regs and ::fpregs dcmds can be used to examine the register set of the representative thread, or of another thread if its register set is currently available. In addition, the register set of the representative thread is exported as a set of named variables. The user can modify the value of one or more registers by applying the > dcmd to the corresponding named variable.
The mdb kernel target exports the virtual address of the corresponding internal thread structure as the identifier for a given thread. The Oracle Solaris Modular Debugger Guide provides more information on debugging support for threads in the Solaris kernel. The mdb process target provides proper support for examination of multi-threaded user processes that use the native lwp_* interfaces, /usr/lib/libthread.so or /usr/lib/lwp/libthread.so. When debugging a live user process, mdb detects if a single threaded process dlopens or closes libthread and automatically adjusts its view of the threading model on-the-fly. The process target thread identifiers corresponds to either the lwpid_t, thread_t, or pthread_t of the representative, depending on the threading model used by the application.
If mdb is debugging a user process target and the target makes use of compiler-supported thread-local storage, mdb automatically evaluates symbol names referring to thread-local storage to the address of the storage corresponding to the current representative thread. The ::tls built-in dcmd can be used to display the value of the symbol for threads other than the representative thread.
mdb provides a set of built-in dcmds that are always defined. Some of these dcmds are only applicable to certain targets: if a dcmd is not applicable to the current target, it fails and prints a message indicating “command is not supported by current target”. In many cases, mdb provides a mnemonic equivalent (::identifier) for the legacy adb(1) dcmd names. For example, ::quit is provided as the equivalent of $q. Programmers who are experienced with adb(1) or who appreciate brevity or arcana can prefer the $ or : forms of the built-ins. Programmers who are new to mdb might prefer the more verbose :: form. The built-ins are shown in alphabetical order. If a $ or : form has a ::identifier equivalent, it is shown underneath the ::identifier form. The built-in dcmds are:
Assign the value of dot to the specified named variable. Some variables are read-only and can not be modified. If the > is followed by a modifier character surrounded by / /, then the value is modified as part of the assignment. The modifier characters are:
unsigned char quantity (1-byte)
unsigned short quantity (2-byte)
unsigned int quantity (4-byte)
unsigned long quantity (4-byte in 32-bit, 8-byte in 64-bit)
Notice that these operators do not perform a cast. Instead, they fetch the specified number of low-order bytes (on little-endian architectures) or high-order bytes (big-endian architectures). Modifiers are provided for backwards compatibility; the mdb */modifier/ and %/modifier/ syntax should be used instead.
Read and execute commands from the specified macro file. The filename can be given as an absolute or relative path. If the filename is a simple name (that is, if it does not contain a '/'), mdb searches for it in the macro file include path. If another macro file is currently being processed, this file is closed and replaced with the new file.
Read and execute commands from the specified macro file (as with $<), but do not close the current open macro file.
Print the process-ID and current signal of the target if it is a user process or core file, and then print the general register set of the representative thread.
Print a C stack backtrace, including stack frame pointer information. If the dcmd is preceded by an explicit address, a backtrace beginning at this virtual memory address is displayed. Otherwise the stack of the representative thread is displayed. If an optional count value is given as an argument, no more than count arguments are displayed for each stack frame in the output.
Get or set the default output radix. If the dcmd is preceded by an explicit expression, the default output radix is set to the given base; otherwise the current radix is printed in base 10 (decimal). The default radix is base 16 (hexadecimal).
Print a list of all known external (global) symbols of type object or function, the value of the symbol, and the first 4 (32-bit mdb) or 8 (64-bit mdb) bytes stored at this location in the target's virtual address space. The ::nm dcmd provides more flexible options for displaying symbol tables.
Set the prompt to the specified prompt-string. The default prompt is '> '. The prompt can also be set using ::set -P or the -P command-line option.
Get or set the symbol matching distance for address-to-symbol-name conversions. The symbol matching distance modes are discussed along with the -s command-line option under OPTIONS. The symbol matching distance can also be modified using the ::set -s option. If no distance is specified, the current setting is displayed.
Print a list of the named variables that have non-zero values. The ::vars dcmd provides other options for listing variables.
Set the output page width to the specified value. Typically, this command is not necessary as mdb queries the terminal for its width and handles resize events.
Re-open the target for writing, as if mdb had been executed with the -w option on the command line. Write mode can also be enabled with the ::set -w option.
If the user process target is active, attach to and debug the specified process-ID or core file. The core file pathname should be specified as a string argument. The process-ID can be specified as the string argument, or as the value of the expression preceding the dcmd. Recall that the default base is hexadecimal, so decimal PIDs obtained using pgrep(1) or ps(1) should be preceded with “0t” when specified as expressions.
Set a breakpoint at the specified locations. The ::bp dcmd sets a breakpoint at each address or symbol specified, including an optional address specified by an explicit expression preceding the dcmd, and each string or immediate value following the dcmd. The arguments can either be symbol names or immediate values denoting a particular virtual address of interest. If a symbol name is specified, it can refer to a symbol that cannot yet be evaluated in the target process. That is, it can consist of an object name and function name in a load object that has not yet been opened. In this case, the breakpoint is deferred and is not active in the target until an object matching the given name is loaded. The breakpoint is automatically enabled when the load object is opened. Breakpoints on symbols defined in a shared library should always be set using a symbol name and not using an address expression, as the address can refer to the corresponding Procedure Linkage Table (PLT) entry instead of the actual symbol definition. Breakpoints set on PLT entries can be overwritten by the run-time link-editor when the PLT entry is subsequently resolved to the actual symbol definition. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd, as described below. If the :b form of the dcmd is used, a breakpoint is only set at the virtual address specified by the expression preceding the dcmd. The arguments following the :b dcmd are concatenated together to form the callback string. If this string contains meta-characters, it must be quoted.
Concatenate and display files. Each filename can be specified as a relative or absolute pathname. The file contents are printed to standard output, but are not passed to the output pager. This dcmd is intended to be used with the | operator; the programmer can initiate a pipeline using a list of addresses stored in an external file.
Suspend the debugger, continue the target program, and wait for it to terminate or stop following a software event of interest. If the target is already running because the debugger was attached to a running program with the -o nostop option enabled, this dcmd simply waits for the target to terminate or stop after an event of interest. If an optional signal name or number (see signal.h(3HEAD)) is specified as an argument, the signal is immediately delivered to the target as part of resuming its execution. If the SIGINT signal is traced, control can be asynchronously returned to the debugger by typing the user-defined interrupt character (usually ^C). This SIGINT signal is automatically cleared and is not observed by the target the next time it is continued. If no target program is currently running, ::cont starts a new program running as if by ::run.
Context switch to the specified process. A context switch operation is only valid when using the kernel target. The process context is specified using the address of its proc structure in the kernel's virtual address space. The special context address “0” is used to denote the context of the kernel itself. mdb can only perform a context switch when examining a crash dump if the dump contains the physical memory pages of the specified user process (as opposed to just kernel pages). The kernel crash dump facility can be configured to dump all pages or the pages of the current user process using dumpadm(1M). The ::status dcmd can be used to display the contents of the current crash dump.
When the user requests a context switch from the kernel target, mdb constructs a new target representing the specified user process. Once the switch occurs, the new target interposes its dcmds at the global level: thus the / dcmd now formats and displays data from the virtual address space of the user process, the ::mappings dcmd displays the mappings in the address space of the user process, and so on. The kernel target can be restored by executing 0::context.
List the available dcmds and print a brief description for each one.
Delete the event specifiers with the given id number. The id number argument is interpreted in decimal by default. If an optional address is specified preceding the dcmd, all event specifiers that are associated with the given virtual address are deleted (for example, all breakpoints or watchpoints affecting that address). If the special argument “all” is given, all event specifiers are deleted, except those that are marked sticky (T flag). The ::events dcmd displays the current list of event specifiers.
Disassemble starting at or around the address specified by the final argument, or the current value of dot. If the address matches the start of a known function, the entire function is disassembled. Otherwise, a “window” of instructions before and after the specified address is printed in order to provide context. By default, instructions are read from the target's virtual address space. If the -f option is present, instructions are read from the target's object file instead. The -f option is enabled by default if the debugger is not currently attached to a live process, core file, or crash dump. The -w option can be used to force “window”-mode, even if the address is the start of a known function. The size of the window defaults to ten instructions; the number of instructions can be specified explicitly using the -n option.
List the available disassembler modes. When a target is initialized, mdb attempts to select the appropriate disassembler mode. The user can change the mode to any of the modes listed using the ::dismode dcmd.
Get or set the disassembler mode. If no argument is specified, print the current disassembler mode. If a mode argument is specified, switch the disassembler to the specified mode. The list of available disassemblers can be displayed using the ::disasms dcmd.
List the loaded debugger modules. If the -l option is specified, the list of the dcmds and walkers associated with each dmod is printed below its name. The output can be restricted to a particular dmod by specifying its name as an additional argument.
Print a hexadecimal and ASCII memory dump of the 16-byte aligned region of memory containing the address specified by dot. If a repeat count is specified for ::dump, this is interpreted as a number of bytes to dump rather than a number of iterations. The ::dump dcmd also recognizes the following options:
Adjusts for endian-ness. The -e option assumes 4-byte words. The -g option can be used to change the default word size.
Reads data from the object file location corresponding to the given virtual address instead of from the target's virtual address space. The -f option is enabled by default if the debugger is not currently attached to a live process, core file, or crash dump.
Displays bytes in groups of bytes. The default group size is 4 bytes. The group size must be a power of two that divides the line width.
Interprets address as a physical address location in the target's address space instead of a virtual address.
Does not print an ASCII decoding of the data.
Numbers lines relative to the start address instead of with the explicit address of each line. This option implies the -u option.
Elides repeated lines.
Only reads from and displays the contents of the specified addresses, instead of reading and printing entire lines.
Unaligns output instead of aligning the output at a paragraph boundary.
Displays paragraphs at 16-byte paragraphs per line. The default number of paragraphs is one. The maximum value accepted for -w is 16.
Print the arguments separated by blanks and terminated by a NEWLINE to standard output. Expressions enclosed in $[ ] is evaluated to a value and printed in the default base.
Evaluate and execute the specified string as a command. If the command contains metacharacters or whitespace, it should be enclosed in double or single quotes.
Display the list of software event specifiers. Each event specifier is assigned a unique ID number that can be used to delete or modify it at a later time. The debugger can also have its own internal events enabled for tracing. These events are only be displayed if the -a option is present. If the -v option is present, a more verbose display, including the reason for any specifier inactivity, are shown. Here is some sample output:
> ::events ID S TA HT LM Description Action ----- - -- -- -- -------------------------------- ------ [ 1 ] - T 1 0 stop on SIGINT - [ 2 ] - T 0 0 stop on SIGQUIT - [ 3 ] - T 0 0 stop on SIGILL - ... [ 11] - T 0 0 stop on SIGXCPU - [ 12] - T 0 0 stop on SIGXFSZ - [ 13] - 2 0 stop at libc`printf ::echo printf >
The following table explains the meaning of each column. A summary of this information is available using ::help events.
The event specifier identifier. The identifier is shown in square brackets [ ] if the specifier is enabled, in parentheses ( ) if the specifier is disabled, or in angle brackets < > if the target program is currently stopped on an event that matches the given specifier.
The event specifier state. The state is one of the following symbols:
The event specifier is idle. When no target program is running, all specifiers are idle. When the target program is running, a specifier can be idle if it cannot be evaluated (for example, a deferred breakpoint in a shared object that is not yet loaded).
The event specifier is active. When the target is continued, events of this type is detected by the debugger.
The event specifier is armed. This state means that the target is currently running with instrumentation for this type of event. This state is only visible if the debugger is attached to a running program with the -o nostop option.
The event specifier was not armed due to an operating system error. The ::events -v option can be used to display more information about the reason the instrumentation failed.
The Temporary, Sticky, and Automatic event specifier properties. One or more of the following symbols can be shown:
The event specifier is temporary, and is deleted the next time the target stops, regardless of whether it is matched.
The event specifier is sticky, and is not be deleted by ::delete all or :z. The specifier can be deleted by explicitly specifying its id number to ::delete.
The event specifier is automatically disabled when the hit count is equal to the hit limit.
The event specifier is automatically deleted when the hit count is equal to the hit limit.
The target automatically stops when the hit count is equal to the hit limit.
The current hit count. This column displays the number of times the corresponding software event has occurred in the target since the creation of this event specifier.
The current hit limit. This column displays the limit on the hit count at which the auto-disable, auto-delete, or auto-stop behavior takes effect. These behaviors can be configured using the ::evset dcmd, described below.
A description of the type of software event that is matched by the given specifier.
The callback string to execute when the corresponding software event occurs. This callback is executed as if it had been typed at the command prompt.
Modify the properties of one or more software event specifiers. The properties are set for each specifier identified by the optional expression preceding the dcmd and an optional list of arguments following the dcmd. The argument list is interpreted as a list of decimal integers, unless an explicit radix is specified. The ::evset dcmd recognizes the following options:
Disables the event specifier when the hit count reaches the hit limit. If the -d form of the option is given, this behavior is disabled. Once an event specifier is disabled, the debugger removes any corresponding instrumentation and ignores the corresponding software events until the specifier is subsequently re-enabled. If the -n option is not present, the specifier is disabled immediately.
Deletes the event specifier when the hit count reaches the hit limit. If the -D form of the option is given, this behavior is disabled. The -D option takes precedence over the -d option. The hit limit can be configured using the -n option.
Enables the event specifier. If the -e form of the option is given, the specifier is disabled.
Stops the target program when the hit count reaches the hit limit. If the -s form of the option is given, this behavior is disabled. The -s behavior tells the debugger to act as if the ::cont were issued following each execution of the specifier's callback, except for the Nth execution, where N is the current value of the specifier's hit limit. The -s option takes precedence over both the -D option and the -d option.
Marks the event specifier as temporary. Temporary specifiers are automatically deleted the next time the target stops, regardless of whether it stopped as the result of a software event corresponding to the given specifier. If the -t form of the option is given, the temporary marker is removed. The -t option takes precedence over the -T option.
Marks the event specifier as sticky. Sticky specifiers are not deleted by ::delete all or :z. They can be deleted by specifying the corresponding specifier ID as an explicit argument to ::delete. If the -T form of the option is given, the sticky property is removed. The default set of event specifiers are all initially marked sticky.
Executes the specified cmd string each time the corresponding software event occurs in the target program. The current callback string can be displayed using ::events.
Sets the current value of the hit limit to count. If no hit limit is currently set and the -n option does not accompany -s or D, the hit limit is set to one.
A summary of this information is available using ::help evset.
Print a list of the known source files (symbols of type STT_FILE present in the various target symbol tables).
Trace the specified machine faults. The faults are identified using an optional fault number preceding the dcmd, or a list of fault names or numbers (see <sys/fault.h>) following the dcmd. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd.
Print the floating-point register set of the representative thread. If a thread is specified, the floating point registers of that thread are displayed. The thread expression should be one of the thread identifiers described under Thread Support, above.
List the available output format characters for use with the /, \, ?, and = formatting dcmds. The formats and their use is described under Formatting dcmds, above.
Evaluate the specified command string, and then print the old value of dot if the new value of dot is non-zero. If the command contains whitespace or metacharacters, it must be quoted. The ::grep dcmd can be used in pipelines to filter a list of addresses.
With no arguments, the ::help dcmd prints a brief overview of the help facilities available in mdb. If a dcmd-name is specified, mdb prints a usage summary for that dcmd.
If the target is a live user process, ignore the specified signal and allow it to be delivered transparently to the target. All event specifiers that are tracing delivery of the specified signal is deleted from the list of traced events. By default, the set of ignored signals is initialized to the complement of the set of signals that cause a process to dump core by default (see signal.h(3HEAD)), except for SIGINT, which is traced by default.
Display the list of signals that are ignored by the debugger and that is handled directly by the target. More information on traced signals can be obtained using the ::events dcmd.
Evaluate the tests and then print the old value of dot if the tests are true.
Use physical rather than virtual addresses.
There are two kinds of tests. The first allows you to test the value of a member of a structure or union. The second allows you to test an offset from dot. These tests can then be joined with AND or OR to build more complex tests.
Start with an example of testing a member of a structure:
::if "struct foo" namep <> 0
This statement would print the old value of dot if the element namep in structure foo were not 0.
::if "struct foo" namep <> 0 AND namep->name <> 0 AND namep->name streq "bar"
The preceding statement would print the old value of dot if the element name in structure pointed to by namep in foo were not 0 and pointed to a string that contained the string bar.
::if "struct foo" name <> 0 AND name streq "bar" and value = 0x123
The preceding statement would print only if the name were bar and value was 0x123.
The second form tests to see if the type at the given offset has the given value. Here is an example:
::if uint_t at 0x34 = 0x123
This will print if the value of an uint_t at offset 0x34 is 0x123. Again this can be joined with other tests using AND or OR. With this form, ::if has a number of built-in types that are internal that can be used even when there is no symbolic type information. These types are:
char, uchar, short, ushort, int, uint, long, ulong, longlong, ulonglong, pointer, addr
If the type is addr, then, rather than the value at that address, the actual address is tested. So, the statement:
::if addr at 0x34 = 0x100034
...would report the value of dot only if dot+0x34 was 0x100034, whereas:
::if pointer at 0x34 = 0x100034
...would print only if the value at dot+0x34.
More usefully, the statement:
::if "char *' at 0x34 streq "foo"
...will print the old value of dot if the string pointed to at offset 0x34 is foo.
Possible tests are:
= - True if the values are equal.
<> - True if the values are not equal.
< - True if the value is less than the right hand value.
<= - True if the value is less than or equal to the right
hand value.
> - True if the value is greater than the right hand value.
>= - True if the value is greater than or equal to the right
hand value.
& - True if the values ANDed together are non zero.
^ - True if the values XORed together are non zero.
streq - True if the strings exactly match.
strneq - True if the strings don't match.
strcaseeq - True if the strings match case insensitively.
strncaseeq - True if the strings do not match case insensitively.
strleneq - True if the string is this length.
strlenneq - True if the string is not this length.
strlengt - True if the string is longer than this.
strlenlt - True if the string is shorter than this.
strlenge - True if the string is this long or longer.
strlenle - True if the string is this long or shorterThe right hand side of a test can be any one of the following:
An element in the structure or union.
The value from a symbol.
An absolute value.
<var — the value of the variable var.
$expr - the value of dot after mdb expression expr is evaluated. This must be quoted using double quotation marks (“"). For example:
"$<var=J"
...would get the value of variable var.
Forcibly terminate the target if it is a live user process. The target is also forcibly terminated when the debugger exits if it was created by the debugger using ::run.
Print the LWPID of the representative thread, if the target is a user process.
Print the LWPIDs of each LWP in the target, if the target is a user process.
Walk through the elements of a linked list data structure and print the address of each element in the list. The address of the first element in the list can be specified using an optional address. Otherwise, the list is assumed to start at the current value of dot. The type parameter must name a C struct or union type and is used to describe the type of the list elements so that mdb can read in objects of the appropriate size. If ::list can determine the type then type can be omitted. The member parameter is used to name the member of type that contains a pointer to the next list element. The ::list dcmd continues iterating until a NULL pointer is encountered, the first element is reached again (a circular list), a loop is detection is enabled (-L) and a loop is detected, back member checking is enabled (-b) and a bad back pointer is detected, or an error occurs while reading an element. If the optional variable-name is specified, the specified variable is assigned the value returned at each step of the walk when mdb invokes the next stage of a pipeline.
Verify that back_member points to the previous member of the list. The first member of the list is allowed to have a NULL back pointer.
Check for loops in the list. Will print every element in the list once.
Read from physical rather than virtual addresses.
The ::list dcmd can only be used with objects that contain symbolic debugging information designed for use with mdb. Refer to NOTES, “Symbolic Debugging Information,” below for more information.
Load the specified dmod. The module name can be given as an absolute or relative path. If module-name is a simple name (that is, does not contain a '/'), mdb searches for it in the module library path. Modules with conflicting names can not be loaded; the existing module must be unloaded first. If the -s option is present, mdb remains silent and not issue any error messages if the module is not found or could not be loaded.
Enable or disable the output log. mdb provides an interactive logging facility where both the input commands and standard output can be logged to a file while still interacting with the user. The -e option enables logging to the specified file, or re-enables logging to the previous log file if no filename is given. The -d option disables logging. If the $> dcmd is used, logging is enabled if a filename argument is specified; otherwise, logging is disabled. If the specified log file already exists, mdb appends any new log output to the file.
Map the value of dot to a corresponding value using the command specified as a string argument, and then print the new value of dot. If the command contains whitespace or metacharacters, it must be quoted. The ::map dcmd can be used in pipelines to transform the list of addresses into a new list of addresses.
Print a list of each mapping in the target's virtual address space, including the address, size, and description of each mapping. If the dcmd is preceded by an address, mdb only shows the mapping that contains the given address. If a string name argument is given, mdb only shows the mapping matching that description.
Step the target program one instruction, but step over subroutine calls. If an optional signal name or number (see signal.h(3HEAD)) is specified as an argument, the signal is immediately delivered to the target as part of resuming its execution. If no target program is currently running, ::next starts a new program running as if by ::run and stop at the first instruction.
Print the symbol tables associated with the current target. If an optional address preceding the dcmd is specified, only the symbol table entry for the symbol corresponding to address is displayed. If an object is specified, only the symbol table for this load object is displayed. The ::nm dcmd also recognizes the following options:
Prints .dynsym (dynamic symbol table) instead of .symtab.
Prints the private symbol table instead of .symtab.
Prints value and size fields in decimal.
Prints only global symbols.
Suppresses the header line.
Sorts symbols by name.
Prints value and size fields in octal.
Prints symbols as a series of ::nmadd commands. This option can be used with -P to produce a macro file that can be subsequently read into the debugger with $<.
Prints only undefined symbols.
Sorts symbols by value.
Prints value and size fields in hexadecimal.
Prints only symbols of the specified type(s). The valid type argument strings are:
STT_NOTYPE
STT_OBJECT
STT_FUNC
STT_SECTION
STT_FILE
STT_COMMON
STT_TLS
STT_SPARC_REGISTER
Prints only the specified symbol information. The valid format argument strings are:
symbol table index
symbol value
size in bytes
symbol type
binding
other
section index
symbol name
C type for symbol (if known)
object which defines symbol
Add the specified symbol name to the private symbol table. mdb provides a private, configurable symbol table that can be used to interpose on the target's symbol table, as described under Symbol Name Resolution above. The ::nmadd dcmd also recognizes the following options:
Sets the size of the symbol to end - value.
Sets the type of the symbol to STT_FUNC.
Sets the type of the symbol to STT_OBJECT.
Sets the size of the symbol to size.
Delete the specified symbol name from the private symbol table.
Print a map of the target's virtual address space, showing only those mappings that correspond to the primary mapping (usually the text section) of each of the known load objects. The -v option displays the version of each load object. Version information is not available for all load objects. Load objects without version information is listed as having a version of “Unknown” in the output for the -v option.
Print the offset of the specified member of the specified type. The type should be the name of a C structure. If no member is specified all the members for this type are reported.
The offset is printed in bytes, unless the member is a bit-field, in which case the offset can be printed in bits. The output is always suffixed with the appropriate units for clarity. The type name can use the backquote (`) scoping operator described under Symbol Name Resolution, above. The ::offsetof dcmd can only be used with objects that contain symbolic debugging information designed for use with mdb. Refer to NOTES, Symbolic Debugging Information, below for more information.
Print the data structure at the specified virtual address using the given type information. The type parameter can name a C struct, union, enum, fundamental integer type, or a pointer to any of these types. If the type name contains whitespace (for example, “struct foo”), it must be enclosed in single or double quotes. The type name can use the backquote (`) scoping operator described under Symbol Name Resolution, above. If the type is a structured type, the ::print dcmd recursively prints each member of the struct or union. If the type argument is not present and a static or global STT_OBJECT symbol matches the address, ::print infers the appropriate type automatically. If the type argument is specified, it can be followed by an optional list of member expressions, in which case only those members and submembers of the specified type are displayed. If type contains other structured types, each member string can refer to a sub-structure element by forming a list of member names separated by period ('.') delimiters. The ::print dcmd can only be used with objects that contain symbolic debugging information designed for use with mdb. Refer to NOTES, Symbolic Debugging Information, below for more information. After displaying the data structure, ::print increments dot by the size of type in bytes.
If the -a option is present, the address of each member is displayed. If the -p option is present, ::print interprets address as a physical memory address instead of a virtual memory address. If the -t option is present, the type of each member is displayed. If the -d or -x options are present, all integers are displayed in decimal (-d) or hexadecimal (-x). By default, a heuristic is used to determine if the value should be displayed in decimal or hexadecimal. The number of characters in a character array that is read and displayed as a string can be limited with the -c option. If the -C option is present, no limit is enforced. The number of elements in a standard array that is read and displayed can be limited with the -l option. If the -L option is present, no limit is enforced and all array elements are shown. The default values for -c and -l can be modified using ::set or the -o command-line option as described under OPTIONS.
If the -i option is specified, the address value is interpreted as an immediate value to be printed. You must give a type with which to interpret the value. If the type is smaller than 64 bits, the immediate value is interpreted as if it were the size of the type. The -i option cannot be used in conjunction with the -p option. If the -a option is given, the addresses shown are byte offsets starting at zero.
Quit the debugger.
Print the general purpose register set of the representative thread. If a thread is specified, the general purpose register set of that thread is displayed. The thread expression should be one of the thread identifiers described under Thread Support, above.
Release the previously attached process or core file. If the -a option is present, the process is released and left stopped and abandoned. It can subsequently be continued by prun(1) (see proc(1)) or it can be resumed by applying mdb or another debugger. By default, a released process is forcibly terminated if it was created by mdb using ::run, or it is released and set running if it was attached to by mdb using the -p option or using the ::attach or :A dcmds.
Start a new target program running with the specified arguments and attach to it. The arguments are not interpreted by the shell. If the debugger is already examining a live running program, it first detaches from this program as if by ::release.
Get or set miscellaneous debugger properties. If no options are specified, the current set of debugger properties is displayed. The ::set dcmd recognizes the following options:
Forcibly takes over the next user process that ::attach is applied to, as if mdb had been executed with the -F option on the command line.
Sets the default path for locating macro files. The path argument can contain any of the special tokens described for the -I command-line option under OPTIONS.
Sets the default path for locating debugger modules. The path argument can contain any of the special tokens described for the -I command-line option under OPTIONS.
Enables the specified debugger option. If the -o form is used, the option is disabled. The option strings are described along with the -o command-line option under OPTIONS.
Sets the command prompt to the specified prompt string.
Sets the symbol matching distance to the specified distance. Refer to the description of the -s command-line option under OPTIONS for more information.
Re-opens the target for writing, as if mdb had been executed with the -w option on the command line.
Display revision information for the hardware and software. With no options specified, general system information is displayed. The -v option displays version information for all load objects, whereas the -p option displays the version information only for the load objects that have been installed on the system as part of a patch. Version information is not available for all load objects. Load objects without version information is omitted from the output for the -p option and is listed as having a version of “Unknown” in the output for the -v option.
Trace delivery of the specified signals. The signals are identified using an optional signal number preceding the dcmd, or a list of signal names or numbers (see signal.h(3HEAD)) following the dcmd. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd. Initially, the set of signals that cause the process to dump core by default (see signal.h(3HEAD)) and SIGINT are traced.
Options are as follows:
Show types for entries of this size.
Show types for entries in range of sizes.
Print the size of the specified type in bytes. The type parameter can name a C struct, union, enum, fundamental integer type, or a pointer to any of these types. The type name can use the backquote (`) scoping operator described under Symbol Name Resolution, above. member can be specified with standard C syntax using the array indexing operator “[index]”, structure member operator “.”.
The ::sizeof dcmd can only be used with objects that contain symbolic debugging information designed for use with mdb. Refer to NOTES, Symbolic Debugging Information, below for more information.
Print a C stack backtrace. If the dcmd is preceded by an explicit address, a backtrace beginning at this virtual memory address is displayed. Otherwise the stack of the representative thread is displayed. If an optional count value is given as an argument, no more than count arguments are displayed for each stack frame in the output.
Print a summary of information related to the current target.
Step the target program one instruction. If an optional signal name or number (see signal.h(3HEAD)) is specified as an argument, the signal is immediately delivered to the target as part of resuming its execution. If the optional “over” argument is specified, ::step steps over subroutine calls. The ::step over argument is the same as the ::next dcmd. If the optional “out” argument is specified, the target program continues until the representative thread returns from the current function. If no target program is currently running, ::step out starts a new program running as if by ::run and stop at the first instruction. The :s dcmd is the same as ::step. The :u dcmd is the same as ::step out.
Trace entry to or exit from the specified system calls. The system calls are identified using an optional system call number preceding the dcmd, or a list of system call names or numbers (see <sys/syscall.h>) following the dcmd. If the -i option is specified (the default), the event specifiers trigger on entry into the kernel for each system call. If the -o option is specified, the event specifiers trigger on exit out from the kernel. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd.
Print the address of the storage for the specified thread-local storage (TLS) symbol in the context of the specified thread. The thread expression should be one of the thread identifiers described under Thread Support, above. The symbol name can use any of the scoping operators described under Symbol Name Resolution, above.
Set attributes for named variables. If one or more variable names are specified, they are defined and set to the value of dot. If the -t option is present, the user-defined tag associated with each variable is set. If the -t option is present, the tag is cleared. If no variable names are specified, the list of variables and their values is printed.
Unload the specified dmod. The list of active dmods can be printed using the ::dmods dcmd. Built-in modules can not be unloaded. Modules that are busy (that is, provide dcmds that are currently executing) can not be unloaded.
Unset (remove) the specified variable(s) from the list of defined variables. Some variables exported by mdb are marked as persistent, and can not be unset by the user.
Print a listing of named variables. If the -n option is present, the output is restricted to variables that currently have non-zero values. If the -p option is present, the variables are printed in a form suitable for re-processing by the debugger using the $< dcmd. This option can be used to record the variables to a macro file and then restore these values later. If the -t option is present, only the tagged variables are printed. Variables can be tagged using the -t option of the ::typeset dcmd.
Print the debugger version number.
Print the physical address mapping for the specified virtual address, if possible. The ::vtop dcmd is only available when examining a kernel target, or when examining a user process inside a kernel crash dump (after a ::context dcmd has been issued).
When examining a kernel target from the kernel context, the -a option can be used to specify the address (as) of an alternate address space structure that should be used for the virtual to physical translation. By default, the kernel's address space is used for translation. This option is available for active address spaces even when the dump content only contains kernel pages.
Walk through the elements of a data structure using the specified walker. The available walkers can be listed using the ::walkers dcmd. Some walkers operate on a global data structure and do not require a starting address. For example, walk the list of proc structures in the kernel. Other walkers operate on a specific data structure whose address must be specified explicitly. For example, given a pointer to an address space, walk the list of segments. When used interactively, the ::walk dcmd prints the address of each element of the data structure in the default base. The dcmd can also be used to provide a list of addresses for a pipeline. The walker name can use the backquote (`) scoping operator described under dcmd and Walker Name Resolution, above. If the optional variable-name is specified, the specified variable is assigned the value returned at each step of the walk when mdb invokes the next stage of the pipeline.
List the available walkers and print a brief description for each one.
Print the dmod that exports the specified dcmds and walkers. These dcmds can be used to determine which dmod is currently providing the global definition of the given dcmd or walker. Refer to the section on dcmd and Walker Name Resolution above for more information on global name resolution. The -v option causes the dcmd to print the alternate definitions of each dcmd and walker in order of precedence.
Set a watchpoint at the specified address. The length in bytes of the watched region can be set by specifying an optional repeat count preceding the dcmd. If no length is explicitly set, the default is one byte. The ::wp dcmd allows the watchpoint to be configured to trigger on any combination of read (-r option), write (-w option), or execute (-x option) access. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd. The :a dcmd sets a read access watchpoint at the specified address. The :p dcmd sets an execute access watchpoint at the specified address. The :w dcmd sets a write access watchpoint at the specified address. The arguments following the :a, :p, and :w dcmds are concatenated together to form the callback string. If this string contains meta-characters, it must be quoted.
List the external data buffers exported by the current target. External data buffers represent information associated with the target that can not be accessed through standard target facilities (that is, an address space, symbol table, or register set). These buffers can be consumed by dcmds; for more information, refer to the Oracle Solaris Modular Debugger Guide.
Delete all event specifiers from the list of traced software events. Event specifiers can also be deleted using ::delete.
The following options are supported:
Disables automatic loading of mdb modules. By default, mdb attempts to load debugger modules corresponding to the active shared libraries in a user process or core file, or to the loaded kernel modules in the live operating system or an operating system crash dump.
Forces raw file debugging mode. By default, mdb attempts to infer whether the object and core file operands refer to a user executable and core dump or to a pair of operating system crash dump files. If the file type cannot be inferred, the debugger defaults to examining the files as plain binary data. The -f option forces mdb to interpret the arguments as a set of raw files to examine.
Forcibly takes over the specified user process, if necessary. By default, mdb refuses to attach to a user process that is already under the control of another debugging tool, such as truss(1). With the -F option, mdb attaches to these processes anyway. This can produce unexpected interactions between mdb and the other tools attempting to control the process.
Sets default path for locating macro files. Macro files are read using the $< or $<< dcmds. The path is a sequence of directory names delimited by colon (:) characters. The -I include path and -L library path (see below) can also contain any of the following tokens:
Expands to the current instruction set architecture (ISA) name ('sparc', 'sparcv9', or 'i386').
Expands to the old value of the path being modified. This is useful for appending or prepending directories to an existing path.
Expands to the current platform string (either uname -i or the platform string stored in the process core file or crash dump).
Expands to the pathname of the root directory. An alternate root directory can be specified using the -R option. If no -R option is present, the root directory is derived dynamically from the path to the mdb executable itself. For example, if /bin/mdb is executed, the root directory is /. If /net/hostname/bin/mdb were executed, the root directory would be derived as /net/hostname.
Expands to the name of the current target. This is either be the literal string 'proc' (a user process or user process core file), 'kvm' (a kernel crash dump or the live operating system), or 'raw' (a raw file).
The default include path for 32-bit mdb is:
%r/usr/platform/%p/lib/adb:%r/usr/lib/adb
The default include path for 64-bit mdb is:
%r/usr/platform/%p/lib/adb/%i:%r/usr/lib/adb/%i
Forces kernel debugging mode. By default, mdb attempts to infer whether the object and core file operands refer to a user executable and core dump, or to a pair of operating system crash dump files. The -k option forces mdb to assume these files are operating system crash dump files. If no object or core operand is specified, but the -k option is specified, mdb defaults to an object file of /dev/ksyms and a core file of /dev/kmem. Read access to /dev/kmem is restricted to group sys. Write access requires ALL privileges.
Load kmdb, stop the live running operating system kernel, and proceed to the kmdb debugger prompt. This option should only be used on the system console, as the subsequent kmdb prompt appears on the system console.
Sets default path for locating debugger modules. Modules are loaded automatically on startup or using the ::load dcmd. The path is a sequence of directory names delimited by colon (:) characters. The -L library path can also contain any of the tokens shown for -I above.
Disables demand-loading of kernel module symbols. By default, mdb processes the list of loaded kernel modules and performs demand loading of per-module symbol tables. If the -m option is specified, mdb does not attempt to process the kernel module list or provide per-module symbol tables. As a result, mdb modules corresponding to active kernel modules are not loaded on startup.
Preloads all kernel module symbols. By default, mdb performs demand-loading for kernel module symbols: the complete symbol table for a module is read when an address is that module's text or data section is referenced. With the -M option, mdb loads the complete symbol table of all kernel modules during startup.
Enables the specified debugger option. If the -o form of the option is used, the specified option is disabled. Unless noted below, each option is off by default. mdb recognizes the following option arguments:
Enables stricter adb(1) compatibility. The prompt is set to the empty string and many mdb features, such as the output pager, is disabled.
Sets the default limit on the number of array members that ::print displays. If limit is the special token none, all array members are displayed by default.
Sets the default limit on the number of characters that ::print attempts to display as an ASCII string when printing a char array. If limit is the special token none, the entire char array is displayed as a string by default.
Sets the debugger behavior for following an exec(2) system call. The mode should be one of the following named constants:
If stdout is a terminal device, the debugger stops after the exec(2) system call has returned and then prompts the user to decide whether to follow the exec or stop. If stdout is not a terminal device, the ask mode defaults to stop.
The debugger follows the exec by automatically continuing the target process and resetting all of its mappings and symbol tables based on the new executable. The follow behavior is discussed in more detail under NOTES, Interaction with Exec, below.
The debugger stops following return from the exec system call. The stop behavior is discussed in more detail under NOTES, Interaction with Exec, below.
Sets the debugger behavior for following a fork(2), fork1(2), or vfork(2) system call. The mode should be one of the following named constants:
If stdout is a terminal device, the debugger stops after the fork(2) system call has returned and then prompts the user to decide whether to follow the parent or child. If stdout is not a terminal device, the ask mode defaults to parent.
The debugger follows the parent process, and detaches from the child process and sets it running.
The debugger follows the child process, and detaches from the parent process and sets it running.
The debugger does not exit when an EOF sequence (^D) is entered at the terminal. The ::quit dcmd must be used to quit.
Does not stop a user process when attaching to it when the -p option is specified or when the ::attach or :A dcmds are applied. The nostop behavior is described in more detail under NOTES, Process Attach and Release, below.
Enables the output pager (default).
If a NEWLINE is entered as the complete command at the terminal, mdb repeats the previous command with the current value of dot. This option is implied by -o adb.
mdb provides support for symbol naming and identification in user applications that make use of link maps other than LM_ID_BASE and LM_ID_LDSO, as described in Symbol Name Resolution, above. Symbols on link maps other than LM_ID_BASE or LM_ID_LDSO is shown as LMlmid`library`symbol, where lmid is the link-map ID in the default output radix (16). The user can optionally configure mdb to show the link-map ID scope of all symbols and objects, including those associated with LM_ID_BASE and LM_ID_LDSO, by enabling the showlmid option. Built-in dcmds that deal with object file names displays link-map IDs according to the value of showlmid above, including ::nm, ::mappings, $m, and ::objects.
Attaches to and stops the specified process-id. mdb uses the /proc/pid/object/a.out file as the executable file pathname.
Sets the command prompt. The default prompt is '> '.
Sets root directory for pathname expansion. By default, the root directory is derived from the pathname of the mdb executable itself. The root directory is substituted in place of the %r token during pathname expansion.
Sets the symbol matching distance for address-to-symbol-name conversions to the specified distance. By default, mdb sets the distance to zero, which enables a smart-matching mode. Each ELF symbol table entry includes a value V and size S, representing the size of the function or data object in bytes. In smart mode, mdb matches an address A with the given symbol if A is in the range [ V, V + S ). If any non-zero distance is specified, the same algorithm is used, but S in the expression above is always the specified absolute distance and the symbol size is ignored.
Suppresses processing of the user's ~/.mdbrc file. By default, mdb reads and processes the macro file .mdbrc if one is present in the user's home directory, as defined by $HOME. If the -S option is present, this file is not read.
Forces user debugging mode. By default, mdb attempts to infer whether the object and core file operands refer to a user executable and core dump, or to a pair of operating system crash dump files. The -u option forces mdb to assume these files are not operating system crash dump files.
Unload kmdb if it is loaded. You should unload kmdb when it is not in use to release the memory used by the kernel debugger back to the free memory available to the operating system.
Sets disassembler version. By default, mdb attempts to infer the appropriate disassembler version for the debug target. The disassembler can be set explicitly using the -V option. The ::disasms dcmd lists the available disassembler versions.
Opens the specified object and core files for writing.
Permit access to memory addresses that are mapped to I/O devices. By default, mdb does not allow such access because many devices do not provide hardware protection against invalid software manipulations. Use this option only when debugging device drivers and with caution.
Sends explicit terminal initialization sequences for tty mode. Some terminals, such as cmdtool(1), require explicit initialization sequences to switch into a tty mode. Without this initialization sequence, terminal features such as standout mode can not be available to mdb.
The following operands are supported:
Specifies an ELF format object file to examine. mdb provides the ability to examine and edit ELF format executables (ET_EXEC), ELF dynamic library files (ET_DYN), ELF relocatable object files (ET_REL), and operating system unix.X symbol table files.
Specifies an ELF process core file (ET_CORE), or an operating system crash dump vmcore.X file. If an ELF core file operand is provided without a corresponding object file, mdb attempts to infer the name of the executable file that produced the core using several different algorithms. If no executable is found, mdb still executes, but some symbol information can be unavailable.
Specifies the numerical suffix representing a pair of operating system crash dump files. For example, if the suffix is '3', mdb infers that it should examine the files 'unix.3' and 'vmcore.3'. If these files do not exist, but 'vmdump.3' does exist, then a message is printed indicating that savecore -f vmdump.3 must be run first in order to uncompress the dump file. The string of digits are not interpreted as a suffix if an actual file of the same name is present in the current directory.
mdb processes all input files (including scripts, object files, core files, and raw data files) in a large file aware fashion. See largefile(5) for more information about the processing of large files, which are files greater than or equal to 2 Gbytes (231 bytes).
The following exit values are returned:
Debugger completed execution successfully.
A fatal error occurred.
Invalid command line options were specified.
This variable is used to determine the maximum length of the command history list. If this variable is not present, the default length is 128.
This variable is used to determine the pathname of the user's home directory, where a .mdbrc file can reside. If this variable is not present, no .mdbrc processing occurs.
This variable is used to determine the pathname of the shell used to process shell escapes requested using the ! meta-character. If this variable is not present, /bin/sh is used.
User mdb initialization file. The .mdbrc file, if present, is processed after the debug target has been initialized, but before module auto-loading is performed or any commands have been read from standard input.
Kernel virtual memory image device. This device special file is used as the core file when examining the live operating system.
Kernel symbol table device. This device special file is used as the object file when examining the live operating system.
Process information files that are read when examining and controlling user processes.
Default directories for macro files that are read with the $< and $<< dcmds. platform-name is the name of the platform, derived either from information in a core file or crash dump, or from the current machine as if by uname -i (see uname(1)).
Default directories for debugger modules that are loaded using the ::load dcmd. platform-name is the name of the platform, derived either from information in a core file or crash dump, or from the current machine as if by uname -i (see uname(1)).
See attributes(5) for descriptions of the following attributes:
|
adb(1), cmdtool(1), gcore(1), proc(1), pgrep(1), ps(1), stty(1), truss(1), uname(1), coreadm(1M), dumpadm(1M), largefile(5), savecore(1M), exec(2), fork(2), _lwp_self(2), pipe(2), vfork(2), dlopen(3C), elf(3ELF), libc_db(3LIB), libkvm(3LIB), libthread(3LIB), signal(3C), signal.h(3HEAD), thr_self(3C), core(4), proc(4), attributes(5), largefile(5), threads(5), ksyms(7D), mem(7D)
Oracle Solaris Modular Debugger Guide
The debugger and its dmods execute in the same address space, and thus it is quite possible that a buggy dmod can cause mdb to dump core or otherwise misbehave. The mdb resume capability, described above under Signal Handling, provides a limited recovery mechanism for these situations. However, it is not possible for mdb to know definitively whether the dmod in question has corrupted only its own state, or the debugger's global state. Therefore a resume operation cannot be guaranteed to be safe, or to prevent a subsequent crash of the debugger. The safest course of action following a resume is to save any important debug information, and then quit and restart the debugger.
The use of the debugger to modify (that is, write to) the address space of live running operating system is extremely dangerous, and can result in a system panic in the event the user damages a kernel data structure.
mdb does not provide support for examining process core files that were generated by a release of Solaris preceding Solaris 2.6. When debugging core files generated by a release of Solaris 9 or an earlier release, symbol information might not be available. Since the text section and read-only data is not present in those core files, the symbol information might not match the data present in the process at the time it dumped core. In releases later than Solaris 9, text sections and read-only data are included in core files by default. Users can configure their processes to exclude that information from core files using coreadm(1M). Thus, the information presented by mdb for those core files can not match the data that was present at the time the process dumped core. Core files from Solaris x86 systems can not be examined on Solaris SPARC systems, and vice-versa.
Crash dumps from Solaris 7 and earlier releases can only be examined with the aid of the libkvm from the corresponding operating system release. If a crash dump from one operating system release is examined using the dmods from a different operating system release, changes in the kernel implementation can prevent some dcmds or walkers from working properly. mdb issues a warning message if it detects this condition. Crash dumps from Solaris x86 systems can not be examined on Solaris SPARC systems, and vice-versa.
mdb provides support for debugging both 32-bit and 64-bit programs. Once it has examined the target and determined its data model, mdb automatically re-executes the mdb binary that has the same data model as the target, if necessary. This approach simplifies the task of writing debugger modules, because the modules that are loaded use the same data model as the primary target. Only the 64-bit debugger can be used to debug 64-bit target programs. The 64-bit debugger can only be used on a system that is running the 64-bit operating environment.
The debugger can also need to re-execute itself when debugging a 32-bit process that execs a 64-bit process, or vice-versa. The handling of this situation is discussed in more detail under Interaction with Exec, below.
When a controlled process performs a successful exec(2), the behavior of the debugger is controlled by the ::set -o follow_exec_mode option, as described above. If the debugger and victim process have the same data model, then the “stop” and “follow” modes determine whether mdb automatically continues the target or returns to the debugger prompt following the exec. If the debugger and victim process have a different data model, then the “follow” behavior causes mdb to automatically re-exec the mdb binary with the appropriate data model and to re-attach to the process, still stopped on return from the exec. Not all debugger state is preserved across this re-exec.
If a 32-bit victim process execs a 64-bit program, then “stop” returns to the command prompt, but the debugger is no longer able to examine the process because it is now using the 64-bit data model. To resume debugging, execute the ::release -a dcmd, quit mdb, and then execute mdb -p pid to re-attach the 64-bit debugger to the process.
If a 64-bit victim process execs a 32-bit program, then “stop” returns to the command prompt, but the debugger only provides limited capabilities for examining the new process. All built-in dcmds work as advertised, but loadable dcmds do not since they do not perform data model conversion of structures. The user should release and re-attach the debugger to the process as described above in order to restore full debugging capabilities.
If the debugger is attached to a process that is stopped by job control (that is, it stopped in response to SIGTSTP, SIGTTIN, or SIGTTOU), the process can not be able to be set running again when it is continued by a continue dcmd. If the victim process is a member of the same session (that is, it shares the same controlling terminal as mdb), mdb attempts to bring the associated process group to the foreground and to continue the process with SIGCONT to resume it from job control stop. When mdb is detached from such a process, it restores the process group to the background before exiting. If the victim process is not a member of the same session, mdb cannot safely bring the process group to the foreground, so it continues the process with respect to the debugger, but the process remains stopped by job control. mdb prints a warning in this case, and the user must issue an “fg” command from the appropriate shell in order to resume the process.
When mdb attaches to a running process, the process is stopped and remains stopped until one of the continue dcmds is applied, or the debugger quits. If the -o nostop option is enabled prior to attaching the debugger to a process with -p, or prior to issuing an ::attach or :A command, mdb attaches to the process but does not stop it. While the process is still running, it can be inspected as usual (albeit with inconsistent results) and breakpoints or other tracing flags might be enabled. If the :c or ::cont dcmds are executed while the process is running, the debugger waits for the process to stop. If no traced software events occur, the user can send an interrupt (^C) after :c or ::cont to force the process to stop and return control to the debugger.
mdb releases the current running process (if any) when the :R, ::release, :r, ::run, $q, or ::quit dcmds are executed, or when the debugger terminates as the result of an EOF or signal. If the process was originally created by the debugger using :r or ::run, it is forcibly terminated as if by SIGKILL when it is released. If the process was already running prior to attaching mdb to it, it is set running again when it is released. A process can be released and left stopped and abandoned using the ::release -a option.
The ::list, ::offsetof, ::print, and ::sizeof dcmds require that one or more load objects contain compressed symbolic debugging information suitable for use with mdb. This information is currently only available for certain Solaris kernel modules.
The Oracle Solaris Modular Debugger Guide provides a more detailed description of mdb features, as well as information for debugger module developers.
The header file <sys/mdb_modapi.h> contains prototypes for the functions in the MDB Module API, and the /source/demo/mdb-examples package provides source code for an example module in the directory /usr/demo/mdb.
|