GNU crash utility commands help text - part 1

There is an excellent help documentation for 'crash' utility available at this link :

http://people.redhat.com/anderson/help.html

However, I put all these documentation in a two pages here.

# cat /tmp/crash-help-strings.txt

*              files          mod            runq           union

alias          foreach        mount          search         vm

ascii          fuser          net            set            vtop

bt             gdb            p              sig            waitq

btop           help           ps             struct         whatis

dev            irq            pte            swap           wr

dis            kmem           ptob           sym            q

eval           list           ptov           sys

exit           log            rd             task

extend         mach           repeat         timer

#

 1. ) Documentation for crash command *:


NAME
  * - pointer-to short-cut

SYNOPSIS
  * (struct or union command arguments)

DESCRIPTION
  This command is a short-cut command that replaces the requirement to enter
  "struct" or "union" command names.  For details on the arguments to
  those commands, enter "help struct" or "help union".

EXAMPLES
  Dump the page structure at address c02943c0:

    crash> *page c02943c0
    struct page {
      next = 0xc0fae740,
      prev = 0xc0018fb0,
      inode = 0x0,
      offset = 0x3f000,
      next_hash = 0xc02d6310,
      count = {
        counter = 0x1
      },
      flags = 0x310,
      wait = 0xc02943d8,
      pprev_hash = 0x0,
      buffers = 0x0
    }



 2. ) Documentation for crash command alias:


NAME
  alias - command aliases

SYNOPSIS
  alias [alias] [command string]

DESCRIPTION
  This command creates an alias for a given command string.  If no arguments
  are entered, the current list of aliases are displayed.  If one argument is
  entered, the command string for that alias, if any, is displayed.

           alias  the single word to be used as an alias
  command string  the word(s) that will be substituted for the alias

  Aliases may be created in four manners:

    1. entering the alias in $HOME/.crashrc.
    2. entering the alias in .crashrc in the current directory.
    3. executing an input file containing the alias command.
    4. during runtime with this command.

  During initialization, $HOME/.crashrc is read first, followed by the
  .crashrc file in the current directory.  Aliases in the .crashrc file
  in the current directory override those in $HOME/.crashrc.  Aliases
  entered with this command or by runtime input file override those
  defined in either .crashrc file.  Aliases may be deleted by entering an
  empty string for the second argument.  If redirection characters are to
  be part of the command string, the command string must be enclosed by
  quotation marks.

  Note that there are a number of helpful built-in aliases -- see the
  first example below.

EXAMPLES
  Display the currently-defined aliases, which in this example, only
  consist of the built-in aliases:

    crash> alias
    ORIGIN   ALIAS    COMMAND
    builtin  man      help
    builtin  ?        help
    builtin  quit     q
    builtin  sf       set scroll off
    builtin  sn       set scroll on
    builtin  hex      set radix 16
    builtin  dec      set radix 10
    builtin  g        gdb
    builtin  px       p -x
    builtin  pd       p -d
    builtin  for      foreach
    builtin  size     *
    builtin  dmesg    log
    builtin  lsmod    mod
    builtin  last     ps -l

  Create a new alias to be added to the list:

    crash> alias kp kmem -p
    ORIGIN   ALIAS    COMMAND
    runtime  kp       kmem -p

  Create an alias with redirection characters:

    crash> alias ksd "kmem -p | grep slab | grep DMA"
    ORIGIN   ALIAS    COMMAND
    runtime  ksd      kmem -p | grep slab | grep DMA

  Remove an alias:

    crash> alias kp ""
    alias deleted: kp



 3. ) Documentation for crash command ascii:


NAME
  ascii - translate a hexadecimal string to ASCII

SYNOPSIS
  ascii value ...

DESCRIPTION
  Translates 32-bit or 64-bit hexadecimal values to ASCII.  If no argument
  is entered, an ASCII chart is displayed.

EXAMPLES
  Translate the hexadecimal value of 0x62696c2f7273752f to ASCII:

    crash> ascii 62696c2f7273752f
    62696c2f7273752f: /usr/lib

  Display an ASCII chart:

    crash> ascii

          0    1   2   3   4   5   6   7
        +-------------------------------
      0 | NUL DLE  SP  0   @   P   '   p
      1 | SOH DC1  !   1   A   Q   a   q
      2 | STX DC2  "   2   B   R   b   r
      3 | ETX DC3  #   3   C   S   c   s
      4 | EOT DC4  $   4   D   T   d   t
      5 | ENQ NAK  %   5   E   U   e   u
      6 | ACK SYN  &   6   F   V   f   v
      7 | BEL ETB  `   7   G   W   g   w
      8 |  BS CAN  (   8   H   X   h   x
      9 |  HT  EM  )   9   I   Y   i   y
      A |  LF SUB  *   :   J   Z   j   z
      B |  VT ESC  +   ;   K   [   k   {
      C |  FF  FS  ,   <   L   \   l   |
      D |  CR  GS  _   =   M   ]   m   }
      E |  SO  RS  .   >   N   ^   n   ~
      F |  SI  US  /   ?   O   -   o  DEL



 4. ) Documentation for crash command bt:


NAME
  bt - backtrace

SYNOPSIS
  bt [-a|-g|-r|-t|-T|-l|-e|-E|-f|-F|-o|-O] [-R ref] [-I ip] [-S sp] [pid | task]

DESCRIPTION
  Display a kernel stack backtrace.  If no arguments are given, the stack
  trace of the current context will be displayed.

       -a  displays the stack traces of the active task on each CPU.
           (only applicable to crash dumps)
       -g  displays the stack traces of all threads in the thread group of
           the target task; the thread group leader will be displayed first.
       -r  display raw stack data, consisting of a memory dump of the two
           pages of memory containing the task_union structure.
       -t  display all text symbols found from the last known stack location
           to the top of the stack. (helpful if the back trace fails)
       -T  display all text symbols found from just above the task_struct or
           thread_info to the top of the stack. (helpful if the back trace
           fails or the -t option starts too high in the process stack).
       -l  show file and line number of each stack trace text location.
       -e  search the stack for possible kernel and user mode exception frames.
       -E  search the IRQ stacks (x86, x86_64 and ppc64), and the exception
           stacks (x86_64) for possible exception frames; all other arguments
           will be ignored since this is not a context-sensitive operation.
       -f  display all stack data contained in a frame; this option can be
           used to determine the arguments passed to each function; on ia64,
           the argument register contents are dumped.
       -F  similar to -f, except that the stack data is displayed symbolically
           when appropriate; if the stack data references a slab cache object,
           the name of the slab cache will be displayed in brackets; on ia64,
           the substitution is done to the argument register contents.
       -o  x86: use old backtrace method, permissable only on kernels that were
           compiled without the -fomit-frame_pointer.
           x86_64: use old backtrace method, which dumps potentially stale
           kernel text return addresses found on the stack.
       -O  x86: use old backtrace method by default, permissable only on kernels
           that were compiled without the -fomit-frame_pointer; subsequent usage
           of this option toggles the backtrace method.
           x86_64: use old backtrace method by default; subsequent usage of this
           option toggles the backtrace method.
   -R ref  display stack trace only if there is a reference to this symbol
           or text address.
    -I ip  use ip as the starting text location.
    -S sp  use sp as the starting stack frame address.
      pid  displays the stack trace(s) of this pid.
    taskp  displays the stack trace the the task referenced by this hexadecimal
           task_struct pointer.

  Multiple pid and taskp arguments may be specified.

  Note that all examples below are for x86 only.  The output format will differ
  for other architectures.  x86 backtraces from kernels that were compiled
  with the --fomit-frame-pointer CFLAG occasionally will drop stack frames,
  or display a stale frame reference.  When in doubt as to the accuracy of a
  backtrace, the -t or -T options may help fill in the blanks.

EXAMPLES
  Display the stack trace of the active task(s) when the kernel panicked:

    crash> bt -a
    PID: 286    TASK: c0b3a000  CPU: 0   COMMAND: "in.rlogind"
    #0 [c0b3be90] crash_save_current_state at c011aed0
    #1 [c0b3bea4] panic at c011367c
    #2 [c0b3bee8] tulip_interrupt at c01bc820
    #3 [c0b3bf08] handle_IRQ_event at c010a551
    #4 [c0b3bf2c] do_8259A_IRQ at c010a319
    #5 [c0b3bf3c] do_IRQ at c010a653
    #6 [c0b3bfbc] ret_from_intr at c0109634
       EAX: 00000000  EBX: c0e68280  ECX: 00000000  EDX: 00000004  EBP: c0b3bfbc
       DS:  0018      ESI: 00000004  ES:  0018      EDI: c0e68284
       CS:  0010      EIP: c012f803  ERR: ffffff09  EFLAGS: 00000246
    #7 [c0b3bfbc] sys_select at c012f803
    #8 [c0b3bfc0] system_call at c0109598
       EAX: 0000008e  EBX: 00000004  ECX: bfffc9a0  EDX: 00000000
       DS:  002b      ESI: bfffc8a0  ES:  002b      EDI: 00000000
       SS:  002b      ESP: bfffc82c  EBP: bfffd224
       CS:  0023      EIP: 400d032e  ERR: 0000008e  EFLAGS: 00000246 

  Display the stack traces of task f2814000 and PID 1592:

    crash> bt f2814000 1592
    PID: 1018   TASK: f2814000  CPU: 1   COMMAND: "java"
     #0 [f2815db4] schedule at c011af85
     #1 [f2815de4] __down at c010600f
     #2 [f2815e14] __down_failed at c01061b3
     #3 [f2815e24] stext_lock (via drain_cpu_caches) at c025fa55
     #4 [f2815ec8] kmem_cache_shrink_nr at c013a53e
     #5 [f2815ed8] do_try_to_free_pages at c013f402
     #6 [f2815f04] try_to_free_pages at c013f8d2
     #7 [f2815f1c] _wrapped_alloc_pages at c01406bd
     #8 [f2815f40] __alloc_pages at c014079d
     #9 [f2815f60] __get_free_pages at c014083e
    #10 [f2815f68] do_fork at c011cebb
    #11 [f2815fa4] sys_clone at c0105ceb
    #12 [f2815fc0] system_call at c010740c
        EAX: 00000078  EBX: 00000f21  ECX: bc1ffbd8  EDX: bc1ffbe0
        DS:  002b      ESI: 00000000  ES:  002b      EDI: bc1ffd04
        SS:  002b      ESP: 0807316c  EBP: 080731bc
        CS:  0023      EIP: 4012881e  ERR: 00000078  EFLAGS: 00000296

    PID: 1592   TASK: c0cec000  CPU: 3   COMMAND: "httpd"
     #0 [c0ceded4] schedule at c011af85
     #1 [c0cedf04] pipe_wait at c0153083
     #2 [c0cedf58] pipe_read at c015317f
     #3 [c0cedf7c] sys_read at c0148be6
     #4 [c0cedfc0] system_call at c010740c
        EAX: 00000003  EBX: 00000004  ECX: bffed4a3  EDX: 00000001
        DS:  002b      ESI: 00000001  ES:  002b      EDI: bffed4a3
        SS:  002b      ESP: bffed458  EBP: bffed488
        CS:  0023      EIP: 4024f1d4  ERR: 00000003  EFLAGS: 00000286

  In order to examine each stack frame's contents use the bt -f option.
  From the extra frame data that is displayed, the arguments passed to each
  function can be determined.  Re-examining the PID 1592 trace above:

    crash> bt -f 1592
    PID: 1592   TASK: c0cec000  CPU: 3   COMMAND: "httpd"
     #0 [c0ceded4] schedule at c011af85
        [RA: c0153088  SP: c0ceded4  FP: c0cedf04  SIZE: 52]
        c0ceded4: c0cedf00  c0cec000  ce1a6000  00000003 
        c0cedee4: c0cec000  f26152c0  cfafc8c0  c0cec000 
        c0cedef4: ef70a0a0  c0cec000  c0cedf28  c0cedf54 
        c0cedf04: c0153088 
     #1 [c0cedf04] pipe_wait at c0153083
        [RA: c0153184  SP: c0cedf08  FP: c0cedf58  SIZE: 84]
        c0cedf08: 00000000  c0cec000  00000000  00000000 
        c0cedf18: 00000000  c0a41fa0  c011d38b  c0394120 
        c0cedf28: 00000000  c0cec000  ceeebf30  ce4adf30 
        c0cedf38: 00000000  d4b60ce0  00000000  c0cedf58 
        c0cedf48: e204f820  ef70a040  00000001  c0cedf78 
        c0cedf58: c0153184 
     #2 [c0cedf58] pipe_read at c015317f
        [RA: c0148be8  SP: c0cedf5c  FP: c0cedf7c  SIZE: 36]
        c0cedf5c: ef70a040  c0cec000  00000000  00000000 
        c0cedf6c: 00000001  f27ae680  ffffffea  c0cedfbc 
        c0cedf7c: c0148be8 
     #3 [c0cedf7c] sys_read at c0148be6
        [RA: c0107413  SP: c0cedf80  FP: c0cedfc0  SIZE: 68]
        c0cedf80: f27ae680  bffed4a3  00000001  f27ae6a0 
        c0cedf90: 40160370  24000000  4019ba28  00000000 
        c0cedfa0: 00000000  fffffffe  bffba207  fffffffe 
        c0cedfb0: c0cec000  00000001  bffed4a3  bffed488 
        c0cedfc0: c0107413 
     #4 [c0cedfc0] system_call at c010740c
        EAX: 00000003  EBX: 00000004  ECX: bffed4a3  EDX: 00000001
        DS:  002b      ESI: 00000001  ES:  002b      EDI: bffed4a3
        SS:  002b      ESP: bffed458  EBP: bffed488
        CS:  0023      EIP: 4024f1d4  ERR: 00000003  EFLAGS: 00000286
        [RA: 4024f1d4  SP: c0cedfc4  FP: c0cedffc  SIZE: 60]
        c0cedfc4: 00000004  bffed4a3  00000001  00000001 
        c0cedfd4: bffed4a3  bffed488  00000003  0000002b 
        c0cedfe4: 0000002b  00000003  4024f1d4  00000023 
        c0cedff4: 00000286  bffed458  0000002b 

    Typically the arguments passed to a function will be the last values
    that were pushed onto the stack by the next higher-numbered function, i.e.,
    the lowest stack addresses in the frame above the called function's
    stack frame.  That can be verified by disassembling the calling function.
    For example, the arguments passed from sys_read() to pipe_read() above
    are the file pointer, the user buffer address, the count, and a pointer
    to the file structure's f_pos field.  Looking at the frame #3 data for
    sys_read(), the last four items pushed onto the stack (lowest addresses)
    are f27ae680, bffed4a3, 00000001, and f27ae6a0 -- which are the 4 arguments
    above, in that order.  Note that the first (highest address) stack content
    in frame #2 data for pipe_read() is c0148be8, which is the return address
    back to sys_read().

  Dump the text symbols found in the current context's stack:

    crash> bt -t
    PID: 1357   TASK: c1aa0000  CPU: 0   COMMAND: "lockd"
          START: schedule at c01190e0
      [c1aa1f28] dput at c0157dbc
      [c1aa1f4c] schedule_timeout at c0124cd4
      [c1aa1f78] svc_recv at cb22c4d8 [sunrpc]
      [c1aa1f98] put_files_struct at c011eb21
      [c1aa1fcc] nlmclnt_proc at cb237bef [lockd]
      [c1aa1ff0] kernel_thread at c0105826
      [c1aa1ff8] nlmclnt_proc at cb237a60 [lockd]

  Search the current stack for possible exception frames:

    crash> bt -e
    PID: 286    TASK: c0b3a000  CPU: 0   COMMAND: "in.rlogind"
   
     KERNEL-MODE EXCEPTION FRAME AT c0b3bf44:
       EAX: 00000000  EBX: c0e68280  ECX: 00000000  EDX: 00000004  EBP: c0b3bfbc
       DS:  0018      ESI: 00000004  ES:  0018      EDI: c0e68284
       CS:  0010      EIP: c012f803  ERR: ffffff09  EFLAGS: 00000246
   
     USER-MODE EXCEPTION FRAME AT c0b3bfc4:
       EAX: 0000008e  EBX: 00000004  ECX: bfffc9a0  EDX: 00000000
       DS:  002b      ESI: bfffc8a0  ES:  002b      EDI: 00000000
       SS:  002b      ESP: bfffc82c  EBP: bfffd224
       CS:  0023      EIP: 400d032e  ERR: 0000008e  EFLAGS: 00000246

  Display the back trace from a dumpfile that resulted from the execution
  of the crash utility's "sys -panic" command:

   crash> bt
   PID: 12523  TASK: c610c000  CPU: 0   COMMAND: "crash"
    #0 [c610de64] die at c01076ec
    #1 [c610de74] do_invalid_op at c01079bc
    #2 [c610df2c] error_code (via invalid_op) at c0107256
       EAX: 0000001d  EBX: c024a4c0  ECX: c02f13c4  EDX: 000026f6  EBP: c610c000
       DS:  0018      ESI: 401de2e0  ES:  0018      EDI: c610c000
       CS:  0010      EIP: c011bbb4  ERR: ffffffff  EFLAGS: 00010296
    #3 [c610df68] panic at c011bbb4
    #4 [c610df78] do_exit at c011f1fe
    #5 [c610dfc0] system_call at c0107154
       EAX: 00000001  EBX: 00000000  ECX: 00001000  EDX: 401df154
       DS:  002b      ESI: 401de2e0  ES:  002b      EDI: 00000000
       SS:  002b      ESP: bffebf0c  EBP: bffebf38
       CS:  0023      EIP: 40163afd  ERR: 00000001  EFLAGS: 00000246

  Display the back trace from a dumpfile that resulted from an attempt to
  insmod the sample "crash.c" kernel module that comes as part of the
  Red Hat netdump package:

   crash> bt
   PID: 1696   TASK: c74de000  CPU: 0   COMMAND: "insmod"
    #0 [c74dfdcc] die at c01076ec
    #1 [c74dfddc] do_page_fault at c0117bbc
    #2 [c74dfee0] error_code (via page_fault) at c0107256
       EAX: 00000013  EBX: cb297000  ECX: 00000000  EDX: c5962000  EBP: c74dff28
       DS:  0018      ESI: 00000000  ES:  0018      EDI: 00000000
       CS:  0010      EIP: cb297076  ERR: ffffffff  EFLAGS: 00010282
    #3 [c74dff1c] crash_init at cb297076 [crash]
    #4 [c74dff2c] sys_init_module at c011d233
    #5 [c74dffc0] system_call at c0107154
       EAX: 00000080  EBX: 08060528  ECX: 08076450  EDX: 0000000a
       DS:  002b      ESI: 0804b305  ES:  002b      EDI: 08074ed0
       SS:  002b      ESP: bffe9a90  EBP: bffe9ac8
       CS:  0023      EIP: 4012066e  ERR: 00000080  EFLAGS: 00000246



 5. ) Documentation for crash command btop:


NAME
  btop - bytes to page

SYNOPSIS
  btop address ...

DESCRIPTION
  This command translates a hexadecimal address to its page number.

EXAMPLES
    crash> btop 512a000
    512a000: 512a



 6. ) Documentation for crash command dev:


NAME
  dev - device data

SYNOPSIS
  dev [-i | -p | -d]

DESCRIPTION
  If no argument is entered, this command dumps character and block
  device data.

    -i  display I/O port usage; on 2.4 kernels, also display I/O memory usage.
    -p  display PCI device data.
    -d  display disk I/O statistics:
         TOTAL: total number of allocated in-progress I/O requests
          SYNC: I/O requests that are synchronous
         ASYNC: I/O requests that are asynchronous
          READ: I/O requests that are reads (older kernels)
         WRITE: I/O requests that are writes (older kernels)
           DRV: I/O requests that are in-flight in the device driver

EXAMPLES
  Display character and block device data:

    crash> dev
    CHRDEV    NAME              CDEV    OPERATIONS
       1      mem             f79b83c0  memory_fops
       4      /dev/vc/0       c07bc560  console_fops
       4      tty             f7af5004  tty_fops
       4      ttyS            f7b02204  tty_fops
       5      /dev/tty        c07bc440  tty_fops
       5      /dev/console    c07bc4a0  console_fops
       5      /dev/ptmx       c07bc500  ptmx_fops
       6      lp              c5797e40  lp_fops
       7      vcs             f7b03d40  vcs_fops
      10      misc            f7f68640  misc_fops
      13      input           f79b8840  input_fops
      21      sg              f7f12840  sg_fops
      29      fb              f7f8c640  fb_fops
     128      ptm             f7b02604  tty_fops
     136      pts             f7b02404  tty_fops
     162      raw             c0693e40  raw_fops
     180      usb             f79b8bc0  usb_fops
     189      usb_device      c06a0300  usbfs_device_file_operations
     216      rfcomm          f5961a04  tty_fops
     254      pcmcia          f79b82c0  ds_fops
   
    BLKDEV    NAME             GENDISK  OPERATIONS
       1      ramdisk         f7b23480  rd_bd_op
       8      sd              f7cab280  sd_fops
       9      md              f7829b80  md_fops
      11      sr              f75c24c0  sr_bdops
      65      sd               (none) 
      66      sd               (none) 
      67      sd               (none) 
      68      sd               (none) 
      69      sd               (none) 
      70      sd               (none) 
      71      sd               (none) 
     128      sd               (none) 
     129      sd               (none) 
     130      sd               (none) 
     131      sd               (none) 
     132      sd               (none) 
     133      sd               (none) 
     134      sd               (none) 
     135      sd               (none) 
     253      device-mapper   c57a0ac0  dm_blk_dops
     254      mdp              (none) 

  Display PCI data:

    crash> dev -p
    PCI_DEV  BU:SL.FN CLASS: VENDOR-DEVICE
    c00051c0 00:00.0  Host bridge: Intel 440BX - 82443BX Host
    c0005250 00:01.0  PCI bridge: Intel 440BX - 82443BX AGP
    c00052e0 00:07.0  ISA bridge: Intel 82371AB PIIX4 ISA
    c0005370 00:07.1  IDE interface: Intel 82371AB PIIX4 IDE
    c0005400 00:07.2  USB Controller: Intel 82371AB PIIX4 USB
    c0005490 00:07.3  Bridge: Intel 82371AB PIIX4 ACPI
    c0005520 00:11.0  Ethernet controller: 3Com 3C905B 100bTX
    c00055b0 00:13.0  PCI bridge: DEC DC21152
    c0005640 01:00.0  VGA compatible controller: NVidia [PCI_DEVICE 28]
    c00056d0 02:0a.0  SCSI storage controller: Adaptec AIC-7890/1
    c0005760 02:0e.0  SCSI storage controller: Adaptec AIC-7880U

  Display I/O port and I/O memory usage:

    crash> dev -i
    RESOURCE    RANGE    NAME
    c03036d4  0000-ffff  PCI IO
    c0302594  0000-001f  dma1
    c03025b0  0020-003f  pic1
    c03025cc  0040-005f  timer
    c03025e8  0060-006f  keyboard
    c0302604  0080-008f  dma page reg
    c0302620  00a0-00bf  pic2
    c030263c  00c0-00df  dma2
    c0302658  00f0-00ff  fpu
    c122ff20  0170-0177  ide1
    c122f240  0213-0213  isapnp read
    c122ff40  02f8-02ff  serial(auto)
    c122ff00  0376-0376  ide1
    c03186e8  03c0-03df  vga+
    c122ff60  03f8-03ff  serial(auto)
    c123851c  0800-083f  Intel Corporation 82371AB PIIX4 ACPI
    c1238538  0840-085f  Intel Corporation 82371AB PIIX4 ACPI
    c122f220  0a79-0a79  isapnp write
    c122f200  0cf8-0cff  PCI conf1
    c1238858  dc00-dc7f  3Com Corporation 3c905B 100BaseTX [Cyclone]
    c122fc00  dc00-dc7f  00:11.0
    c12380c8  dce0-dcff  Intel Corporation 82371AB PIIX4 USB
    c1238d1c  e000-efff  PCI Bus #02
    c1237858  e800-e8ff  Adaptec AIC-7880U
    c1237458  ec00-ecff  Adaptec AHA-2940U2/W / 7890
    c1239cc8  ffa0-ffaf  Intel Corporation 82371AB PIIX4 IDE
   
    RESOURCE        RANGE        NAME
    c03036f0  00000000-ffffffff  PCI mem
    c0004000  00000000-0009ffff  System RAM
    c03026ac  000a0000-000bffff  Video RAM area
    c03026fc  000c0000-000c7fff  Video ROM
    c0302718  000c9800-000cdfff  Extension ROM
    c0302734  000ce000-000ce7ff  Extension ROM
    c0302750  000ce800-000cffff  Extension ROM
    c03026e0  000f0000-000fffff  System ROM
    c0004040  00100000-07ffdfff  System RAM
    c0302674  00100000-0028682b  Kernel code
    c0302690  0028682c-0031c63f  Kernel data
    c0004060  07ffe000-07ffffff  reserved
    c1239058  ec000000-efffffff  Intel Corporation 440BX/ZX - 82443BX/ZX Host
                                 bridge
    c1238d54  f1000000-f1ffffff  PCI Bus #02
    c1239554  f2000000-f5ffffff  PCI Bus #01
    c1237074  f4000000-f5ffffff  nVidia Corporation Riva TnT2 [NV5]
    c1238d38  fa000000-fbffffff  PCI Bus #02
    c1237874  faffe000-faffefff  Adaptec AIC-7880U
    c127ec40  faffe000-faffefff  aic7xxx
    c1237474  fafff000-faffffff  Adaptec AHA-2940U2/W / 7890
    c127eec0  fafff000-faffffff  aic7xxx
    c1239538  fc000000-fdffffff  PCI Bus #01
    c1237058  fc000000-fcffffff  nVidia Corporation Riva TnT2 [NV5]
    c1238874  fe000000-fe00007f  3Com Corporation 3c905B 100BaseTX [Cyclone]
    c0004080  fec00000-fec0ffff  reserved
    c00040a0  fee00000-fee0ffff  reserved
    c00040c0  ffe00000-ffffffff  reserved

  Display disk I/O statistics:

    crash> dev -d
    MAJOR GENDISK            NAME     REQUEST QUEUE      TOTAL  READ WRITE   DRV
        2 0xffff81012d8a5000 fd0      0xffff81012dc053c0    12     0    12     0
       22 0xffff81012dc6b000 hdc      0xffff81012d8ae340     2     2     0     0
        8 0xffff81012dd71000 sda      0xffff81012d8af040     6     0     6     6
        8 0xffff81012dc77000 sdb      0xffff81012d8b5740     0     0     0     0
        8 0xffff81012d8d0c00 sdc      0xffff81012d8ae9c0     0     0     0     0



 7. ) Documentation for crash command dis:


NAME
  dis - disassemble

SYNOPSIS
  dis [-rludx][-b [num]] [address | symbol | (expression)] [count]

DESCRIPTION
  This command disassembles source code instructions starting (or ending) at
  a text address that may be expressed by value, symbol or expression:

            -r  (reverse) displays all instructions from the start of the
                routine up to and including the designated address.
            -l  displays source code line number data in addition to the
                disassembly output.
            -u  address is a user virtual address in the current context;
                otherwise the address is assumed to be a kernel virtual address.
                If this option is used, then -r and -l are ignored.
            -x  override default output format with hexadecimal format.
            -d  override default output format with decimal format.
      -b [num]  modify the pre-calculated number of encoded bytes to skip after
                a kernel BUG ("ud2a") instruction; with no argument, displays
                the current number of bytes being skipped. (x86 and x86_64 only)
       address  starting hexadecimal text address.
        symbol  symbol of starting text address.  On ppc64, the symbol
                preceded by '.' is used.
  (expression)  expression evaluating to a starting text address.
         count  the number of instructions to be disassembled (default is 1).
                If no count argument is entered, and the starting address
                is entered as a text symbol, then the whole routine will be
                disassembled.  The count argument is ignored when used with
                the -r option.

EXAMPLES
  Disassemble the sys_signal() routine without, and then with, line numbers:

    crash> dis sys_signal
    0xc0112c88 :        push   %ebp
    0xc0112c89 :      mov    %esp,%ebp
    0xc0112c8b :      sub    $0x28,%esp
    0xc0112c8e :      mov    0xc(%ebp),%eax
    0xc0112c91 :      mov    %eax,0xffffffec(%ebp)
    0xc0112c94 :     movl   $0xc0000000,0xfffffff0(%ebp)
    0xc0112c9b :     lea    0xffffffd8(%ebp),%eax
    0xc0112c9e :     push   %eax
    0xc0112c9f :     lea    0xffffffec(%ebp),%eax
    0xc0112ca2 :     push   %eax
    0xc0112ca3 :     pushl  0x8(%ebp)
    0xc0112ca6 :     call   0xc01124b8
    0xc0112cab :     test   %eax,%eax
    0xc0112cad :     jne    0xc0112cb2
    0xc0112caf :     mov    0xffffffd8(%ebp),%eax
    0xc0112cb2 :     leave
    0xc0112cb3 :     ret

    crash> dis -l sys_signal
    /usr/src/linux-2.2.5/kernel/signal.c: 1074
    0xc0112c88 :        push   %ebp
    0xc0112c89 :      mov    %esp,%ebp
    0xc0112c8b :      sub    $0x28,%esp
    0xc0112c8e :      mov    0xc(%ebp),%eax
    /usr/src/linux-2.2.5/kernel/signal.c: 1078
    0xc0112c91 :      mov    %eax,0xffffffec(%ebp)
    /usr/src/linux-2.2.5/kernel/signal.c: 1079
    0xc0112c94 :     movl   $0xc0000000,0xfffffff0(%ebp)
    /usr/src/linux-2.2.5/kernel/signal.c: 1081
    0xc0112c9b :     lea    0xffffffd8(%ebp),%eax
    0xc0112c9e :     push   %eax
    0xc0112c9f :     lea    0xffffffec(%ebp),%eax
    0xc0112ca2 :     push   %eax
    0xc0112ca3 :     pushl  0x8(%ebp)
    0xc0112ca6 :     call   0xc01124b8
    /usr/src/linux-2.2.5/kernel/signal.c: 1083
    0xc0112cab :     test   %eax,%eax
    0xc0112cad :     jne    0xc0112cb2
    0xc0112caf :     mov    0xffffffd8(%ebp),%eax
    /usr/src/linux-2.2.5/kernel/signal.c: 1084
    0xc0112cb2 :     leave
    0xc0112cb3 :     ret

  Given a return address expression of "do_no_page+65", find out the
  function that do_no_page() calls by using the reverse flag:

    crash> dis -r (do_no_page+65)
    0xc011ea68 :        push   %ebp
    0xc011ea69 :      mov    %esp,%ebp
    0xc011ea6b :      push   %edi
    0xc011ea6c :      push   %esi
    0xc011ea6d :      push   %ebx
    0xc011ea6e :      mov    0xc(%ebp),%ebx
    0xc011ea71 :      mov    0x10(%ebp),%edx
    0xc011ea74 :     mov    0x14(%ebp),%edi
    0xc011ea77 :     mov    0x28(%ebx),%eax
    0xc011ea7a :     test   %eax,%eax
    0xc011ea7c :     je     0xc011ea85
    0xc011ea7e :     mov    0x18(%eax),%ecx
    0xc011ea81 :     test   %ecx,%ecx
    0xc011ea83 :     jne    0xc011eab0
    0xc011ea85 :     mov    $0xffffe000,%eax
    0xc011ea8a :     and    %esp,%eax
    0xc011ea8c :     decl   0x30(%eax)
    0xc011ea8f :     jns    0xc011ea9a
    0xc011ea91 :     lock btrl $0x0,0xc022fb60
    0xc011ea9a :     push   %edi
    0xc011ea9b :     mov    0x18(%ebp),%esi
    0xc011ea9e :     push   %esi
    0xc011ea9f :     push   %ebx
    0xc011eaa0 :     mov    0x8(%ebp),%esi
    0xc011eaa3 :     push   %esi
    0xc011eaa4 :     call   0xc011e9e4
    0xc011eaa9 :     jmp    0xc011eb47
   
  Disassemble 10 instructions starting at user virtual address 0x81ec624:

    crash> dis -u 81ec624 10
    0x81ec624:      push   %ebp
    0x81ec625:      mov    %esp,%ebp
    0x81ec627:      sub    $0x18,%esp
    0x81ec62a:      movl   $0x1,0x8(%ebp)
    0x81ec631:      mov    0x82f9040,%eax
    0x81ec636:      mov    0x10(%eax),%edx
    0x81ec639:      and    $0x100,%edx
    0x81ec63f:      mov    0x14(%eax),%ecx
    0x81ec642:      and    $0x0,%ecx
    0x81ec645:      mov    %ecx,%eax

  Override the current decimal output radix format:

    crash> dis sys_read 10 -x
    0xffffffff8001178f :  push   %r13
    0xffffffff80011791 :      mov    %rsi,%r13
    0xffffffff80011794 :      push   %r12
    0xffffffff80011796 :      mov    $0xfffffffffffffff7,%r12
    0xffffffff8001179d :      push   %rbp
    0xffffffff8001179e :      mov    %rdx,%rbp
    0xffffffff800117a1 :     push   %rbx
    0xffffffff800117a2 :     sub    $0x18,%rsp
    0xffffffff800117a6 :     lea    0x14(%rsp),%rsi
    0xffffffff800117ab :     callq  0xffffffff8000b5b4



 8. ) Documentation for crash command eval:


NAME
  eval - evaluate

SYNOPSIS
  eval [-b][-l] (expression) | value

DESCRIPTION
  This command evaluates an expression or numeric value, and displays its
  result in hexadecimal, decimal, octal and binary. If the resultant value
  is an integral number of gigabytes, megabytes, or kilobytes, a short-hand
  translation of the number will also be shown next to the hexadecimal
  value.  If the most significant bit is set, the decimal display will show
  both unsigned and signed (negative) values.  Expressions must of the format
  (x operator y), where "x" and "y" may be either numeric values or
  symbols.  The list of operators are:

                     +  -  &  |  ^  *  %  /  <<  >>

  Enclosing the expression within parentheses is optional except when the
  "|", "<<" or ">>" operators are used.  The single "value" argument may
  be a number or symbol.  Number arguments must be hexadecimal or decimal.
  A leading "0x" identifies a number as hexadecimal, but is not required
  when obvious.  Numbers may be followed by the letters "k" or "K", "m"
  or "M", and "g" or "G", which multiplies the value by a factor of 1024,
  1 megabyte or 1 gigabyte, respectively.  Numeric arguments may be preceded
  by the one's complement operator ~.

    -b  Indicate which bit positions in the resultant value are set.
    -l  Numeric arguments are presumed to be 64-bit values, and the result
        will be expressed as a 64-bit value. (ignored on 64-bit processors)
        However, if either operand or the resultant value are 64-bit values,
        then the result will be also be expressed as a 64-bit value.

 The -b and -l options must precede the expression or value arguments.

EXAMPLES
   crash> eval 128m
   hexadecimal: 8000000  (128MB)
       decimal: 134217728 
         octal: 1000000000
        binary: 00001000000000000000000000000000
   
   crash> eval 128 * 1m
   hexadecimal: 8000000  (128MB)
       decimal: 134217728 
         octal: 1000000000
        binary: 00001000000000000000000000000000
   
   crash> eval (1 << 27)
   hexadecimal: 8000000  (128MB)
       decimal: 134217728 
         octal: 1000000000
        binary: 00001000000000000000000000000000
  
   crash> eval (1 << 32)
   hexadecimal: 100000000  (4GB)
       decimal: 4294967296
         octal: 40000000000
        binary: 0000000000000000000000000000000100000000000000000000000000000000

   crash> eval -b 41dc065
   hexadecimal: 41dc065
       decimal: 69058661 
         octal: 407340145
        binary: 00000100000111011100000001100101
      bits set: 26 20 19 18 16 15 14 6 5 2 0

   crash> eval -lb 64g
   hexadecimal: 1000000000  (64GB)
       decimal: 68719476736
         octal: 1000000000000
        binary: 0000000000000000000000000001000000000000000000000000000000000000
      bits set: 36



 9. ) Documentation for crash command exit:


NAME
  exit - exit this session

SYNOPSIS
  exit 

DESCRIPTION
  Bail out of the current crash session.

NOTE
  This command is equivalent to the "q" command.



 10. ) Documentation for crash command extend:


NAME
  extend - extend the crash command set

SYNOPSIS
  extend [shared-object ...] | [-u [shared-object ...]]

DESCRIPTION
  This command dynamically loads or unloads crash extension shared object
  libraries:

    shared-object     load the specified shared object file; more than one
                      one object file may be entered.
    -u shared-object  unload the specified shared object file; if no file
                      arguments are specified, unload all objects.

  If the shared-object filename is not expressed with a fully-qualified
  pathname, the following directories will be searched in the order shown,
  and the first instance of the file that is found will be selected:

     1. the current working directory
     2. the directory specified in the CRASH_EXTENSIONS environment variable
     3. /usr/lib64/crash/extensions (64-bit architectures)
     4. /usr/lib/crash/extensions

  If no arguments are entered, the current set of shared object files and
  a list of their commands will be displayed.  The registered commands
  contained in each shared object file will appear automatically in the
  "help" command screen.

  An example of a shared object prototype file, and how to compile it
  into a shared object, is appended below.

EXAMPLES
  Load two shared object files:

    crash> extend extlib1.so extlib2.so
    ./extlib1.so: shared object loaded
    ./extlib2.so: shared object loaded

  Display the current set of shared object files and their commands:

    crash> extend
    SHARED OBJECT  COMMANDS
    ./extlib1.so   echo util bin
    ./extlib2.so   smp show

  Unload one of the shared object files:

    crash> extend -u extlib1.so
    ./extlib1.so: shared object unloaded

  Unload all currently-loaded object files:

    crash> extend -u
    ./extlib2.so: shared object unloaded

CREATING A SHARED OBJECT
  The extend command loads shared object files using dlopen(3), which in
  turn calls the shared object's _init() function.  The shared object's _init()
  function should register its command set by calling register_extension(),
  passing it a pointer to an array of one or more structures of the
  following type:

    struct command_table_entry {
            char *name;
            cmd_func_t func;
            char **help_data,
            ulong flags;
    };

  Each command_table_entry structure contains the ASCII name of a command,
  the command's function address, a pointer to an array of help data strings,
  and a flags field.  The help_data field is optional; if it is non-NULL, it
  should point to an array of character strings used by the "help"
  command, and during command failures.  The flags field currently has one
  available bit setting, REFRESH_TASK_TABLE, which should be set if it is
  preferable to reload the current set of running processes just prior to
  executing the command (on a live system).  Terminate the array of
  command_table_entry structures with an entry with a NULL command name. 

  Below is an example shared object file consisting of just one command,
  called "echo", which simply echoes back all arguments passed to it.
  Note the comments contained within it for further details.  Cut and paste
  the following output into a file, and call it, for example, "echo.c".
  Then compiled in either of two manners.  Either manually like so:

  gcc -nostartfiles -shared -rdynamic -o echo.so echo.c -fPIC -D $(TARGET_CFLAGS)

  where must be one of the MACHINE_TYPE #define's in defs.h,
  and where $(TARGET_CFLAGS) is the same as it is declared in the top-level
  Makefile after a build is completed.  Or alternatively, the "echo.c" file
  can be copied into the "extensions" subdirectory, and compiled automatically
  like so:

  make extensions

  The echo.so file may be dynamically linked into crash during runtime, or
  during initialization by putting "extend echo.so" into a .crashrc file
  located in the current directory, or in the user's $HOME directory.
 
---------------------------------- cut here ----------------------------------

#include "defs.h"    /* From the crash source top-level directory */

void cmd_echo();     /* Declare the commands and their help data. */
char *help_echo[];

static struct command_table_entry command_table[] = {
        "echo", cmd_echo, help_echo, 0,           /* One or more commands, */
        NULL,                                     /* terminated by NULL, */
};


_init() /* Register the command set. */
{
        register_extension(command_table);
}

/*
 *  The _fini() function is called if the shared object is unloaded.
 *  If desired, perform any cleanups here.
 */
_fini() { }


/*
 *  Arguments are passed to the command functions in the global args[argcnt]
 *  array.  See getopt(3) for info on dash arguments.  Check out defs.h and
 *  other crash commands for usage of the myriad of utility routines available
 *  to accomplish what your task.
 */
void
cmd_echo()
{
        int c;

        while ((c = getopt(argcnt, args, "")) != EOF) {
                switch(c)
                {
                default:
                        argerrs++;
                        break;
                }
        }

        if (argerrs)
                cmd_usage(pc->curcmd, SYNOPSIS);

        while (args[optind])
                fprintf(fp, "%s ", args[optind++]);

        fprintf(fp, "\n");
}

/*
 *  The optional help data is simply an array of strings in a defined format.
 *  For example, the "help echo" command will use the help_echo[] string
 *  array below to create a help page that looks like this:
 *
 *    NAME
 *      echo - echoes back its arguments
 *
 *    SYNOPSIS
 *      echo arg ...
 *
 *    DESCRIPTION
 *      This command simply echoes back its arguments.
 *
 *    EXAMPLE
 *      Echo back all command arguments:
 *
 *        crash> echo hello, world
 *        hello, world
 *
 */

char *help_echo[] = {
        "echo",                        /* command name */
        "echo",                        /* command name */
        "echoes back its arguments",   /* short description */
        "arg ...",                     /* argument synopsis, or " " if none */

        "  This command simply echoes back its arguments.",
        "\nEXAMPLE",
        "  Echo back all command arguments:\n",
        "    crash> echo hello, world",
        "    hello, world",
        NULL
};


 11. ) Documentation for crash command files:


NAME
  files - open files

SYNOPSIS
  files [-d dentry] | [-R reference] [pid | taskp] ...

DESCRIPTION
  This command displays information about open files of a context.
  It prints the context's current root directory and current working
  directory, and then for each open file descriptor it prints a pointer
  to its file struct, a pointer to its dentry struct, a pointer to the
  inode, the file type, and the pathname.  If no arguments are entered,
  the current context is used.  The -R option, typically invoked from
  "foreach files", searches for references to a supplied number, address,
  or filename argument, and prints only the essential information leading
  up to and including the reference.  The -d option is not context
  specific, and only shows the data requested.

     -d dentry  given a hexadecimal dentry address, display its inode,
                super block, file type, and full pathname.
  -R reference  search for references to this file descriptor number,
                filename, or dentry, inode, or file structure address.
           pid  a process PID.
         taskp  a hexadecimal task_struct pointer.

EXAMPLES
  Display the open files of the current context:

    crash> files
    PID: 720    TASK: c67f2000  CPU: 1   COMMAND: "innd"
    ROOT: /    CWD: /var/spool/news/articles
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  c6b9c740  c7cc45a0  c7c939e0  CHR   /dev/null
      1  c6b9c800  c537bb20  c54d0000  REG   /var/log/news/news
      2  c6df9600  c537b420  c5c36360  REG   /var/log/news/errlog
      3  c74182c0  c6ede260  c6da3d40  PIPE
      4  c6df9720  c696c620  c69398c0  SOCK
      5  c6b9cc20  c68e7000  c6938d80  SOCK
      6  c6b9c920  c7cc45a0  c7c939e0  CHR   /dev/null
      7  c6b9c680  c58fa5c0  c58a1200  REG   /var/lib/news/history
      8  c6df9f00  c6ede760  c6da3200  PIPE
      9  c6b9c6e0  c58fa140  c5929560  REG   /var/lib/news/history.dir
     10  c7fa9320  c7fab160  c7fafd40  CHR   /dev/console
     11  c6b9c7a0  c58fa5c0  c58a1200  REG   /var/lib/news/history
     12  c377ec60  c58fa5c0  c58a1200  REG   /var/lib/news/history
     13  c4528aa0  c58fa6c0  c52fbb00  REG   /var/lib/news/history.pag
     14  c6df9420  c68e7700  c6938360  SOCK
     15  c6df9360  c68e7780  c6938120  SOCK
     16  c6b9c0e0  c68e7800  c6772000  SOCK
     17  c6b9c200  c6b5f9c0  c6b5cea0  REG   /var/lib/news/active
     21  c6b9c080  c6ede760  c6da3200  PIPE

  Display the files opened by the "crond" daemon, which is PID 462:

  crash> files 462
    PID: 462    TASK: f7220000  CPU: 2   COMMAND: "crond"
    ROOT: /    CWD: /var/spool
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  f7534ae0  f7538de0  f7518dc0  CHR   /dev/console
      1  f7368f80  f72c7a40  f72f27e0  FIFO  pipe:/[1456]
      2  f74f3c80  f72c79c0  f72f2600  FIFO  pipe:/[1457]
      3  f7368b60  f72a5be0  f74300c0  REG   /var/run/crond.pid
      4  f7534360  f73408c0  f72c2840  REG   /var/log/cron
      7  f7368ce0  f72c7940  f72f2420  FIFO  pipe:/[1458]
      8  f7295de0  f72c7940  f72f2420  FIFO  pipe:/[1458]
     21  f74f36e0  f747cdc0  f747e840  CHR   /dev/null

  The -R option is typically invoked from "foreach files".  This example
  shows all tasks that have "/dev/pts/4" open:

    crash> foreach files -R pts/4
    PID: 18633  TASK: c310a000  CPU: 0   COMMAND: "crash"
    ROOT: /    CWD: /home/CVS_pool/crash
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      1  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      2  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
   
    PID: 18664  TASK: c2392000  CPU: 1   COMMAND: "less"
    ROOT: /    CWD: /home/CVS_pool/crash
     FD    FILE     DENTRY    INODE    TYPE  PATH
      1  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      2  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
   
    PID: 23162  TASK: c5088000  CPU: 1   COMMAND: "bash"
    ROOT: /    CWD: /home/CVS_pool/crash
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      1  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      2  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
    255  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
   
    PID: 23159  TASK: c10fc000  CPU: 1   COMMAND: "xterm"
    ROOT: /    CWD: /homes/anderson/
     FD    FILE     DENTRY    INODE    TYPE  PATH
      5  c1560da0  c2cb96d0  c2cad430  CHR   /dev/pts/4

  Display information about the dentry at address f745fd60:

    crash> files -d f745fd60
     DENTRY    INODE    SUPERBLK  TYPE  PATH
     f745fd60  f7284640  f73a3e00  REG   /var/spool/lpd/lpd.lock



 12. ) Documentation for crash command foreach:


NAME
  foreach - display command data for multiple tasks in the system

SYNOPSIS
  foreach [[pid | taskp | name | [kernel | user]] ...] command [flag] [argument]

DESCRIPTION
  This command allows for a an examination of various kernel data associated
  with any, or all, tasks in the system, without having to set the context
  to each targeted task.

      pid  perform the command(s) on this PID.
    taskp  perform the command(s) on task referenced by this hexadecimal
           task_struct pointer.
     name  perform the command(s) on all commands with this name.  If the
           command name can be confused with a foreach command name, then
           precede the name string with a "\".
     user  perform the command(s) on all user (non-kernel) threads.
   kernel  perform the command(s) on all kernel threads.
   active  perform the command(s) on the active thread on each CPU.

  If none of the task-identifying arguments above are entered, the command
  will be performed on all tasks.

  command  select one or more of the following commands to be run on the tasks
           selected, or on all tasks:

             bt  run the "bt" command  (optional flags: -r -t -l -e -R -f -F -o)
             vm  run the "vm" command  (optional flags: -p -v -m -R)
           task  run the "task" command  (optional flags: -R -d -x)
          files  run the "files" command  (optional flag: -R)
            net  run the "net" command  (optional flags: -s -S -R)
            set  run the "set" command
            sig  run the "sig" command (optional flag: -g)
           vtop  run the "vtop" command  (optional flags: -c -u -k)

     flag  Pass this optional flag to the command selected.
 argument  Pass this argument to the command selected.

  A header containing the PID, task address, cpu and command name will be
  pre-pended before the command output for each selected task.  Consult the
  help page of each of the command types above for details.

EXAMPLES
  Display the stack traces for all tasks:

    crash> foreach bt
    PID: 4752   TASK: c7680000  CPU: 1   COMMAND: "xterm"
     #0 [c7681edc] schedule at c01135f6
        (void)
     #1 [c7681f34] schedule_timeout at c01131ff
        (24)
     #2 [c7681f64] do_select at c0132838
        (5, c7681fa4, c7681fa0)
     #3 [c7681fbc] sys_select at c0132dad
        (5, 8070300, 8070380, 0, 0)
     #4 [bffffb0c] system_call at c0109944
        EAX: 0000008e  EBX: 00000005  ECX: 08070300  EDX: 08070380
        DS:  002b      ESI: 00000000  ES:  002b      EDI: 00000000
        SS:  002b      ESP: bffffadc  EBP: bffffb0c
        CS:  0023      EIP: 402259ee  ERR: 0000008e  EFLAGS: 00000246
   
    PID: 557    TASK: c5600000  CPU: 0   COMMAND: "nfsd"
     #0 [c5601f38] schedule at c01135f6
        (void)
     #1 [c5601f90] schedule_timeout at c01131ff
        (c5600000)
     #2 [c5601fb8] svc_recv at c805363a
        (c0096f40, c5602800, 7fffffff, 100, c65c9f1c)
     #3 [c5601fec] (nfsd module) at c806e303
        (c5602800, c5602800, c0096f40, 6c6e0002, 50)
     #4 [c65c9f24] kernel_thread at c010834f
        (0, 0, ext2_file_inode_operations)
   
    PID: 824    TASK: c7c84000  CPU: 0   COMMAND: "mingetty"
    ...

  Display the task_struct structure for each "bash" command:

    crash> foreach bash task
    ...

  Display the open files for all tasks:

    crash> foreach files
    ...




 13. ) Documentation for crash command fuser:


NAME
  fuser - file users

SYNOPSIS
  fuser [pathname | inode]

DESCRIPTION
  This command displays the tasks using specified files or sockets.
  Tasks will be listed that reference the file as the current working
  directory, root directory, an open file descriptor, or that mmap the
  file.  If the file is held open in the kernel by the lockd server on
  behalf of a client discretionary file lock, the client hostname is
  listed.

    pathname  the full pathname of the file.
    inode     the hexadecimal inode address for the file.

EXAMPLES
  Display the tasks using file /usr/lib/libkfm.so.2.0.0

    crash> fuser /usr/lib/libkfm.so.2.0.0
     PID    TASK    COMM            USAGE
     779  c5e82000  "kwm"           mmap
     808  c5a8e000  "krootwm"       mmap
     806  c5b42000  "kfm"           mmap
     809  c5dde000  "kpanel"        mmap



 14. ) Documentation for crash command gdb:


NAME
  gdb - gdb command

SYNOPSIS
  gdb command ...

DESCRIPTION
  This command passes its arguments directly to gdb for processing.
  This is typically not necessary, but where ambiguities between crash and
  gdb command names exist, this will force the command to be executed by gdb.

EXAMPLES
    crash> gdb help
    List of classes of commands:
   
    aliases -- Aliases of other commands
    breakpoints -- Making program stop at certain points
    data -- Examining data
    files -- Specifying and examining files
    internals -- Maintenance commands
    obscure -- Obscure features
    running -- Running the program
    stack -- Examining the stack
    status -- Status inquiries
    support -- Support facilities
    tracepoints -- Tracing of program execution without stopping the program
    user-defined -- User-defined commands
   
    Type "help" followed by a class name for a list of commands in that class.
    Type "help" followed by command name for full documentation.
    Command name abbreviations are allowed if unambiguous.



 15. ) Documentation for crash command help:


NAME
  help - get help

SYNOPSIS
  help [command | all] [-]

DESCRIPTION
  When entered with no argument, a list of all currently available crash
  commands is listed.  If a name of a crash command is entered, a man-like
  page for the command is displayed.  If "all" is entered, help pages
  for all commands will be displayed.  If neither of the above is entered,
  the argument string will be passed on to the gdb help command.

  A number of internal debug, statistical, and other dumpfile related
  data is available with the following options:

    -e - extension table data
    -a - alias data
    -b - shared buffer data
    -B - build data
    -c - numargs cache
    -d - device table
    -D - dumpfile contents/statistics
    -f - filesys table
    -k - kernel_table
    -K - kernel_table (verbose)
    -M machine specific
    -m - machdep_table
    -n - dumpfile contents/statistics
    -s - symbol table data
    -v - vm_table
    -V - vm_table (verbose)
    -o - offset_table and size_table
    -t - task_table
    -x - text cache
    -T - task_table plus context_array
    -p - program_context
    -h - hash_table data
    -H - hash_table data (verbose)
    -L - LKCD page cache environment



 16. ) Documentation for crash command irq:


NAME
  irq - IRQ data

SYNOPSIS
  irq [[[index ...] | -u ] | -d | -b | -a | -s [-c cpu]]

DESCRIPTION
  This command collaborates the data in an irq_desc_t, along with its
  associated hw_interrupt_type and irqaction structure data, into a
  consolidated per-IRQ display.  For kernel versions 2.6.37 and later
  the display consists of the irq_desc/irq_data address, its irqaction
  address(es), and the irqaction name strings.  Alternatively, the
  intel interrupt descriptor table, bottom half data, cpu affinity for
  in-use irqs, or kernel irq stats may be displayed.  If no index value
  argument(s) nor any options are entered, the IRQ data for all IRQs will
  be displayed.

    index   a valid IRQ index.
       -u   dump data for in-use IRQs only.
       -d   dump the intel interrupt descriptor table.
       -b   dump bottom half data.
       -a   dump cpu affinity for in-use IRQs.
       -s   dump the kernel irq stats; if no cpu specified with -c, the
            irq stats of all cpus will be displayed.
   -c cpu   only usable with the -s option, dump the irq stats of the
            specified cpu[s]; cpu can be specified as "1,3,5", "1-3",
            or "1,3,5-7,10".

EXAMPLES
  Display the relevant data for IRQ 18 from a pre-2.6.37 kernel:

    crash> irq 18
        IRQ: 18
     STATUS: 0
    HANDLER: c02301e0 
             typename: c01f9e0c  "IO-APIC-level"
              startup: c0110234 
             shutdown: c01101cc 
               handle: c0110518 
               enable: c0110234 
              disable: c01101cc 
     ACTION: c009c6b0
              handler: c01ce818 
                flags: 4000000  (SA_SHIRQ)
                 mask: 0
                 name: c0217780  "aic7xxx"
               dev_id: c0090078
                 next: c009c770
     ACTION: c009c770
              handler: c01ce818 
                flags: 4000000  (SA_SHIRQ)
                 mask: 0
                 name: c0217780  "aic7xxx"
               dev_id: c0091078
                 next: 0
      DEPTH: 0

  Display the relevant data for IRQ 21 from a 2.6.37 kernel:

    crash> irq 21
     IRQ   IRQ_DESC/_DATA      IRQACTION      NAME
     21   ffff88003787f780  ffff8800379a8b40  "ehci_hcd:usb2"
                            ffff8800379cbac0  "uhci_hcd:usb5"
                            ffff8800379cb140  "uhci_hcd:usb7"

  Display the intel interrupt descriptor table entries:

    crash> irq -d
      [0] divide_error
      [1] debug
      [2] nmi
      [3] int3
      [4] overflow
      [5] bounds
      [6] invalid_op
      [7] device_not_available
      [8] double_fault
      [9] coprocessor_segment_overrun
     [10] invalid_TSS
     [11] segment_not_present
     [12] stack_segment
     [13] general_protection
     [14] page_fault
     [15] spurious_interrupt_bug
     [16] coprocessor_error
     [17] alignment_check
     [18] ignore_int
     [19] ignore_int
     [20] ignore_int
     [21] ignore_int
    ...

    [250] IRQ0xda_interrupt
    [251] IRQ0xdb_interrupt
    [252] IRQ0xdc_interrupt
    [253] IRQ0xdd_interrupt
    [254] IRQ0xde_interrupt
    [255] spurious_interrupt

  Display the bottom half data:

    crash> irq -b
    SOFTIRQ_VEC      ACTION    
        [0]     ffffffff81068f60 
        [1]     ffffffff81071b80 
        [2]     ffffffff813e6f30 
        [3]     ffffffff813ee370 
        [4]     ffffffff81211a60 
        [5]     ffffffff812122f0 
        [6]     ffffffff81069090 
        [7]     ffffffff81058830 
        [8]     ffffffff81087f00 
        [9]     ffffffff810ca7a0 

  Display the cpu affinity for in-use IRQs:

    crash> irq -a
    IRQ NAME                 AFFINITY
      0 timer                0-23
      1 i8042                0-23
      8 rtc0                 0-23
      9 acpi                 0-23
     16 ehci_hcd:usb2,uhci_hcd:usb3,uhci_hcd:usb6 0,6,18
     17 uhci_hcd:usb4,uhci_hcd:usb7 0-23
     18 ehci_hcd:usb1,uhci_hcd:usb5,uhci_hcd:usb8,ioc0 0,11,23
     24 dmar0                0
     35 pciehp               0-23
     36 pciehp               0-23
     37 pciehp               0-23
     38 pciehp               0-23
     39 megasas              0-5,12-17
     40 lpfc:sp              0-5,12-17
     41 lpfc:fp              0,6-11,18-23
     42 lpfc:sp              0,6-11,18-23
     43 lpfc:fp              0,6-11,18-23
    ...

     80 ioat-msix            0-23
     81 ioat-msix            0-23
     82 ioat-msix            0-23
     83 ioat-msix            0-23
     84 ioat-msix            0-23
     85 ioat-msix            0-23
     86 ioat-msix            0-23
     87 ioat-msix            0-23
     88 eth4                 0,17

  Display the kernel irq stats:

    crash>irq -c 0,2 -s
               CPU0       CPU2
      0: 2068161471          0 IR-IO-APIC-edge     timer
      1:          9          0 IR-IO-APIC-edge     i8042
      8:          1          0 IR-IO-APIC-edge     rtc0
      9:          0          0 IR-IO-APIC-fasteoi  acpi
     16:         36          0 IR-IO-APIC-fasteoi  ehci_hcd:usb2
    ...

     85:          3          0 IR-PCI-MSI-edge     ioat-msix
     86:          3          0 IR-PCI-MSI-edge     ioat-msix
     87:          3          0 IR-PCI-MSI-edge     ioat-msix
     88:         24        295 IR-PCI-MSI-edge     eth4



 17. ) Documentation for crash command kmem:


NAME
  kmem - kernel memory

SYNOPSIS
  kmem [-f|-F|-p|-c|-C|-i|-s|-S|-v|-V|-n|-z-o] [slab] [[-P] address]
       [-g [flags]]

DESCRIPTION
  This command displays information about the use of kernel memory.

        -f  displays the contents of the system free memory headers.
            also verifies that the page count equals nr_free_pages.
        -F  same as -f, but also dumps all pages linked to that header.
        -p  displays basic information about each page in the system
            mem_map[] array.
        -c  walks through the page_hash_table and verifies page_cache_size.
        -C  same as -c, but also dumps all pages in the page_hash_table.
        -i  displays general memory usage information
        -s  displays basic kmalloc() slab data.
        -S  displays all kmalloc() slab data, including all slab objects,
            and whether each object is in use or is free.  If CONFIG_SLUB,
            slab data for each per-cpu slab is displayed, along with the
            address of each kmem_cache_node, its count of full and partial
            slabs, and a list of all tracked slabs.
        -v  displays the vmlist entries.
        -V  displays the kernel vm_stat table if it exists, the cumulative
            page_states counter values if they exist, and/or the cumulative
            vm_event_states counter values if they exist.
        -n  display memory node data (if supported).
        -z  displays per-zone memory statistics.
        -o  displays each cpu's offset value that is added to per-cpu symbol
            values to translate them into kernel virtual addresses.
        -g  displays the enumerator value of all bits in the page structure's
            "flags" field.
     flags  when used with -g, translates all bits in this hexadecimal page
            structure flags value into its enumerator values.
      slab  when used with -s or -S, limits the command to only the slab cache
            of name "slab".  If the slab argument is "list", then
            all slab cache names and addresses are listed.
        -P  declares that the following address argument is a physical address.
   address  when used without any flag, the address can be a kernel virtual,
            or physical address; a search is made through the symbol table,
            the kmalloc() slab subsystem, the free list, the page_hash_table,
            the vmalloc() vmlist subsystem, the current set of task_structs
            and kernel stacks, and the mem_map array.  If found in any of
            those areas, the information will be dumped in the same manner as
            if the location-specific flags were used; if contained within a
            curent task_struct or kernel stack, that task's context will be
            displayed.
   address  when used with -s or -S, searches the kmalloc() slab subsystem
            for the slab containing of this virtual address, showing whether
            it is in use or free.
   address  when used with -f, the address can be either a page pointer,
            a physical address, or a kernel virtual address; the free_area
            header containing the page (if any) is displayed.
   address  when used with -p, the address can be either a page pointer, a
            physical address, or a kernel virtual address; its basic mem_map
            page information is displayed.
   address  when used with -c, the address must be a page pointer address;
            the page_hash_table entry containing the page is displayed.
   address  when used with -l, the address must be a page pointer address;
            the page address is displayed if it is contained with the list.
   address  when used with -v, the address can be a mapped kernel virtual
            address or physical address; the vmlist containing the address
            is displayed.

  All address arguments above must be expressed in hexadecimal format.

EXAMPLES
  Display memory usage information:

    crash> kmem -i
                  PAGES        TOTAL      PERCENTAGE
     TOTAL MEM    63602     248.4 MB         ----
          FREE      993       3.9 MB    1% of TOTAL MEM
          USED    62609     244.6 MB   98% of TOTAL MEM
        SHARED    34035     132.9 MB   53% of TOTAL MEM
       BUFFERS    10928      42.7 MB   17% of TOTAL MEM
        CACHED    35249     137.7 MB   55% of TOTAL MEM
          SLAB     2823        11 MB    4% of TOTAL MEM
   
    TOTAL HIGH        0            0    0% of TOTAL MEM
     FREE HIGH        0            0    0% of TOTAL HIGH
     TOTAL LOW    63602     248.4 MB  100% of TOTAL MEM
      FREE LOW      993       3.9 MB    1% of TOTAL LOW
   
    TOTAL SWAP   129792       507 MB         ----
     SWAP USED    14727      57.5 MB   11% of TOTAL SWAP
     SWAP FREE   115065     449.5 MB   88% of TOTAL SWAP
   
    ZONE  NAME      FREE   ACTIVE  INACTIVE_DIRTY  INACTIVE_CLEAN  MIN/LOW/HIGH
      0   DMA        240     1166               7             161  128/256/384
      1   Normal     753    17009           27834               0  255/510/765
      2   HighMem      0        0               0               0     0/0/0   

  Display and verify free memory data:

    crash> kmem -f
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0    
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb004           2      2
      1     8k      c02eb010           3      6
      2    16k      c02eb01c           5     20
      3    32k      c02eb028           4     32
      4    64k      c02eb034           5     80
      5   128k      c02eb040           3     96
      6   256k      c02eb04c           3    192
      7   512k      c02eb058           1    128
      8  1024k      c02eb064           1    256
      9  2048k      c02eb070           5   2560

    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      1   Normal    225280  202269  c4044040    1000000        4096  
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb0b8           1      1
      1     8k      c02eb0c4           2      4
      2    16k      c02eb0d0           0      0
      3    32k      c02eb0dc           1      8
      4    64k      c02eb0e8           1     16
      5   128k      c02eb0f4           0      0
      6   256k      c02eb100           0      0
      7   512k      c02eb10c           0      0
      8  1024k      c02eb118           0      0
      9  2048k      c02eb124         395 202240

    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      2   HighMem   819200  748686  c4ee0040    38000000      229376 
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb16c          10     10
      1     8k      c02eb178           2      4
      2    16k      c02eb184           0      0
      3    32k      c02eb190           2     16
      4    64k      c02eb19c           1     16
      5   128k      c02eb1a8           1     32
      6   256k      c02eb1b4           1     64
      7   512k      c02eb1c0           0      0
      8  1024k      c02eb1cc           0      0
      9  2048k      c02eb1d8        1462 748544
   
    nr_free_pages: 954327  (verified)

  Dump all the base addresses of each free memory area from above:

    crash> kmem -F
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0    
    AREA  SIZE  FREE_AREA_STRUCT
      0     4k      c02eb004     
    c400ded8
    c4042528
    AREA  SIZE  FREE_AREA_STRUCT
      1     8k      c02eb010     
    c400de50
    c400cee8
    c40424a0
    AREA  SIZE  FREE_AREA_STRUCT
      2    16k      c02eb01c     
    c400dd40
    c400cf70
    c40425b0
    c400f7d0
    c40028a0
    AREA  SIZE  FREE_AREA_STRUCT
      3    32k      c02eb028     
    c4042280
    c400f8e0
    c4002680
    c4000260
    AREA  SIZE  FREE_AREA_STRUCT
      4    64k      c02eb034     
    c400d080
    c4041e40
    ...

  Dump the mem_map[] array:

    crash> kmem -p
          PAGE       PHYSICAL      MAPPING       INDEX CNT FLAGS
    ffffea0000000000        0                0        0  0 0
    ffffea0000000038     1000                0        0  1 400
    ffffea0000000070     2000                0        0  1 400
    ffffea00000000a8     3000                0        0  1 400
    ffffea00000000e0     4000                0        0  1 400
    ffffea0000000118     5000                0        0  1 400
    ffffea0000000150     6000                0        0  1 400
    ffffea0000000188     7000                0        0  1 80
    ffffea00000001c0     8000                0        0  1 400
    ffffea00000001f8     9000                0        0  1 80
    ffffea0000000230     a000                0        0  1 80
    ffffea0000000268     b000 ffff880012d9bd68     695b  1 2002c
    ffffea00000002a0     c000                0        0  1 80
    ffffea00000002d8     d000 ffff88002a9ee210        9  1 2002c
    ffffea0000000310     e000 ffff880010b265d8      33c  1 2002c
    ffffea0000000348     f000 ffff88001404dd68      2d1  2 868
    ffffea0000000380    10000 ffff88001404dd68      2d6  2 868
    ffffea00000003b8    11000 ffff88001404dd68      2d7  2 868
    ffffea00000003f0    12000 ffff88001404dd68      2d8  2 868
    ffffea0000000428    13000 ffff88001404dd68      2d9  2 868
    ffffea0000000460    14000 ffff88001404dd68      2da  2 868
    ffffea0000000498    15000 ffff88001404dd68      2db  2 868
    ffffea00000004d0    16000 ffff88001404dd68      2dc  2 868
    ...
   
  Use the commands above with a page pointer or a physical address argument:

    crash> kmem -f c40425b0
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0    
    AREA  SIZE  FREE_AREA_STRUCT
      2    16k      c02eb01c     
    c40425b0  (c40425b0 is 1st of 4 pages)

    crash> kmem -p c035de00
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c035de00   50c0000         0    129000  0  uptodate

    crash> kmem -p 50c0000
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c035de00   50c0000         0    129000  0  uptodate

  Display the vmlist entry data:

    crash> kmem -v
    VM_STRUCT      ADDRESS RANGE       SIZE
     c009c560   c8000000 - c8002000    8192
     c009c620   c8002000 - c8004000    8192
     c009cda0   c8004000 - c8016000   73728
     c009cd70   c8016000 - c8019000   12288
     c009cf80   c8019000 - c801b000    8192
     c009cfb0   c801b000 - c801d000    8192
     c009cef0   c801d000 - c802d000   65536
     c3afd060   c802d000 - c8032000   20480
     c3afd090   c8032000 - c8035000   12288
     c3afd0c0   c8035000 - c8037000    8192
     c3afd150   c8037000 - c8039000    8192
     c3afd180   c8039000 - c803b000    8192
     c3afd210   c803b000 - c803d000    8192
     c3afd2a0   c803d000 - c8040000   12288
     c3afd2d0   c8040000 - c8043000   12288
     c3afd300   c8043000 - c8047000   16384
     c3afddb0   c8047000 - c804d000   24576
     c2f8a320   c804d000 - c805c000   61440
     c2f8a380   c805c000 - c8065000   36864
     c2f8a3b0   c8065000 - c806e000   36864
     c2f8aa70   c806e000 - c8095000  159744
     c2f8ab60   c8095000 - c8097000    8192
     c2f519e0   c8097000 - c8099000    8192

  Dump the vm_table contents:

    crash> kmem -V
           NR_ANON_PAGES: 38989
          NR_FILE_MAPPED: 3106
           NR_FILE_PAGES: 169570
                 NR_SLAB: 32439
            NR_PAGETABLE: 1181
           NR_FILE_DIRTY: 4633
            NR_WRITEBACK: 0
         NR_UNSTABLE_NFS: 0
               NR_BOUNCE: 0
                NUMA_HIT: 63545992
               NUMA_MISS: 0
            NUMA_FOREIGN: 0
     NUMA_INTERLEAVE_HIT: 24002
              NUMA_LOCAL: 63545992
              NUMA_OTHER: 0

  Determine (and verify) the page cache size:

    crash> kmem -c
    page_cache_size: 18431 (verified)

  Dump all pages in the page_hash_table:

    crash> kmem -C
    page_hash_table[0]
    c0325b40
    c03a0598
    c03b4070
    c0364c28
    c0357690
    c02ef338
    c02d7c60
    c02c11e0
    c02a3d70
    page_hash_table[1]
    c0394ce8
    c03c4218
    c03b4048
    c0364c00
    c0357668
    c02d6e50
    c02d7dc8
    c02c0cb8
    c02db630
    c02ebad0
    page_hash_table[2]
    c037e808
    c034e248
    c03b4020
    c02ec868
    c03baa60
    ...
    page_hash_table[2047]
    c033a798
    c0390b48
    c03b4098
    c0364890
    c03576b8
    c02d2c38
    c02d7c88
    c02de5d8
   
    page_cache_size: 18437 (verified)
   
  Find the page_hash_table entry containing page c03576b8:

    crash> kmem -c c03576b8
    page_hash_table[2047]
    c03576b8

  Display kmalloc() slab data:

    crash> kmem -s
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c02eadc0 kmem_cache               232         58        68      4     4k
    f79c2888 ip_vs_conn               128          0         0      0     4k
    f79c2970 tcp_tw_bucket             96          0         0      0     4k
    f79c2a58 tcp_bind_bucket           32         12       565      5     4k
    f79c2b40 tcp_open_request          64          0        59      1     4k
    f79c2c28 inet_peer_cache           64          1        59      1     4k
    f79c2d10 ip_fib_hash               32         11       339      3     4k
    f79c2df8 ip_dst_cache             160          8       120      5     4k
    f79c2ee0 arp_cache                128          1        30      1     4k
    c8402970 blkdev_requests           96      30208     37800    945     4k
    c8402a58 nfs_read_data            384          0         0      0     4k
    c8402b40 nfs_write_data           384          0         0      0     4k
    c8402c28 nfs_page                  96          0         0      0     4k
    c8402d10 dnotify cache             20          0         0      0     4k
    c8402df8 file lock cache           92          3       336      8     4k
    c8402ee0 fasync cache              16          0         0      0     4k
    c84027a0 uid_cache                 32          3       339      3     4k
    c84026b8 skbuff_head_cache        160        320       624     26     4k
    c84025d0 sock                     832         32       180     20     8k
    c84024e8 sigqueue                 132          0       203      7     4k
    c8402400 cdev_cache                64         19       472      8     4k
    c8402318 bdev_cache                64          8       236      4     4k
    c8402230 mnt_cache                 96         11       120      3     4k
    c8402148 inode_cache              480        817       848    106     4k
    c8402060 dentry_cache             128       1352      1470     49     4k
    c8403ee0 filp                      96        244       440     11     4k
    c8403df8 names_cache             4096          0        12     12     4k
    c8403d10 buffer_head               96      14936     16000    400     4k
    c8403c28 mm_struct                128         25       240      8     4k
    c8403b40 vm_area_struct            64        393      1298     22     4k
    c8403a58 fs_cache                  64         30       472      8     4k
    c8403970 files_cache              416         30       135     15     4k
    c8403888 signal_act              1312         32        99     33     4k
    c84037a0 size-131072(DMA)      131072          0         0      0   128k
    c84036b8 size-131072           131072          1         1      1   128k
    c84035d0 size-65536(DMA)        65536          0         0      0    64k
    c84034e8 size-65536             65536          0         0      0    64k
    c8403400 size-32768(DMA)        32768          0         0      0    32k
    c8403318 size-32768             32768          0         1      1    32k
    c8403230 size-16384(DMA)        16384          0         0      0    16k
    c8403148 size-16384             16384          0         0      0    16k
    c8403060 size-8192(DMA)          8192          0         0      0     8k
    c8401ee0 size-8192               8192          1         2      2     8k
    c8401df8 size-4096(DMA)          4096          0         0      0     4k
    c8401d10 size-4096               4096         30        30     30     4k
    c8401c28 size-2048(DMA)          2048          0         0      0     4k
    c8401b40 size-2048               2048         37       132     66     4k
    c8401a58 size-1024(DMA)          1024          0         0      0     4k
    c8401970 size-1024               1024        301       328     82     4k
    c8401888 size-512(DMA)            512          0         0      0     4k
    c84017a0 size-512                 512        141       168     21     4k
    c84016b8 size-256(DMA)            256          0         0      0     4k
    c84015d0 size-256                 256         80       435     29     4k
    c84014e8 size-128(DMA)            128          0         0      0     4k
    c8401400 size-128                 128        508       840     28     4k
    c8401318 size-64(DMA)              64          0         0      0     4k
    c8401230 size-64                   64        978      1357     23     4k
    c8401148 size-32(DMA)              32          0         0      0     4k
    c8401060 size-32                   32       1244      1808     16     4k

  Display all slab data in the "arp_cache" cache:

    crash> kmem -S arp_cache
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    f79c2ee0 arp_cache                128          1        30      1     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    f729d000  f729d0a0     30          1    29
    FREE / [ALLOCATED]
       f729d0a0  (cpu 7 cache)
       f729d120  (cpu 7 cache)
       f729d1a0  (cpu 7 cache)
       f729d220  (cpu 7 cache)
       f729d2a0  (cpu 7 cache)
       f729d320  (cpu 7 cache)
       f729d3a0  (cpu 7 cache)
       f729d420  (cpu 7 cache)
       f729d4a0  (cpu 7 cache)
       f729d520  (cpu 7 cache)
       f729d5a0  (cpu 7 cache)
       f729d620  (cpu 7 cache)
       f729d6a0  (cpu 7 cache)
       f729d720  (cpu 7 cache)
       f729d7a0  (cpu 7 cache)
       f729d820  (cpu 7 cache)
       f729d8a0  (cpu 7 cache)
       f729d920  (cpu 7 cache)
       f729d9a0  (cpu 7 cache)
       f729da20  (cpu 7 cache)
       f729daa0  (cpu 7 cache)
       f729db20  (cpu 7 cache)
       f729dba0  (cpu 7 cache)
       f729dc20  (cpu 7 cache)
       f729dca0  (cpu 7 cache)
       f729dd20  (cpu 7 cache)
       f729dda0  (cpu 7 cache)
       f729de20  (cpu 7 cache)
       f729dea0  (cpu 3 cache)
      [f729df20]

  Search the kmalloc() slab subsystem for address c3fbdb60:

    crash> kmem -s c3fbdb60
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c8402970 blkdev_requests           96      30208     37800    945     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    c3fbd020  c3fbd0e0     40         40     0
    FREE / [ALLOCATED]
      [c3fbdb60]

  Make a generic search (no flags) for the same address c3fbdb60:

    crash> kmem c3fbdb60
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c8402970 blkdev_requests           96      30208     37800    945     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    c3fbd020  c3fbd0e0     40         40     0
    FREE / [ALLOCATED]
      [c3fbdb60]

      PAGE     PHYSICAL   MAPPING    INDEX CNT FLAGS
    c410ee74    3fbd000         0         0  1 slab

  Display memory node data (if supported):

    crash> kmem -n
    NODE    SIZE    PGLIST_DATA   BOOTMEM_DATA   NODE_ZONES
      0    130933     c0332ee0      c0403a44      c0332ee0
                                                  c03333e0
                                                  c03338e0
    MEM_MAP   START_PADDR  START_MAPNR
    c1000030       0            0
   
    ZONE  NAME        SIZE   MEM_MAP  START_PADDR  START_MAPNR
      0   DMA         4096  c1000030            0            0
      1   Normal    126837  c1038030      1000000         4096
      2   HighMem        0         0            0            0

  Translate a page structure's flags field contents:

    crash> kmem -g 4080
    FLAGS: 4080
      PAGE-FLAG        BIT  VALUE
      PG_slab            7  0000080
      PG_head           14  0004000
    crash>Part 2 Continuation...