GNU Crash utility help text for all commands

I don't find GNU Crash utility help text for all commands anywhere.  So, I collected this list for public reference.



 1. ) Documentation for crash command *:


NAME
  * - pointer-to short-cut

SYNOPSIS
  * (struct or union command arguments)

DESCRIPTION
  This command is a short-cut command that replaces the requirement to enter
  "struct" or "union" command names.  For details on the arguments to
  those commands, enter "help struct" or "help union".

EXAMPLES
  Dump the page structure at address c02943c0:

    crash> *page c02943c0
    struct page {
      next = 0xc0fae740,
      prev = 0xc0018fb0,
      inode = 0x0,
      offset = 0x3f000,
      next_hash = 0xc02d6310,
      count = {
        counter = 0x1
      },
      flags = 0x310,
      wait = 0xc02943d8,
      pprev_hash = 0x0,
      buffers = 0x0
    }



 2. ) Documentation for crash command alias:


NAME
  alias - command aliases

SYNOPSIS
  alias [alias] [command string]

DESCRIPTION
  This command creates an alias for a given command string.  If no arguments
  are entered, the current list of aliases are displayed.  If one argument is
  entered, the command string for that alias, if any, is displayed.

           alias  the single word to be used as an alias
  command string  the word(s) that will be substituted for the alias

  Aliases may be created in four manners:

    1. entering the alias in $HOME/.crashrc.
    2. entering the alias in .crashrc in the current directory.
    3. executing an input file containing the alias command.
    4. during runtime with this command.

  During initialization, $HOME/.crashrc is read first, followed by the
  .crashrc file in the current directory.  Aliases in the .crashrc file
  in the current directory override those in $HOME/.crashrc.  Aliases
  entered with this command or by runtime input file override those
  defined in either .crashrc file.  Aliases may be deleted by entering an
  empty string for the second argument.  If redirection characters are to
  be part of the command string, the command string must be enclosed by
  quotation marks.

  Note that there are a number of helpful built-in aliases -- see the
  first example below.

EXAMPLES
  Display the currently-defined aliases, which in this example, only
  consist of the built-in aliases:

    crash> alias
    ORIGIN   ALIAS    COMMAND
    builtin  man      help
    builtin  ?        help
    builtin  quit     q
    builtin  sf       set scroll off
    builtin  sn       set scroll on
    builtin  hex      set radix 16
    builtin  dec      set radix 10
    builtin  g        gdb
    builtin  px       p -x
    builtin  pd       p -d
    builtin  for      foreach
    builtin  size     *
    builtin  dmesg    log
    builtin  lsmod    mod
    builtin  last     ps -l

  Create a new alias to be added to the list:

    crash> alias kp kmem -p
    ORIGIN   ALIAS    COMMAND
    runtime  kp       kmem -p

  Create an alias with redirection characters:

    crash> alias ksd "kmem -p | grep slab | grep DMA"
    ORIGIN   ALIAS    COMMAND
    runtime  ksd      kmem -p | grep slab | grep DMA

  Remove an alias:

    crash> alias kp ""
    alias deleted: kp



 3. ) Documentation for crash command ascii:


NAME
  ascii - translate a hexadecimal string to ASCII

SYNOPSIS
  ascii value ...

DESCRIPTION
  Translates 32-bit or 64-bit hexadecimal values to ASCII.  If no argument
  is entered, an ASCII chart is displayed.

EXAMPLES
  Translate the hexadecimal value of 0x62696c2f7273752f to ASCII:

    crash> ascii 62696c2f7273752f
    62696c2f7273752f: /usr/lib

  Display an ASCII chart:

    crash> ascii

          0    1   2   3   4   5   6   7
        +-------------------------------
      0 | NUL DLE  SP  0   @   P   '   p
      1 | SOH DC1  !   1   A   Q   a   q
      2 | STX DC2  "   2   B   R   b   r
      3 | ETX DC3  #   3   C   S   c   s
      4 | EOT DC4  $   4   D   T   d   t
      5 | ENQ NAK  %   5   E   U   e   u
      6 | ACK SYN  &   6   F   V   f   v
      7 | BEL ETB  `   7   G   W   g   w
      8 |  BS CAN  (   8   H   X   h   x
      9 |  HT  EM  )   9   I   Y   i   y
      A |  LF SUB  *   :   J   Z   j   z
      B |  VT ESC  +   ;   K   [   k   {
      C |  FF  FS  ,   <   L   \   l   |
      D |  CR  GS  _   =   M   ]   m   }
      E |  SO  RS  .   >   N   ^   n   ~
      F |  SI  US  /   ?   O   -   o  DEL



 4. ) Documentation for crash command bt:


NAME
  bt - backtrace

SYNOPSIS
  bt [-a|-g|-r|-t|-T|-l|-e|-E|-f|-F|-o|-O] [-R ref] [-I ip] [-S sp] [pid | task]

DESCRIPTION
  Display a kernel stack backtrace.  If no arguments are given, the stack
  trace of the current context will be displayed.

       -a  displays the stack traces of the active task on each CPU.
           (only applicable to crash dumps)
       -g  displays the stack traces of all threads in the thread group of
           the target task; the thread group leader will be displayed first.
       -r  display raw stack data, consisting of a memory dump of the two
           pages of memory containing the task_union structure.
       -t  display all text symbols found from the last known stack location
           to the top of the stack. (helpful if the back trace fails)
       -T  display all text symbols found from just above the task_struct or
           thread_info to the top of the stack. (helpful if the back trace
           fails or the -t option starts too high in the process stack).
       -l  show file and line number of each stack trace text location.
       -e  search the stack for possible kernel and user mode exception frames.
       -E  search the IRQ stacks (x86, x86_64 and ppc64), and the exception
           stacks (x86_64) for possible exception frames; all other arguments
           will be ignored since this is not a context-sensitive operation.
       -f  display all stack data contained in a frame; this option can be
           used to determine the arguments passed to each function; on ia64,
           the argument register contents are dumped.
       -F  similar to -f, except that the stack data is displayed symbolically
           when appropriate; if the stack data references a slab cache object,
           the name of the slab cache will be displayed in brackets; on ia64,
           the substitution is done to the argument register contents.
       -o  x86: use old backtrace method, permissable only on kernels that were
           compiled without the -fomit-frame_pointer.
           x86_64: use old backtrace method, which dumps potentially stale
           kernel text return addresses found on the stack.
       -O  x86: use old backtrace method by default, permissable only on kernels
           that were compiled without the -fomit-frame_pointer; subsequent usage
           of this option toggles the backtrace method.
           x86_64: use old backtrace method by default; subsequent usage of this
           option toggles the backtrace method.
   -R ref  display stack trace only if there is a reference to this symbol
           or text address.
    -I ip  use ip as the starting text location.
    -S sp  use sp as the starting stack frame address.
      pid  displays the stack trace(s) of this pid.
    taskp  displays the stack trace the the task referenced by this hexadecimal
           task_struct pointer.

  Multiple pid and taskp arguments may be specified.

  Note that all examples below are for x86 only.  The output format will differ
  for other architectures.  x86 backtraces from kernels that were compiled
  with the --fomit-frame-pointer CFLAG occasionally will drop stack frames,
  or display a stale frame reference.  When in doubt as to the accuracy of a
  backtrace, the -t or -T options may help fill in the blanks.

EXAMPLES
  Display the stack trace of the active task(s) when the kernel panicked:

    crash> bt -a
    PID: 286    TASK: c0b3a000  CPU: 0   COMMAND: "in.rlogind"
    #0 [c0b3be90] crash_save_current_state at c011aed0
    #1 [c0b3bea4] panic at c011367c
    #2 [c0b3bee8] tulip_interrupt at c01bc820
    #3 [c0b3bf08] handle_IRQ_event at c010a551
    #4 [c0b3bf2c] do_8259A_IRQ at c010a319
    #5 [c0b3bf3c] do_IRQ at c010a653
    #6 [c0b3bfbc] ret_from_intr at c0109634
       EAX: 00000000  EBX: c0e68280  ECX: 00000000  EDX: 00000004  EBP: c0b3bfbc
       DS:  0018      ESI: 00000004  ES:  0018      EDI: c0e68284
       CS:  0010      EIP: c012f803  ERR: ffffff09  EFLAGS: 00000246
    #7 [c0b3bfbc] sys_select at c012f803
    #8 [c0b3bfc0] system_call at c0109598
       EAX: 0000008e  EBX: 00000004  ECX: bfffc9a0  EDX: 00000000
       DS:  002b      ESI: bfffc8a0  ES:  002b      EDI: 00000000
       SS:  002b      ESP: bfffc82c  EBP: bfffd224
       CS:  0023      EIP: 400d032e  ERR: 0000008e  EFLAGS: 00000246 

  Display the stack traces of task f2814000 and PID 1592:

    crash> bt f2814000 1592
    PID: 1018   TASK: f2814000  CPU: 1   COMMAND: "java"
     #0 [f2815db4] schedule at c011af85
     #1 [f2815de4] __down at c010600f
     #2 [f2815e14] __down_failed at c01061b3
     #3 [f2815e24] stext_lock (via drain_cpu_caches) at c025fa55
     #4 [f2815ec8] kmem_cache_shrink_nr at c013a53e
     #5 [f2815ed8] do_try_to_free_pages at c013f402
     #6 [f2815f04] try_to_free_pages at c013f8d2
     #7 [f2815f1c] _wrapped_alloc_pages at c01406bd
     #8 [f2815f40] __alloc_pages at c014079d
     #9 [f2815f60] __get_free_pages at c014083e
    #10 [f2815f68] do_fork at c011cebb
    #11 [f2815fa4] sys_clone at c0105ceb
    #12 [f2815fc0] system_call at c010740c
        EAX: 00000078  EBX: 00000f21  ECX: bc1ffbd8  EDX: bc1ffbe0
        DS:  002b      ESI: 00000000  ES:  002b      EDI: bc1ffd04
        SS:  002b      ESP: 0807316c  EBP: 080731bc
        CS:  0023      EIP: 4012881e  ERR: 00000078  EFLAGS: 00000296

    PID: 1592   TASK: c0cec000  CPU: 3   COMMAND: "httpd"
     #0 [c0ceded4] schedule at c011af85
     #1 [c0cedf04] pipe_wait at c0153083
     #2 [c0cedf58] pipe_read at c015317f
     #3 [c0cedf7c] sys_read at c0148be6
     #4 [c0cedfc0] system_call at c010740c
        EAX: 00000003  EBX: 00000004  ECX: bffed4a3  EDX: 00000001
        DS:  002b      ESI: 00000001  ES:  002b      EDI: bffed4a3
        SS:  002b      ESP: bffed458  EBP: bffed488
        CS:  0023      EIP: 4024f1d4  ERR: 00000003  EFLAGS: 00000286

  In order to examine each stack frame's contents use the bt -f option.
  From the extra frame data that is displayed, the arguments passed to each
  function can be determined.  Re-examining the PID 1592 trace above:

    crash> bt -f 1592
    PID: 1592   TASK: c0cec000  CPU: 3   COMMAND: "httpd"
     #0 [c0ceded4] schedule at c011af85
        [RA: c0153088  SP: c0ceded4  FP: c0cedf04  SIZE: 52]
        c0ceded4: c0cedf00  c0cec000  ce1a6000  00000003 
        c0cedee4: c0cec000  f26152c0  cfafc8c0  c0cec000 
        c0cedef4: ef70a0a0  c0cec000  c0cedf28  c0cedf54 
        c0cedf04: c0153088 
     #1 [c0cedf04] pipe_wait at c0153083
        [RA: c0153184  SP: c0cedf08  FP: c0cedf58  SIZE: 84]
        c0cedf08: 00000000  c0cec000  00000000  00000000 
        c0cedf18: 00000000  c0a41fa0  c011d38b  c0394120 
        c0cedf28: 00000000  c0cec000  ceeebf30  ce4adf30 
        c0cedf38: 00000000  d4b60ce0  00000000  c0cedf58 
        c0cedf48: e204f820  ef70a040  00000001  c0cedf78 
        c0cedf58: c0153184 
     #2 [c0cedf58] pipe_read at c015317f
        [RA: c0148be8  SP: c0cedf5c  FP: c0cedf7c  SIZE: 36]
        c0cedf5c: ef70a040  c0cec000  00000000  00000000 
        c0cedf6c: 00000001  f27ae680  ffffffea  c0cedfbc 
        c0cedf7c: c0148be8 
     #3 [c0cedf7c] sys_read at c0148be6
        [RA: c0107413  SP: c0cedf80  FP: c0cedfc0  SIZE: 68]
        c0cedf80: f27ae680  bffed4a3  00000001  f27ae6a0 
        c0cedf90: 40160370  24000000  4019ba28  00000000 
        c0cedfa0: 00000000  fffffffe  bffba207  fffffffe 
        c0cedfb0: c0cec000  00000001  bffed4a3  bffed488 
        c0cedfc0: c0107413 
     #4 [c0cedfc0] system_call at c010740c
        EAX: 00000003  EBX: 00000004  ECX: bffed4a3  EDX: 00000001
        DS:  002b      ESI: 00000001  ES:  002b      EDI: bffed4a3
        SS:  002b      ESP: bffed458  EBP: bffed488
        CS:  0023      EIP: 4024f1d4  ERR: 00000003  EFLAGS: 00000286
        [RA: 4024f1d4  SP: c0cedfc4  FP: c0cedffc  SIZE: 60]
        c0cedfc4: 00000004  bffed4a3  00000001  00000001 
        c0cedfd4: bffed4a3  bffed488  00000003  0000002b 
        c0cedfe4: 0000002b  00000003  4024f1d4  00000023 
        c0cedff4: 00000286  bffed458  0000002b 

    Typically the arguments passed to a function will be the last values
    that were pushed onto the stack by the next higher-numbered function, i.e.,
    the lowest stack addresses in the frame above the called function's
    stack frame.  That can be verified by disassembling the calling function.
    For example, the arguments passed from sys_read() to pipe_read() above
    are the file pointer, the user buffer address, the count, and a pointer
    to the file structure's f_pos field.  Looking at the frame #3 data for
    sys_read(), the last four items pushed onto the stack (lowest addresses)
    are f27ae680, bffed4a3, 00000001, and f27ae6a0 -- which are the 4 arguments
    above, in that order.  Note that the first (highest address) stack content
    in frame #2 data for pipe_read() is c0148be8, which is the return address
    back to sys_read().

  Dump the text symbols found in the current context's stack:

    crash> bt -t
    PID: 1357   TASK: c1aa0000  CPU: 0   COMMAND: "lockd"
          START: schedule at c01190e0
      [c1aa1f28] dput at c0157dbc
      [c1aa1f4c] schedule_timeout at c0124cd4
      [c1aa1f78] svc_recv at cb22c4d8 [sunrpc]
      [c1aa1f98] put_files_struct at c011eb21
      [c1aa1fcc] nlmclnt_proc at cb237bef [lockd]
      [c1aa1ff0] kernel_thread at c0105826
      [c1aa1ff8] nlmclnt_proc at cb237a60 [lockd]

  Search the current stack for possible exception frames:

    crash> bt -e
    PID: 286    TASK: c0b3a000  CPU: 0   COMMAND: "in.rlogind"
   
     KERNEL-MODE EXCEPTION FRAME AT c0b3bf44:
       EAX: 00000000  EBX: c0e68280  ECX: 00000000  EDX: 00000004  EBP: c0b3bfbc
       DS:  0018      ESI: 00000004  ES:  0018      EDI: c0e68284
       CS:  0010      EIP: c012f803  ERR: ffffff09  EFLAGS: 00000246
   
     USER-MODE EXCEPTION FRAME AT c0b3bfc4:
       EAX: 0000008e  EBX: 00000004  ECX: bfffc9a0  EDX: 00000000
       DS:  002b      ESI: bfffc8a0  ES:  002b      EDI: 00000000
       SS:  002b      ESP: bfffc82c  EBP: bfffd224
       CS:  0023      EIP: 400d032e  ERR: 0000008e  EFLAGS: 00000246

  Display the back trace from a dumpfile that resulted from the execution
  of the crash utility's "sys -panic" command:

   crash> bt
   PID: 12523  TASK: c610c000  CPU: 0   COMMAND: "crash"
    #0 [c610de64] die at c01076ec
    #1 [c610de74] do_invalid_op at c01079bc
    #2 [c610df2c] error_code (via invalid_op) at c0107256
       EAX: 0000001d  EBX: c024a4c0  ECX: c02f13c4  EDX: 000026f6  EBP: c610c000
       DS:  0018      ESI: 401de2e0  ES:  0018      EDI: c610c000
       CS:  0010      EIP: c011bbb4  ERR: ffffffff  EFLAGS: 00010296
    #3 [c610df68] panic at c011bbb4
    #4 [c610df78] do_exit at c011f1fe
    #5 [c610dfc0] system_call at c0107154
       EAX: 00000001  EBX: 00000000  ECX: 00001000  EDX: 401df154
       DS:  002b      ESI: 401de2e0  ES:  002b      EDI: 00000000
       SS:  002b      ESP: bffebf0c  EBP: bffebf38
       CS:  0023      EIP: 40163afd  ERR: 00000001  EFLAGS: 00000246

  Display the back trace from a dumpfile that resulted from an attempt to
  insmod the sample "crash.c" kernel module that comes as part of the
  Red Hat netdump package:

   crash> bt
   PID: 1696   TASK: c74de000  CPU: 0   COMMAND: "insmod"
    #0 [c74dfdcc] die at c01076ec
    #1 [c74dfddc] do_page_fault at c0117bbc
    #2 [c74dfee0] error_code (via page_fault) at c0107256
       EAX: 00000013  EBX: cb297000  ECX: 00000000  EDX: c5962000  EBP: c74dff28
       DS:  0018      ESI: 00000000  ES:  0018      EDI: 00000000
       CS:  0010      EIP: cb297076  ERR: ffffffff  EFLAGS: 00010282
    #3 [c74dff1c] crash_init at cb297076 [crash]
    #4 [c74dff2c] sys_init_module at c011d233
    #5 [c74dffc0] system_call at c0107154
       EAX: 00000080  EBX: 08060528  ECX: 08076450  EDX: 0000000a
       DS:  002b      ESI: 0804b305  ES:  002b      EDI: 08074ed0
       SS:  002b      ESP: bffe9a90  EBP: bffe9ac8
       CS:  0023      EIP: 4012066e  ERR: 00000080  EFLAGS: 00000246



 5. ) Documentation for crash command btop:


NAME
  btop - bytes to page

SYNOPSIS
  btop address ...

DESCRIPTION
  This command translates a hexadecimal address to its page number.

EXAMPLES
    crash> btop 512a000
    512a000: 512a



 6. ) Documentation for crash command dev:


NAME
  dev - device data

SYNOPSIS
  dev [-i | -p | -d]

DESCRIPTION
  If no argument is entered, this command dumps character and block
  device data.

    -i  display I/O port usage; on 2.4 kernels, also display I/O memory usage.
    -p  display PCI device data.
    -d  display disk I/O statistics:
         TOTAL: total number of allocated in-progress I/O requests
          SYNC: I/O requests that are synchronous
         ASYNC: I/O requests that are asynchronous
          READ: I/O requests that are reads (older kernels)
         WRITE: I/O requests that are writes (older kernels)
           DRV: I/O requests that are in-flight in the device driver

EXAMPLES
  Display character and block device data:

    crash> dev
    CHRDEV    NAME              CDEV    OPERATIONS
       1      mem             f79b83c0  memory_fops
       4      /dev/vc/0       c07bc560  console_fops
       4      tty             f7af5004  tty_fops
       4      ttyS            f7b02204  tty_fops
       5      /dev/tty        c07bc440  tty_fops
       5      /dev/console    c07bc4a0  console_fops
       5      /dev/ptmx       c07bc500  ptmx_fops
       6      lp              c5797e40  lp_fops
       7      vcs             f7b03d40  vcs_fops
      10      misc            f7f68640  misc_fops
      13      input           f79b8840  input_fops
      21      sg              f7f12840  sg_fops
      29      fb              f7f8c640  fb_fops
     128      ptm             f7b02604  tty_fops
     136      pts             f7b02404  tty_fops
     162      raw             c0693e40  raw_fops
     180      usb             f79b8bc0  usb_fops
     189      usb_device      c06a0300  usbfs_device_file_operations
     216      rfcomm          f5961a04  tty_fops
     254      pcmcia          f79b82c0  ds_fops
   
    BLKDEV    NAME             GENDISK  OPERATIONS
       1      ramdisk         f7b23480  rd_bd_op
       8      sd              f7cab280  sd_fops
       9      md              f7829b80  md_fops
      11      sr              f75c24c0  sr_bdops
      65      sd               (none) 
      66      sd               (none) 
      67      sd               (none) 
      68      sd               (none) 
      69      sd               (none) 
      70      sd               (none) 
      71      sd               (none) 
     128      sd               (none) 
     129      sd               (none) 
     130      sd               (none) 
     131      sd               (none) 
     132      sd               (none) 
     133      sd               (none) 
     134      sd               (none) 
     135      sd               (none) 
     253      device-mapper   c57a0ac0  dm_blk_dops
     254      mdp              (none) 

  Display PCI data:

    crash> dev -p
    PCI_DEV  BU:SL.FN CLASS: VENDOR-DEVICE
    c00051c0 00:00.0  Host bridge: Intel 440BX - 82443BX Host
    c0005250 00:01.0  PCI bridge: Intel 440BX - 82443BX AGP
    c00052e0 00:07.0  ISA bridge: Intel 82371AB PIIX4 ISA
    c0005370 00:07.1  IDE interface: Intel 82371AB PIIX4 IDE
    c0005400 00:07.2  USB Controller: Intel 82371AB PIIX4 USB
    c0005490 00:07.3  Bridge: Intel 82371AB PIIX4 ACPI
    c0005520 00:11.0  Ethernet controller: 3Com 3C905B 100bTX
    c00055b0 00:13.0  PCI bridge: DEC DC21152
    c0005640 01:00.0  VGA compatible controller: NVidia [PCI_DEVICE 28]
    c00056d0 02:0a.0  SCSI storage controller: Adaptec AIC-7890/1
    c0005760 02:0e.0  SCSI storage controller: Adaptec AIC-7880U

  Display I/O port and I/O memory usage:

    crash> dev -i
    RESOURCE    RANGE    NAME
    c03036d4  0000-ffff  PCI IO
    c0302594  0000-001f  dma1
    c03025b0  0020-003f  pic1
    c03025cc  0040-005f  timer
    c03025e8  0060-006f  keyboard
    c0302604  0080-008f  dma page reg
    c0302620  00a0-00bf  pic2
    c030263c  00c0-00df  dma2
    c0302658  00f0-00ff  fpu
    c122ff20  0170-0177  ide1
    c122f240  0213-0213  isapnp read
    c122ff40  02f8-02ff  serial(auto)
    c122ff00  0376-0376  ide1
    c03186e8  03c0-03df  vga+
    c122ff60  03f8-03ff  serial(auto)
    c123851c  0800-083f  Intel Corporation 82371AB PIIX4 ACPI
    c1238538  0840-085f  Intel Corporation 82371AB PIIX4 ACPI
    c122f220  0a79-0a79  isapnp write
    c122f200  0cf8-0cff  PCI conf1
    c1238858  dc00-dc7f  3Com Corporation 3c905B 100BaseTX [Cyclone]
    c122fc00  dc00-dc7f  00:11.0
    c12380c8  dce0-dcff  Intel Corporation 82371AB PIIX4 USB
    c1238d1c  e000-efff  PCI Bus #02
    c1237858  e800-e8ff  Adaptec AIC-7880U
    c1237458  ec00-ecff  Adaptec AHA-2940U2/W / 7890
    c1239cc8  ffa0-ffaf  Intel Corporation 82371AB PIIX4 IDE
   
    RESOURCE        RANGE        NAME
    c03036f0  00000000-ffffffff  PCI mem
    c0004000  00000000-0009ffff  System RAM
    c03026ac  000a0000-000bffff  Video RAM area
    c03026fc  000c0000-000c7fff  Video ROM
    c0302718  000c9800-000cdfff  Extension ROM
    c0302734  000ce000-000ce7ff  Extension ROM
    c0302750  000ce800-000cffff  Extension ROM
    c03026e0  000f0000-000fffff  System ROM
    c0004040  00100000-07ffdfff  System RAM
    c0302674  00100000-0028682b  Kernel code
    c0302690  0028682c-0031c63f  Kernel data
    c0004060  07ffe000-07ffffff  reserved
    c1239058  ec000000-efffffff  Intel Corporation 440BX/ZX - 82443BX/ZX Host
                                 bridge
    c1238d54  f1000000-f1ffffff  PCI Bus #02
    c1239554  f2000000-f5ffffff  PCI Bus #01
    c1237074  f4000000-f5ffffff  nVidia Corporation Riva TnT2 [NV5]
    c1238d38  fa000000-fbffffff  PCI Bus #02
    c1237874  faffe000-faffefff  Adaptec AIC-7880U
    c127ec40  faffe000-faffefff  aic7xxx
    c1237474  fafff000-faffffff  Adaptec AHA-2940U2/W / 7890
    c127eec0  fafff000-faffffff  aic7xxx
    c1239538  fc000000-fdffffff  PCI Bus #01
    c1237058  fc000000-fcffffff  nVidia Corporation Riva TnT2 [NV5]
    c1238874  fe000000-fe00007f  3Com Corporation 3c905B 100BaseTX [Cyclone]
    c0004080  fec00000-fec0ffff  reserved
    c00040a0  fee00000-fee0ffff  reserved
    c00040c0  ffe00000-ffffffff  reserved

  Display disk I/O statistics:

    crash> dev -d
    MAJOR GENDISK            NAME     REQUEST QUEUE      TOTAL  READ WRITE   DRV
        2 0xffff81012d8a5000 fd0      0xffff81012dc053c0    12     0    12     0
       22 0xffff81012dc6b000 hdc      0xffff81012d8ae340     2     2     0     0
        8 0xffff81012dd71000 sda      0xffff81012d8af040     6     0     6     6
        8 0xffff81012dc77000 sdb      0xffff81012d8b5740     0     0     0     0
        8 0xffff81012d8d0c00 sdc      0xffff81012d8ae9c0     0     0     0     0



 7. ) Documentation for crash command dis:


NAME
  dis - disassemble

SYNOPSIS
  dis [-rludx][-b [num]] [address | symbol | (expression)] [count]

DESCRIPTION
  This command disassembles source code instructions starting (or ending) at
  a text address that may be expressed by value, symbol or expression:

            -r  (reverse) displays all instructions from the start of the
                routine up to and including the designated address.
            -l  displays source code line number data in addition to the
                disassembly output.
            -u  address is a user virtual address in the current context;
                otherwise the address is assumed to be a kernel virtual address.
                If this option is used, then -r and -l are ignored.
            -x  override default output format with hexadecimal format.
            -d  override default output format with decimal format.
      -b [num]  modify the pre-calculated number of encoded bytes to skip after
                a kernel BUG ("ud2a") instruction; with no argument, displays
                the current number of bytes being skipped. (x86 and x86_64 only)
       address  starting hexadecimal text address.
        symbol  symbol of starting text address.  On ppc64, the symbol
                preceded by '.' is used.
  (expression)  expression evaluating to a starting text address.
         count  the number of instructions to be disassembled (default is 1).
                If no count argument is entered, and the starting address
                is entered as a text symbol, then the whole routine will be
                disassembled.  The count argument is ignored when used with
                the -r option.

EXAMPLES
  Disassemble the sys_signal() routine without, and then with, line numbers:

    crash> dis sys_signal
    0xc0112c88 :        push   %ebp
    0xc0112c89 :      mov    %esp,%ebp
    0xc0112c8b :      sub    $0x28,%esp
    0xc0112c8e :      mov    0xc(%ebp),%eax
    0xc0112c91 :      mov    %eax,0xffffffec(%ebp)
    0xc0112c94 :     movl   $0xc0000000,0xfffffff0(%ebp)
    0xc0112c9b :     lea    0xffffffd8(%ebp),%eax
    0xc0112c9e :     push   %eax
    0xc0112c9f :     lea    0xffffffec(%ebp),%eax
    0xc0112ca2 :     push   %eax
    0xc0112ca3 :     pushl  0x8(%ebp)
    0xc0112ca6 :     call   0xc01124b8
    0xc0112cab :     test   %eax,%eax
    0xc0112cad :     jne    0xc0112cb2
    0xc0112caf :     mov    0xffffffd8(%ebp),%eax
    0xc0112cb2 :     leave
    0xc0112cb3 :     ret

    crash> dis -l sys_signal
    /usr/src/linux-2.2.5/kernel/signal.c: 1074
    0xc0112c88 :        push   %ebp
    0xc0112c89 :      mov    %esp,%ebp
    0xc0112c8b :      sub    $0x28,%esp
    0xc0112c8e :      mov    0xc(%ebp),%eax
    /usr/src/linux-2.2.5/kernel/signal.c: 1078
    0xc0112c91 :      mov    %eax,0xffffffec(%ebp)
    /usr/src/linux-2.2.5/kernel/signal.c: 1079
    0xc0112c94 :     movl   $0xc0000000,0xfffffff0(%ebp)
    /usr/src/linux-2.2.5/kernel/signal.c: 1081
    0xc0112c9b :     lea    0xffffffd8(%ebp),%eax
    0xc0112c9e :     push   %eax
    0xc0112c9f :     lea    0xffffffec(%ebp),%eax
    0xc0112ca2 :     push   %eax
    0xc0112ca3 :     pushl  0x8(%ebp)
    0xc0112ca6 :     call   0xc01124b8
    /usr/src/linux-2.2.5/kernel/signal.c: 1083
    0xc0112cab :     test   %eax,%eax
    0xc0112cad :     jne    0xc0112cb2
    0xc0112caf :     mov    0xffffffd8(%ebp),%eax
    /usr/src/linux-2.2.5/kernel/signal.c: 1084
    0xc0112cb2 :     leave
    0xc0112cb3 :     ret

  Given a return address expression of "do_no_page+65", find out the
  function that do_no_page() calls by using the reverse flag:

    crash> dis -r (do_no_page+65)
    0xc011ea68 :        push   %ebp
    0xc011ea69 :      mov    %esp,%ebp
    0xc011ea6b :      push   %edi
    0xc011ea6c :      push   %esi
    0xc011ea6d :      push   %ebx
    0xc011ea6e :      mov    0xc(%ebp),%ebx
    0xc011ea71 :      mov    0x10(%ebp),%edx
    0xc011ea74 :     mov    0x14(%ebp),%edi
    0xc011ea77 :     mov    0x28(%ebx),%eax
    0xc011ea7a :     test   %eax,%eax
    0xc011ea7c :     je     0xc011ea85
    0xc011ea7e :     mov    0x18(%eax),%ecx
    0xc011ea81 :     test   %ecx,%ecx
    0xc011ea83 :     jne    0xc011eab0
    0xc011ea85 :     mov    $0xffffe000,%eax
    0xc011ea8a :     and    %esp,%eax
    0xc011ea8c :     decl   0x30(%eax)
    0xc011ea8f :     jns    0xc011ea9a
    0xc011ea91 :     lock btrl $0x0,0xc022fb60
    0xc011ea9a :     push   %edi
    0xc011ea9b :     mov    0x18(%ebp),%esi
    0xc011ea9e :     push   %esi
    0xc011ea9f :     push   %ebx
    0xc011eaa0 :     mov    0x8(%ebp),%esi
    0xc011eaa3 :     push   %esi
    0xc011eaa4 :     call   0xc011e9e4
    0xc011eaa9 :     jmp    0xc011eb47
   
  Disassemble 10 instructions starting at user virtual address 0x81ec624:

    crash> dis -u 81ec624 10
    0x81ec624:      push   %ebp
    0x81ec625:      mov    %esp,%ebp
    0x81ec627:      sub    $0x18,%esp
    0x81ec62a:      movl   $0x1,0x8(%ebp)
    0x81ec631:      mov    0x82f9040,%eax
    0x81ec636:      mov    0x10(%eax),%edx
    0x81ec639:      and    $0x100,%edx
    0x81ec63f:      mov    0x14(%eax),%ecx
    0x81ec642:      and    $0x0,%ecx
    0x81ec645:      mov    %ecx,%eax

  Override the current decimal output radix format:

    crash> dis sys_read 10 -x
    0xffffffff8001178f :  push   %r13
    0xffffffff80011791 :      mov    %rsi,%r13
    0xffffffff80011794 :      push   %r12
    0xffffffff80011796 :      mov    $0xfffffffffffffff7,%r12
    0xffffffff8001179d :      push   %rbp
    0xffffffff8001179e :      mov    %rdx,%rbp
    0xffffffff800117a1 :     push   %rbx
    0xffffffff800117a2 :     sub    $0x18,%rsp
    0xffffffff800117a6 :     lea    0x14(%rsp),%rsi
    0xffffffff800117ab :     callq  0xffffffff8000b5b4



 8. ) Documentation for crash command eval:


NAME
  eval - evaluate

SYNOPSIS
  eval [-b][-l] (expression) | value

DESCRIPTION
  This command evaluates an expression or numeric value, and displays its
  result in hexadecimal, decimal, octal and binary. If the resultant value
  is an integral number of gigabytes, megabytes, or kilobytes, a short-hand
  translation of the number will also be shown next to the hexadecimal
  value.  If the most significant bit is set, the decimal display will show
  both unsigned and signed (negative) values.  Expressions must of the format
  (x operator y), where "x" and "y" may be either numeric values or
  symbols.  The list of operators are:

                     +  -  &  |  ^  *  %  /  <<  >>

  Enclosing the expression within parentheses is optional except when the
  "|", "<<" or ">>" operators are used.  The single "value" argument may
  be a number or symbol.  Number arguments must be hexadecimal or decimal.
  A leading "0x" identifies a number as hexadecimal, but is not required
  when obvious.  Numbers may be followed by the letters "k" or "K", "m"
  or "M", and "g" or "G", which multiplies the value by a factor of 1024,
  1 megabyte or 1 gigabyte, respectively.  Numeric arguments may be preceded
  by the one's complement operator ~.

    -b  Indicate which bit positions in the resultant value are set.
    -l  Numeric arguments are presumed to be 64-bit values, and the result
        will be expressed as a 64-bit value. (ignored on 64-bit processors)
        However, if either operand or the resultant value are 64-bit values,
        then the result will be also be expressed as a 64-bit value.

 The -b and -l options must precede the expression or value arguments.

EXAMPLES
   crash> eval 128m
   hexadecimal: 8000000  (128MB)
       decimal: 134217728 
         octal: 1000000000
        binary: 00001000000000000000000000000000
   
   crash> eval 128 * 1m
   hexadecimal: 8000000  (128MB)
       decimal: 134217728 
         octal: 1000000000
        binary: 00001000000000000000000000000000
   
   crash> eval (1 << 27)
   hexadecimal: 8000000  (128MB)
       decimal: 134217728 
         octal: 1000000000
        binary: 00001000000000000000000000000000
  
   crash> eval (1 << 32)
   hexadecimal: 100000000  (4GB)
       decimal: 4294967296
         octal: 40000000000
        binary: 0000000000000000000000000000000100000000000000000000000000000000

   crash> eval -b 41dc065
   hexadecimal: 41dc065
       decimal: 69058661 
         octal: 407340145
        binary: 00000100000111011100000001100101
      bits set: 26 20 19 18 16 15 14 6 5 2 0

   crash> eval -lb 64g
   hexadecimal: 1000000000  (64GB)
       decimal: 68719476736
         octal: 1000000000000
        binary: 0000000000000000000000000001000000000000000000000000000000000000
      bits set: 36



 9. ) Documentation for crash command exit:


NAME
  exit - exit this session

SYNOPSIS
  exit 

DESCRIPTION
  Bail out of the current crash session.

NOTE
  This command is equivalent to the "q" command.



 10. ) Documentation for crash command extend:


NAME
  extend - extend the crash command set

SYNOPSIS
  extend [shared-object ...] | [-u [shared-object ...]]

DESCRIPTION
  This command dynamically loads or unloads crash extension shared object
  libraries:

    shared-object     load the specified shared object file; more than one
                      one object file may be entered.
    -u shared-object  unload the specified shared object file; if no file
                      arguments are specified, unload all objects.

  If the shared-object filename is not expressed with a fully-qualified
  pathname, the following directories will be searched in the order shown,
  and the first instance of the file that is found will be selected:

     1. the current working directory
     2. the directory specified in the CRASH_EXTENSIONS environment variable
     3. /usr/lib64/crash/extensions (64-bit architectures)
     4. /usr/lib/crash/extensions

  If no arguments are entered, the current set of shared object files and
  a list of their commands will be displayed.  The registered commands
  contained in each shared object file will appear automatically in the
  "help" command screen.

  An example of a shared object prototype file, and how to compile it
  into a shared object, is appended below.

EXAMPLES
  Load two shared object files:

    crash> extend extlib1.so extlib2.so
    ./extlib1.so: shared object loaded
    ./extlib2.so: shared object loaded

  Display the current set of shared object files and their commands:

    crash> extend
    SHARED OBJECT  COMMANDS
    ./extlib1.so   echo util bin
    ./extlib2.so   smp show

  Unload one of the shared object files:

    crash> extend -u extlib1.so
    ./extlib1.so: shared object unloaded

  Unload all currently-loaded object files:

    crash> extend -u
    ./extlib2.so: shared object unloaded

CREATING A SHARED OBJECT
  The extend command loads shared object files using dlopen(3), which in
  turn calls the shared object's _init() function.  The shared object's _init()
  function should register its command set by calling register_extension(),
  passing it a pointer to an array of one or more structures of the
  following type:

    struct command_table_entry {
            char *name;
            cmd_func_t func;
            char **help_data,
            ulong flags;
    };

  Each command_table_entry structure contains the ASCII name of a command,
  the command's function address, a pointer to an array of help data strings,
  and a flags field.  The help_data field is optional; if it is non-NULL, it
  should point to an array of character strings used by the "help"
  command, and during command failures.  The flags field currently has one
  available bit setting, REFRESH_TASK_TABLE, which should be set if it is
  preferable to reload the current set of running processes just prior to
  executing the command (on a live system).  Terminate the array of
  command_table_entry structures with an entry with a NULL command name. 

  Below is an example shared object file consisting of just one command,
  called "echo", which simply echoes back all arguments passed to it.
  Note the comments contained within it for further details.  Cut and paste
  the following output into a file, and call it, for example, "echo.c".
  Then compiled in either of two manners.  Either manually like so:

  gcc -nostartfiles -shared -rdynamic -o echo.so echo.c -fPIC -D $(TARGET_CFLAGS)

  where must be one of the MACHINE_TYPE #define's in defs.h,
  and where $(TARGET_CFLAGS) is the same as it is declared in the top-level
  Makefile after a build is completed.  Or alternatively, the "echo.c" file
  can be copied into the "extensions" subdirectory, and compiled automatically
  like so:

  make extensions

  The echo.so file may be dynamically linked into crash during runtime, or
  during initialization by putting "extend echo.so" into a .crashrc file
  located in the current directory, or in the user's $HOME directory.
 
---------------------------------- cut here ----------------------------------

#include "defs.h"    /* From the crash source top-level directory */

void cmd_echo();     /* Declare the commands and their help data. */
char *help_echo[];

static struct command_table_entry command_table[] = {
        "echo", cmd_echo, help_echo, 0,           /* One or more commands, */
        NULL,                                     /* terminated by NULL, */
};


_init() /* Register the command set. */
{
        register_extension(command_table);
}

/*
 *  The _fini() function is called if the shared object is unloaded.
 *  If desired, perform any cleanups here.
 */
_fini() { }


/*
 *  Arguments are passed to the command functions in the global args[argcnt]
 *  array.  See getopt(3) for info on dash arguments.  Check out defs.h and
 *  other crash commands for usage of the myriad of utility routines available
 *  to accomplish what your task.
 */
void
cmd_echo()
{
        int c;

        while ((c = getopt(argcnt, args, "")) != EOF) {
                switch(c)
                {
                default:
                        argerrs++;
                        break;
                }
        }

        if (argerrs)
                cmd_usage(pc->curcmd, SYNOPSIS);

        while (args[optind])
                fprintf(fp, "%s ", args[optind++]);

        fprintf(fp, "\n");
}

/*
 *  The optional help data is simply an array of strings in a defined format.
 *  For example, the "help echo" command will use the help_echo[] string
 *  array below to create a help page that looks like this:
 *
 *    NAME
 *      echo - echoes back its arguments
 *
 *    SYNOPSIS
 *      echo arg ...
 *
 *    DESCRIPTION
 *      This command simply echoes back its arguments.
 *
 *    EXAMPLE
 *      Echo back all command arguments:
 *
 *        crash> echo hello, world
 *        hello, world
 *
 */

char *help_echo[] = {
        "echo",                        /* command name */
        "echoes back its arguments",   /* short description */
        "arg ...",                     /* argument synopsis, or " " if none */

        "  This command simply echoes back its arguments.",
        "\nEXAMPLE",
        "  Echo back all command arguments:\n",
        "    crash> echo hello, world",
        "    hello, world",
        NULL
};




 11. ) Documentation for crash command files:


NAME
  files - open files

SYNOPSIS
  files [-d dentry] | [-R reference] [pid | taskp] ...

DESCRIPTION
  This command displays information about open files of a context.
  It prints the context's current root directory and current working
  directory, and then for each open file descriptor it prints a pointer
  to its file struct, a pointer to its dentry struct, a pointer to the
  inode, the file type, and the pathname.  If no arguments are entered,
  the current context is used.  The -R option, typically invoked from
  "foreach files", searches for references to a supplied number, address,
  or filename argument, and prints only the essential information leading
  up to and including the reference.  The -d option is not context
  specific, and only shows the data requested.

     -d dentry  given a hexadecimal dentry address, display its inode,
                super block, file type, and full pathname.
  -R reference  search for references to this file descriptor number,
                filename, or dentry, inode, or file structure address.
           pid  a process PID.
         taskp  a hexadecimal task_struct pointer.

EXAMPLES
  Display the open files of the current context:

    crash> files
    PID: 720    TASK: c67f2000  CPU: 1   COMMAND: "innd"
    ROOT: /    CWD: /var/spool/news/articles
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  c6b9c740  c7cc45a0  c7c939e0  CHR   /dev/null
      1  c6b9c800  c537bb20  c54d0000  REG   /var/log/news/news
      2  c6df9600  c537b420  c5c36360  REG   /var/log/news/errlog
      3  c74182c0  c6ede260  c6da3d40  PIPE
      4  c6df9720  c696c620  c69398c0  SOCK
      5  c6b9cc20  c68e7000  c6938d80  SOCK
      6  c6b9c920  c7cc45a0  c7c939e0  CHR   /dev/null
      7  c6b9c680  c58fa5c0  c58a1200  REG   /var/lib/news/history
      8  c6df9f00  c6ede760  c6da3200  PIPE
      9  c6b9c6e0  c58fa140  c5929560  REG   /var/lib/news/history.dir
     10  c7fa9320  c7fab160  c7fafd40  CHR   /dev/console
     11  c6b9c7a0  c58fa5c0  c58a1200  REG   /var/lib/news/history
     12  c377ec60  c58fa5c0  c58a1200  REG   /var/lib/news/history
     13  c4528aa0  c58fa6c0  c52fbb00  REG   /var/lib/news/history.pag
     14  c6df9420  c68e7700  c6938360  SOCK
     15  c6df9360  c68e7780  c6938120  SOCK
     16  c6b9c0e0  c68e7800  c6772000  SOCK
     17  c6b9c200  c6b5f9c0  c6b5cea0  REG   /var/lib/news/active
     21  c6b9c080  c6ede760  c6da3200  PIPE

  Display the files opened by the "crond" daemon, which is PID 462:

  crash> files 462
    PID: 462    TASK: f7220000  CPU: 2   COMMAND: "crond"
    ROOT: /    CWD: /var/spool
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  f7534ae0  f7538de0  f7518dc0  CHR   /dev/console
      1  f7368f80  f72c7a40  f72f27e0  FIFO  pipe:/[1456]
      2  f74f3c80  f72c79c0  f72f2600  FIFO  pipe:/[1457]
      3  f7368b60  f72a5be0  f74300c0  REG   /var/run/crond.pid
      4  f7534360  f73408c0  f72c2840  REG   /var/log/cron
      7  f7368ce0  f72c7940  f72f2420  FIFO  pipe:/[1458]
      8  f7295de0  f72c7940  f72f2420  FIFO  pipe:/[1458]
     21  f74f36e0  f747cdc0  f747e840  CHR   /dev/null

  The -R option is typically invoked from "foreach files".  This example
  shows all tasks that have "/dev/pts/4" open:

    crash> foreach files -R pts/4
    PID: 18633  TASK: c310a000  CPU: 0   COMMAND: "crash"
    ROOT: /    CWD: /home/CVS_pool/crash
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      1  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      2  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
   
    PID: 18664  TASK: c2392000  CPU: 1   COMMAND: "less"
    ROOT: /    CWD: /home/CVS_pool/crash
     FD    FILE     DENTRY    INODE    TYPE  PATH
      1  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      2  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
   
    PID: 23162  TASK: c5088000  CPU: 1   COMMAND: "bash"
    ROOT: /    CWD: /home/CVS_pool/crash
     FD    FILE     DENTRY    INODE    TYPE  PATH
      0  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      1  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
      2  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
    255  c1412850  c2cb96d0  c2cad430  CHR   /dev/pts/4
   
    PID: 23159  TASK: c10fc000  CPU: 1   COMMAND: "xterm"
    ROOT: /    CWD: /homes/anderson/
     FD    FILE     DENTRY    INODE    TYPE  PATH
      5  c1560da0  c2cb96d0  c2cad430  CHR   /dev/pts/4

  Display information about the dentry at address f745fd60:

    crash> files -d f745fd60
     DENTRY    INODE    SUPERBLK  TYPE  PATH
     f745fd60  f7284640  f73a3e00  REG   /var/spool/lpd/lpd.lock



 12. ) Documentation for crash command foreach:


NAME
  foreach - display command data for multiple tasks in the system

SYNOPSIS
  foreach [[pid | taskp | name | [kernel | user]] ...] command [flag] [argument]

DESCRIPTION
  This command allows for a an examination of various kernel data associated
  with any, or all, tasks in the system, without having to set the context
  to each targeted task.

      pid  perform the command(s) on this PID.
    taskp  perform the command(s) on task referenced by this hexadecimal
           task_struct pointer.
     name  perform the command(s) on all commands with this name.  If the
           command name can be confused with a foreach command name, then
           precede the name string with a "\".
     user  perform the command(s) on all user (non-kernel) threads.
   kernel  perform the command(s) on all kernel threads.
   active  perform the command(s) on the active thread on each CPU.

  If none of the task-identifying arguments above are entered, the command
  will be performed on all tasks.

  command  select one or more of the following commands to be run on the tasks
           selected, or on all tasks:

             bt  run the "bt" command  (optional flags: -r -t -l -e -R -f -F -o)
             vm  run the "vm" command  (optional flags: -p -v -m -R)
           task  run the "task" command  (optional flags: -R -d -x)
          files  run the "files" command  (optional flag: -R)
            net  run the "net" command  (optional flags: -s -S -R)
            set  run the "set" command
            sig  run the "sig" command (optional flag: -g)
           vtop  run the "vtop" command  (optional flags: -c -u -k)

     flag  Pass this optional flag to the command selected.
 argument  Pass this argument to the command selected.

  A header containing the PID, task address, cpu and command name will be
  pre-pended before the command output for each selected task.  Consult the
  help page of each of the command types above for details.

EXAMPLES
  Display the stack traces for all tasks:

    crash> foreach bt
    PID: 4752   TASK: c7680000  CPU: 1   COMMAND: "xterm"
     #0 [c7681edc] schedule at c01135f6
        (void)
     #1 [c7681f34] schedule_timeout at c01131ff
        (24)
     #2 [c7681f64] do_select at c0132838
        (5, c7681fa4, c7681fa0)
     #3 [c7681fbc] sys_select at c0132dad
        (5, 8070300, 8070380, 0, 0)
     #4 [bffffb0c] system_call at c0109944
        EAX: 0000008e  EBX: 00000005  ECX: 08070300  EDX: 08070380
        DS:  002b      ESI: 00000000  ES:  002b      EDI: 00000000
        SS:  002b      ESP: bffffadc  EBP: bffffb0c
        CS:  0023      EIP: 402259ee  ERR: 0000008e  EFLAGS: 00000246
   
    PID: 557    TASK: c5600000  CPU: 0   COMMAND: "nfsd"
     #0 [c5601f38] schedule at c01135f6
        (void)
     #1 [c5601f90] schedule_timeout at c01131ff
        (c5600000)
     #2 [c5601fb8] svc_recv at c805363a
        (c0096f40, c5602800, 7fffffff, 100, c65c9f1c)
     #3 [c5601fec] (nfsd module) at c806e303
        (c5602800, c5602800, c0096f40, 6c6e0002, 50)
     #4 [c65c9f24] kernel_thread at c010834f
        (0, 0, ext2_file_inode_operations)
   
    PID: 824    TASK: c7c84000  CPU: 0   COMMAND: "mingetty"
    ...

  Display the task_struct structure for each "bash" command:

    crash> foreach bash task
    ...

  Display the open files for all tasks:

    crash> foreach files
    ...




 13. ) Documentation for crash command fuser:


NAME
  fuser - file users

SYNOPSIS
  fuser [pathname | inode]

DESCRIPTION
  This command displays the tasks using specified files or sockets.
  Tasks will be listed that reference the file as the current working
  directory, root directory, an open file descriptor, or that mmap the
  file.  If the file is held open in the kernel by the lockd server on
  behalf of a client discretionary file lock, the client hostname is
  listed.

    pathname  the full pathname of the file.
    inode     the hexadecimal inode address for the file.

EXAMPLES
  Display the tasks using file /usr/lib/libkfm.so.2.0.0

    crash> fuser /usr/lib/libkfm.so.2.0.0
     PID    TASK    COMM            USAGE
     779  c5e82000  "kwm"           mmap
     808  c5a8e000  "krootwm"       mmap
     806  c5b42000  "kfm"           mmap
     809  c5dde000  "kpanel"        mmap



 14. ) Documentation for crash command gdb:


NAME
  gdb - gdb command

SYNOPSIS
  gdb command ...

DESCRIPTION
  This command passes its arguments directly to gdb for processing.
  This is typically not necessary, but where ambiguities between crash and
  gdb command names exist, this will force the command to be executed by gdb.

EXAMPLES
    crash> gdb help
    List of classes of commands:
   
    aliases -- Aliases of other commands
    breakpoints -- Making program stop at certain points
    data -- Examining data
    files -- Specifying and examining files
    internals -- Maintenance commands
    obscure -- Obscure features
    running -- Running the program
    stack -- Examining the stack
    status -- Status inquiries
    support -- Support facilities
    tracepoints -- Tracing of program execution without stopping the program
    user-defined -- User-defined commands
   
    Type "help" followed by a class name for a list of commands in that class.
    Type "help" followed by command name for full documentation.
    Command name abbreviations are allowed if unambiguous.



 15. ) Documentation for crash command help:


NAME
  help - get help

SYNOPSIS
  help [command | all] [-]

DESCRIPTION
  When entered with no argument, a list of all currently available crash
  commands is listed.  If a name of a crash command is entered, a man-like
  page for the command is displayed.  If "all" is entered, help pages
  for all commands will be displayed.  If neither of the above is entered,
  the argument string will be passed on to the gdb help command.

  A number of internal debug, statistical, and other dumpfile related
  data is available with the following options:

    -e - extension table data
    -a - alias data
    -b - shared buffer data
    -B - build data
    -c - numargs cache
    -d - device table
    -D - dumpfile contents/statistics
    -f - filesys table
    -k - kernel_table
    -K - kernel_table (verbose)
    -M machine specific
    -m - machdep_table
    -n - dumpfile contents/statistics
    -s - symbol table data
    -v - vm_table
    -V - vm_table (verbose)
    -o - offset_table and size_table
    -t - task_table
    -x - text cache
    -T - task_table plus context_array
    -p - program_context
    -h - hash_table data
    -H - hash_table data (verbose)
    -L - LKCD page cache environment



 16. ) Documentation for crash command irq:


NAME
  irq - IRQ data

SYNOPSIS
  irq [[[index ...] | -u ] | -d | -b | -a | -s [-c cpu]]

DESCRIPTION
  This command collaborates the data in an irq_desc_t, along with its
  associated hw_interrupt_type and irqaction structure data, into a
  consolidated per-IRQ display.  For kernel versions 2.6.37 and later
  the display consists of the irq_desc/irq_data address, its irqaction
  address(es), and the irqaction name strings.  Alternatively, the
  intel interrupt descriptor table, bottom half data, cpu affinity for
  in-use irqs, or kernel irq stats may be displayed.  If no index value
  argument(s) nor any options are entered, the IRQ data for all IRQs will
  be displayed.

    index   a valid IRQ index.
       -u   dump data for in-use IRQs only.
       -d   dump the intel interrupt descriptor table.
       -b   dump bottom half data.
       -a   dump cpu affinity for in-use IRQs.
       -s   dump the kernel irq stats; if no cpu specified with -c, the
            irq stats of all cpus will be displayed.
   -c cpu   only usable with the -s option, dump the irq stats of the
            specified cpu[s]; cpu can be specified as "1,3,5", "1-3",
            or "1,3,5-7,10".

EXAMPLES
  Display the relevant data for IRQ 18 from a pre-2.6.37 kernel:

    crash> irq 18
        IRQ: 18
     STATUS: 0
    HANDLER: c02301e0 
             typename: c01f9e0c  "IO-APIC-level"
              startup: c0110234 
             shutdown: c01101cc 
               handle: c0110518 
               enable: c0110234 
              disable: c01101cc 
     ACTION: c009c6b0
              handler: c01ce818 
                flags: 4000000  (SA_SHIRQ)
                 mask: 0
                 name: c0217780  "aic7xxx"
               dev_id: c0090078
                 next: c009c770
     ACTION: c009c770
              handler: c01ce818 
                flags: 4000000  (SA_SHIRQ)
                 mask: 0
                 name: c0217780  "aic7xxx"
               dev_id: c0091078
                 next: 0
      DEPTH: 0

  Display the relevant data for IRQ 21 from a 2.6.37 kernel:

    crash> irq 21
     IRQ   IRQ_DESC/_DATA      IRQACTION      NAME
     21   ffff88003787f780  ffff8800379a8b40  "ehci_hcd:usb2"
                            ffff8800379cbac0  "uhci_hcd:usb5"
                            ffff8800379cb140  "uhci_hcd:usb7"

  Display the intel interrupt descriptor table entries:

    crash> irq -d
      [0] divide_error
      [1] debug
      [2] nmi
      [3] int3
      [4] overflow
      [5] bounds
      [6] invalid_op
      [7] device_not_available
      [8] double_fault
      [9] coprocessor_segment_overrun
     [10] invalid_TSS
     [11] segment_not_present
     [12] stack_segment
     [13] general_protection
     [14] page_fault
     [15] spurious_interrupt_bug
     [16] coprocessor_error
     [17] alignment_check
     [18] ignore_int
     [19] ignore_int
     [20] ignore_int
     [21] ignore_int
    ...

    [250] IRQ0xda_interrupt
    [251] IRQ0xdb_interrupt
    [252] IRQ0xdc_interrupt
    [253] IRQ0xdd_interrupt
    [254] IRQ0xde_interrupt
    [255] spurious_interrupt

  Display the bottom half data:

    crash> irq -b
    SOFTIRQ_VEC      ACTION    
        [0]     ffffffff81068f60 
        [1]     ffffffff81071b80 
        [2]     ffffffff813e6f30 
        [3]     ffffffff813ee370 
        [4]     ffffffff81211a60 
        [5]     ffffffff812122f0 
        [6]     ffffffff81069090 
        [7]     ffffffff81058830 
        [8]     ffffffff81087f00 
        [9]     ffffffff810ca7a0 

  Display the cpu affinity for in-use IRQs:

    crash> irq -a
    IRQ NAME                 AFFINITY
      0 timer                0-23
      1 i8042                0-23
      8 rtc0                 0-23
      9 acpi                 0-23
     16 ehci_hcd:usb2,uhci_hcd:usb3,uhci_hcd:usb6 0,6,18
     17 uhci_hcd:usb4,uhci_hcd:usb7 0-23
     18 ehci_hcd:usb1,uhci_hcd:usb5,uhci_hcd:usb8,ioc0 0,11,23
     24 dmar0                0
     35 pciehp               0-23
     36 pciehp               0-23
     37 pciehp               0-23
     38 pciehp               0-23
     39 megasas              0-5,12-17
     40 lpfc:sp              0-5,12-17
     41 lpfc:fp              0,6-11,18-23
     42 lpfc:sp              0,6-11,18-23
     43 lpfc:fp              0,6-11,18-23
    ...

     80 ioat-msix            0-23
     81 ioat-msix            0-23
     82 ioat-msix            0-23
     83 ioat-msix            0-23
     84 ioat-msix            0-23
     85 ioat-msix            0-23
     86 ioat-msix            0-23
     87 ioat-msix            0-23
     88 eth4                 0,17

  Display the kernel irq stats:

    crash>irq -c 0,2 -s
               CPU0       CPU2
      0: 2068161471          0 IR-IO-APIC-edge     timer
      1:          9          0 IR-IO-APIC-edge     i8042
      8:          1          0 IR-IO-APIC-edge     rtc0
      9:          0          0 IR-IO-APIC-fasteoi  acpi
     16:         36          0 IR-IO-APIC-fasteoi  ehci_hcd:usb2
    ...

     85:          3          0 IR-PCI-MSI-edge     ioat-msix
     86:          3          0 IR-PCI-MSI-edge     ioat-msix
     87:          3          0 IR-PCI-MSI-edge     ioat-msix
     88:         24        295 IR-PCI-MSI-edge     eth4



 17. ) Documentation for crash command kmem:


NAME
  kmem - kernel memory

SYNOPSIS
  kmem [-f|-F|-p|-c|-C|-i|-s|-S|-v|-V|-n|-z-o] [slab] [[-P] address]
       [-g [flags]]

DESCRIPTION
  This command displays information about the use of kernel memory.

        -f  displays the contents of the system free memory headers.
            also verifies that the page count equals nr_free_pages.
        -F  same as -f, but also dumps all pages linked to that header.
        -p  displays basic information about each page in the system
            mem_map[] array.
        -c  walks through the page_hash_table and verifies page_cache_size.
        -C  same as -c, but also dumps all pages in the page_hash_table.
        -i  displays general memory usage information
        -s  displays basic kmalloc() slab data.
        -S  displays all kmalloc() slab data, including all slab objects,
            and whether each object is in use or is free.  If CONFIG_SLUB,
            slab data for each per-cpu slab is displayed, along with the
            address of each kmem_cache_node, its count of full and partial
            slabs, and a list of all tracked slabs.
        -v  displays the vmlist entries.
        -V  displays the kernel vm_stat table if it exists, the cumulative
            page_states counter values if they exist, and/or the cumulative
            vm_event_states counter values if they exist.
        -n  display memory node data (if supported).
        -z  displays per-zone memory statistics.
        -o  displays each cpu's offset value that is added to per-cpu symbol
            values to translate them into kernel virtual addresses.
        -g  displays the enumerator value of all bits in the page structure's
            "flags" field.
     flags  when used with -g, translates all bits in this hexadecimal page
            structure flags value into its enumerator values.
      slab  when used with -s or -S, limits the command to only the slab cache
            of name "slab".  If the slab argument is "list", then
            all slab cache names and addresses are listed.
        -P  declares that the following address argument is a physical address.
   address  when used without any flag, the address can be a kernel virtual,
            or physical address; a search is made through the symbol table,
            the kmalloc() slab subsystem, the free list, the page_hash_table,
            the vmalloc() vmlist subsystem, the current set of task_structs
            and kernel stacks, and the mem_map array.  If found in any of
            those areas, the information will be dumped in the same manner as
            if the location-specific flags were used; if contained within a
            curent task_struct or kernel stack, that task's context will be
            displayed.
   address  when used with -s or -S, searches the kmalloc() slab subsystem
            for the slab containing of this virtual address, showing whether
            it is in use or free.
   address  when used with -f, the address can be either a page pointer,
            a physical address, or a kernel virtual address; the free_area
            header containing the page (if any) is displayed.
   address  when used with -p, the address can be either a page pointer, a
            physical address, or a kernel virtual address; its basic mem_map
            page information is displayed.
   address  when used with -c, the address must be a page pointer address;
            the page_hash_table entry containing the page is displayed.
   address  when used with -l, the address must be a page pointer address;
            the page address is displayed if it is contained with the list.
   address  when used with -v, the address can be a mapped kernel virtual
            address or physical address; the vmlist containing the address
            is displayed.

  All address arguments above must be expressed in hexadecimal format.

EXAMPLES
  Display memory usage information:

    crash> kmem -i
                  PAGES        TOTAL      PERCENTAGE
     TOTAL MEM    63602     248.4 MB         ----
          FREE      993       3.9 MB    1% of TOTAL MEM
          USED    62609     244.6 MB   98% of TOTAL MEM
        SHARED    34035     132.9 MB   53% of TOTAL MEM
       BUFFERS    10928      42.7 MB   17% of TOTAL MEM
        CACHED    35249     137.7 MB   55% of TOTAL MEM
          SLAB     2823        11 MB    4% of TOTAL MEM
   
    TOTAL HIGH        0            0    0% of TOTAL MEM
     FREE HIGH        0            0    0% of TOTAL HIGH
     TOTAL LOW    63602     248.4 MB  100% of TOTAL MEM
      FREE LOW      993       3.9 MB    1% of TOTAL LOW
   
    TOTAL SWAP   129792       507 MB         ----
     SWAP USED    14727      57.5 MB   11% of TOTAL SWAP
     SWAP FREE   115065     449.5 MB   88% of TOTAL SWAP
   
    ZONE  NAME      FREE   ACTIVE  INACTIVE_DIRTY  INACTIVE_CLEAN  MIN/LOW/HIGH
      0   DMA        240     1166               7             161  128/256/384
      1   Normal     753    17009           27834               0  255/510/765
      2   HighMem      0        0               0               0     0/0/0   

  Display and verify free memory data:

    crash> kmem -f
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0    
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb004           2      2
      1     8k      c02eb010           3      6
      2    16k      c02eb01c           5     20
      3    32k      c02eb028           4     32
      4    64k      c02eb034           5     80
      5   128k      c02eb040           3     96
      6   256k      c02eb04c           3    192
      7   512k      c02eb058           1    128
      8  1024k      c02eb064           1    256
      9  2048k      c02eb070           5   2560

    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      1   Normal    225280  202269  c4044040    1000000        4096  
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb0b8           1      1
      1     8k      c02eb0c4           2      4
      2    16k      c02eb0d0           0      0
      3    32k      c02eb0dc           1      8
      4    64k      c02eb0e8           1     16
      5   128k      c02eb0f4           0      0
      6   256k      c02eb100           0      0
      7   512k      c02eb10c           0      0
      8  1024k      c02eb118           0      0
      9  2048k      c02eb124         395 202240

    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      2   HighMem   819200  748686  c4ee0040    38000000      229376 
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb16c          10     10
      1     8k      c02eb178           2      4
      2    16k      c02eb184           0      0
      3    32k      c02eb190           2     16
      4    64k      c02eb19c           1     16
      5   128k      c02eb1a8           1     32
      6   256k      c02eb1b4           1     64
      7   512k      c02eb1c0           0      0
      8  1024k      c02eb1cc           0      0
      9  2048k      c02eb1d8        1462 748544
   
    nr_free_pages: 954327  (verified)

  Dump all the base addresses of each free memory area from above:

    crash> kmem -F
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0    
    AREA  SIZE  FREE_AREA_STRUCT
      0     4k      c02eb004     
    c400ded8
    c4042528
    AREA  SIZE  FREE_AREA_STRUCT
      1     8k      c02eb010     
    c400de50
    c400cee8
    c40424a0
    AREA  SIZE  FREE_AREA_STRUCT
      2    16k      c02eb01c     
    c400dd40
    c400cf70
    c40425b0
    c400f7d0
    c40028a0
    AREA  SIZE  FREE_AREA_STRUCT
      3    32k      c02eb028     
    c4042280
    c400f8e0
    c4002680
    c4000260
    AREA  SIZE  FREE_AREA_STRUCT
      4    64k      c02eb034     
    c400d080
    c4041e40
    ...

  Dump the mem_map[] array:

    crash> kmem -p
          PAGE       PHYSICAL      MAPPING       INDEX CNT FLAGS
    ffffea0000000000        0                0        0  0 0
    ffffea0000000038     1000                0        0  1 400
    ffffea0000000070     2000                0        0  1 400
    ffffea00000000a8     3000                0        0  1 400
    ffffea00000000e0     4000                0        0  1 400
    ffffea0000000118     5000                0        0  1 400
    ffffea0000000150     6000                0        0  1 400
    ffffea0000000188     7000                0        0  1 80
    ffffea00000001c0     8000                0        0  1 400
    ffffea00000001f8     9000                0        0  1 80
    ffffea0000000230     a000                0        0  1 80
    ffffea0000000268     b000 ffff880012d9bd68     695b  1 2002c
    ffffea00000002a0     c000                0        0  1 80
    ffffea00000002d8     d000 ffff88002a9ee210        9  1 2002c
    ffffea0000000310     e000 ffff880010b265d8      33c  1 2002c
    ffffea0000000348     f000 ffff88001404dd68      2d1  2 868
    ffffea0000000380    10000 ffff88001404dd68      2d6  2 868
    ffffea00000003b8    11000 ffff88001404dd68      2d7  2 868
    ffffea00000003f0    12000 ffff88001404dd68      2d8  2 868
    ffffea0000000428    13000 ffff88001404dd68      2d9  2 868
    ffffea0000000460    14000 ffff88001404dd68      2da  2 868
    ffffea0000000498    15000 ffff88001404dd68      2db  2 868
    ffffea00000004d0    16000 ffff88001404dd68      2dc  2 868
    ...
   
  Use the commands above with a page pointer or a physical address argument:

    crash> kmem -f c40425b0
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0    
    AREA  SIZE  FREE_AREA_STRUCT
      2    16k      c02eb01c     
    c40425b0  (c40425b0 is 1st of 4 pages)

    crash> kmem -p c035de00
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c035de00   50c0000         0    129000  0  uptodate

    crash> kmem -p 50c0000
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c035de00   50c0000         0    129000  0  uptodate

  Display the vmlist entry data:

    crash> kmem -v
    VM_STRUCT      ADDRESS RANGE       SIZE
     c009c560   c8000000 - c8002000    8192
     c009c620   c8002000 - c8004000    8192
     c009cda0   c8004000 - c8016000   73728
     c009cd70   c8016000 - c8019000   12288
     c009cf80   c8019000 - c801b000    8192
     c009cfb0   c801b000 - c801d000    8192
     c009cef0   c801d000 - c802d000   65536
     c3afd060   c802d000 - c8032000   20480
     c3afd090   c8032000 - c8035000   12288
     c3afd0c0   c8035000 - c8037000    8192
     c3afd150   c8037000 - c8039000    8192
     c3afd180   c8039000 - c803b000    8192
     c3afd210   c803b000 - c803d000    8192
     c3afd2a0   c803d000 - c8040000   12288
     c3afd2d0   c8040000 - c8043000   12288
     c3afd300   c8043000 - c8047000   16384
     c3afddb0   c8047000 - c804d000   24576
     c2f8a320   c804d000 - c805c000   61440
     c2f8a380   c805c000 - c8065000   36864
     c2f8a3b0   c8065000 - c806e000   36864
     c2f8aa70   c806e000 - c8095000  159744
     c2f8ab60   c8095000 - c8097000    8192
     c2f519e0   c8097000 - c8099000    8192

  Dump the vm_table contents:

    crash> kmem -V
           NR_ANON_PAGES: 38989
          NR_FILE_MAPPED: 3106
           NR_FILE_PAGES: 169570
                 NR_SLAB: 32439
            NR_PAGETABLE: 1181
           NR_FILE_DIRTY: 4633
            NR_WRITEBACK: 0
         NR_UNSTABLE_NFS: 0
               NR_BOUNCE: 0
                NUMA_HIT: 63545992
               NUMA_MISS: 0
            NUMA_FOREIGN: 0
     NUMA_INTERLEAVE_HIT: 24002
              NUMA_LOCAL: 63545992
              NUMA_OTHER: 0

  Determine (and verify) the page cache size:

    crash> kmem -c
    page_cache_size: 18431 (verified)

  Dump all pages in the page_hash_table:

    crash> kmem -C
    page_hash_table[0]
    c0325b40
    c03a0598
    c03b4070
    c0364c28
    c0357690
    c02ef338
    c02d7c60
    c02c11e0
    c02a3d70
    page_hash_table[1]
    c0394ce8
    c03c4218
    c03b4048
    c0364c00
    c0357668
    c02d6e50
    c02d7dc8
    c02c0cb8
    c02db630
    c02ebad0
    page_hash_table[2]
    c037e808
    c034e248
    c03b4020
    c02ec868
    c03baa60
    ...
    page_hash_table[2047]
    c033a798
    c0390b48
    c03b4098
    c0364890
    c03576b8
    c02d2c38
    c02d7c88
    c02de5d8
   
    page_cache_size: 18437 (verified)
   
  Find the page_hash_table entry containing page c03576b8:

    crash> kmem -c c03576b8
    page_hash_table[2047]
    c03576b8

  Display kmalloc() slab data:

    crash> kmem -s
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c02eadc0 kmem_cache               232         58        68      4     4k
    f79c2888 ip_vs_conn               128          0         0      0     4k
    f79c2970 tcp_tw_bucket             96          0         0      0     4k
    f79c2a58 tcp_bind_bucket           32         12       565      5     4k
    f79c2b40 tcp_open_request          64          0        59      1     4k
    f79c2c28 inet_peer_cache           64          1        59      1     4k
    f79c2d10 ip_fib_hash               32         11       339      3     4k
    f79c2df8 ip_dst_cache             160          8       120      5     4k
    f79c2ee0 arp_cache                128          1        30      1     4k
    c8402970 blkdev_requests           96      30208     37800    945     4k
    c8402a58 nfs_read_data            384          0         0      0     4k
    c8402b40 nfs_write_data           384          0         0      0     4k
    c8402c28 nfs_page                  96          0         0      0     4k
    c8402d10 dnotify cache             20          0         0      0     4k
    c8402df8 file lock cache           92          3       336      8     4k
    c8402ee0 fasync cache              16          0         0      0     4k
    c84027a0 uid_cache                 32          3       339      3     4k
    c84026b8 skbuff_head_cache        160        320       624     26     4k
    c84025d0 sock                     832         32       180     20     8k
    c84024e8 sigqueue                 132          0       203      7     4k
    c8402400 cdev_cache                64         19       472      8     4k
    c8402318 bdev_cache                64          8       236      4     4k
    c8402230 mnt_cache                 96         11       120      3     4k
    c8402148 inode_cache              480        817       848    106     4k
    c8402060 dentry_cache             128       1352      1470     49     4k
    c8403ee0 filp                      96        244       440     11     4k
    c8403df8 names_cache             4096          0        12     12     4k
    c8403d10 buffer_head               96      14936     16000    400     4k
    c8403c28 mm_struct                128         25       240      8     4k
    c8403b40 vm_area_struct            64        393      1298     22     4k
    c8403a58 fs_cache                  64         30       472      8     4k
    c8403970 files_cache              416         30       135     15     4k
    c8403888 signal_act              1312         32        99     33     4k
    c84037a0 size-131072(DMA)      131072          0         0      0   128k
    c84036b8 size-131072           131072          1         1      1   128k
    c84035d0 size-65536(DMA)        65536          0         0      0    64k
    c84034e8 size-65536             65536          0         0      0    64k
    c8403400 size-32768(DMA)        32768          0         0      0    32k
    c8403318 size-32768             32768          0         1      1    32k
    c8403230 size-16384(DMA)        16384          0         0      0    16k
    c8403148 size-16384             16384          0         0      0    16k
    c8403060 size-8192(DMA)          8192          0         0      0     8k
    c8401ee0 size-8192               8192          1         2      2     8k
    c8401df8 size-4096(DMA)          4096          0         0      0     4k
    c8401d10 size-4096               4096         30        30     30     4k
    c8401c28 size-2048(DMA)          2048          0         0      0     4k
    c8401b40 size-2048               2048         37       132     66     4k
    c8401a58 size-1024(DMA)          1024          0         0      0     4k
    c8401970 size-1024               1024        301       328     82     4k
    c8401888 size-512(DMA)            512          0         0      0     4k
    c84017a0 size-512                 512        141       168     21     4k
    c84016b8 size-256(DMA)            256          0         0      0     4k
    c84015d0 size-256                 256         80       435     29     4k
    c84014e8 size-128(DMA)            128          0         0      0     4k
    c8401400 size-128                 128        508       840     28     4k
    c8401318 size-64(DMA)              64          0         0      0     4k
    c8401230 size-64                   64        978      1357     23     4k
    c8401148 size-32(DMA)              32          0         0      0     4k
    c8401060 size-32                   32       1244      1808     16     4k

  Display all slab data in the "arp_cache" cache:

    crash> kmem -S arp_cache
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    f79c2ee0 arp_cache                128          1        30      1     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    f729d000  f729d0a0     30          1    29
    FREE / [ALLOCATED]
       f729d0a0  (cpu 7 cache)
       f729d120  (cpu 7 cache)
       f729d1a0  (cpu 7 cache)
       f729d220  (cpu 7 cache)
       f729d2a0  (cpu 7 cache)
       f729d320  (cpu 7 cache)
       f729d3a0  (cpu 7 cache)
       f729d420  (cpu 7 cache)
       f729d4a0  (cpu 7 cache)
       f729d520  (cpu 7 cache)
       f729d5a0  (cpu 7 cache)
       f729d620  (cpu 7 cache)
       f729d6a0  (cpu 7 cache)
       f729d720  (cpu 7 cache)
       f729d7a0  (cpu 7 cache)
       f729d820  (cpu 7 cache)
       f729d8a0  (cpu 7 cache)
       f729d920  (cpu 7 cache)
       f729d9a0  (cpu 7 cache)
       f729da20  (cpu 7 cache)
       f729daa0  (cpu 7 cache)
       f729db20  (cpu 7 cache)
       f729dba0  (cpu 7 cache)
       f729dc20  (cpu 7 cache)
       f729dca0  (cpu 7 cache)
       f729dd20  (cpu 7 cache)
       f729dda0  (cpu 7 cache)
       f729de20  (cpu 7 cache)
       f729dea0  (cpu 3 cache)
      [f729df20]

  Search the kmalloc() slab subsystem for address c3fbdb60:

    crash> kmem -s c3fbdb60
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c8402970 blkdev_requests           96      30208     37800    945     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    c3fbd020  c3fbd0e0     40         40     0
    FREE / [ALLOCATED]
      [c3fbdb60]

  Make a generic search (no flags) for the same address c3fbdb60:

    crash> kmem c3fbdb60
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c8402970 blkdev_requests           96      30208     37800    945     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    c3fbd020  c3fbd0e0     40         40     0
    FREE / [ALLOCATED]
      [c3fbdb60]

      PAGE     PHYSICAL   MAPPING    INDEX CNT FLAGS
    c410ee74    3fbd000         0         0  1 slab

  Display memory node data (if supported):

    crash> kmem -n
    NODE    SIZE    PGLIST_DATA   BOOTMEM_DATA   NODE_ZONES
      0    130933     c0332ee0      c0403a44      c0332ee0
                                                  c03333e0
                                                  c03338e0
    MEM_MAP   START_PADDR  START_MAPNR
    c1000030       0            0
   
    ZONE  NAME        SIZE   MEM_MAP  START_PADDR  START_MAPNR
      0   DMA         4096  c1000030            0            0
      1   Normal    126837  c1038030      1000000         4096
      2   HighMem        0         0            0            0

  Translate a page structure's flags field contents:

    crash> kmem -g 4080
    FLAGS: 4080
      PAGE-FLAG        BIT  VALUE
      PG_slab            7  0000080
      PG_head           14  0004000
    crash>



 18. ) Documentation for crash command list:


NAME
  list - linked list

SYNOPSIS
  list [[-o] offset] [-e end] [-s struct[.member[,member]]] [-H] start

DESCRIPTION
  This command dumps the contents of a linked list.  The entries in a linked
  list are typically data structures that are tied together in one of two
  formats:

  1. A starting address points to a data structure; that structure contains
     a member that is a pointer to the next structure, and so on.  The list
     typically ends when a "next" pointer value contains one of the
     following:

       a. a NULL pointer.
       b. a pointer to the start address.
       c. a pointer to the first item pointed to by the start address.
       d. a pointer to its containing structure.
 
  2. Most Linux lists are linked via embedded list_head structures contained
     within the data structures in the list.  The linked list is headed by an
     external LIST_HEAD, which is simply a list_head structure initialized to
     point to itself, signifying that the list is empty:

       struct list_head {
           struct list_head *next, *prev;
       };

       #define LIST_HEAD_INIT(name) { &(name), &(name) }
       #define LIST_HEAD(name) struct list_head name = LIST_HEAD_INIT(name)

     In the case of list_head-type lists, the "next" pointer is the address
     of the embedded list_head structure in the next structure, and not the
     address of the structure itself.  The list typically ends when the
     list_head's next pointer points back to the LIST_HEAD address.

  This command can handle both types of linked list; in both cases the list
  of addresses that are dumped are the addresses of the data structures
  themselves.

  The arguments are as follows:

  [-o] offset  The offset within the structure to the "next" pointer
               (default is 0).  If non-zero, the offset may be entered
               in either of two manners:

               1. In "structure.member" format; the "-o" is not necessary.
               2. A number of bytes; the "-o" is only necessary on processors
                  where the offset value could be misconstrued as a kernel
                  virtual address.

       -e end  If the list ends in a manner unlike the typical manners that
               are described above, an explicit ending address value may be
               entered.
    -s struct  For each address in list, format and print as this type of
               structure; use the "struct.member" format in order to display
               a particular member of the structure.  To display multiple
               members of a structure, use a comma-separated list of members.

  The meaning of the "start" argument, which can be expressed either
  symbolically or in hexadecimal format, depends upon whether the -H option
  is pre-pended or not:

      start  The address of the first structure in the list.
   -H start  The address of the list_head structure, typically expressed
             symbolically, but also can be an expression evaluating to the
             address of the starting list_head structure.

EXAMPLES
  Note that each task_struct is linked to its parent's task_struct via the
  p_pptr member:

    crash> struct task_struct.p_pptr
    struct task_struct {
       [136] struct task_struct *p_pptr;
    }

  That being the case, given a task_struct pointer of c169a000, show its
  parental hierarchy back to the "init_task" (the "swapper" task):

    crash> list task_struct.p_pptr c169a000
    c169a000
    c0440000
    c50d0000
    c0562000
    c0d28000
    c7894000
    c6a98000
    c009a000
    c0252000

  Given that the "task_struct.p_pptr" offset is 136 bytes, the same
  result could be accomplished like so:

    crash> list 136 c169a000
    c169a000
    c0440000
    c50d0000
    c0562000
    c0d28000
    c7894000
    c6a98000
    c009a000
    c0252000

  The list of currently-registered file system types are headed up by a
  struct file_system_type pointer named "file_systems", and linked by
  the "next" field in each file_system_type structure.  The following
  sequence displays the structure address followed by the name and
  fs_flags members of each registered file system type:

    crash> p file_systems
    file_systems = $1 = (struct file_system_type *) 0xc03adc90
    crash> list file_system_type.next -s file_system_type.name,fs_flags 0xc03adc90
    c03adc90
      name = 0xc02c05c8 "rootfs",
      fs_flags = 0x30,
    c03abf94
      name = 0xc02c0319 "bdev",
      fs_flags = 0x10,
    c03acb40
      name = 0xc02c07c4 "proc",
      fs_flags = 0x8,
    c03e9834
      name = 0xc02cfc83 "sockfs",
      fs_flags = 0x10,
    c03ab8e4
      name = 0xc02bf512 "tmpfs",
      fs_flags = 0x20,
    c03ab8c8
      name = 0xc02c3d6b "shm",
      fs_flags = 0x20,
    c03ac394
      name = 0xc02c03cf "pipefs",
      fs_flags = 0x10,
    c03ada74
      name = 0xc02c0e6b "ext2",
      fs_flags = 0x1,
    c03adc74
      name = 0xc02c0e70 "ramfs",
      fs_flags = 0x20,
    c03ade74
      name = 0xc02c0e76 "hugetlbfs",
      fs_flags = 0x20,
    c03adf8c
      name = 0xc02c0f84 "iso9660",
      fs_flags = 0x1,
    c03aec14
      name = 0xc02c0ffd "devpts",
      fs_flags = 0x8,
    c03e93f4
      name = 0xc02cf1b9 "pcihpfs",
      fs_flags = 0x28,
    e0831a14
      name = 0xe082f89f "ext3",
      fs_flags = 0x1,
    e0846af4
      name = 0xe0841ac6 "usbdevfs",
      fs_flags = 0x8,
    e0846b10
      name = 0xe0841acf "usbfs",
      fs_flags = 0x8,
    e0992370
      name = 0xe099176c "autofs",
      fs_flags = 0x0,
    e2dcc030
      name = 0xe2dc8849 "nfs",
      fs_flags = 0x48000,

  In some kernels, the system run queue is a linked list headed up by the
  "runqueue_head", which is defined like so:

    static LIST_HEAD(runqueue_head);

  The run queue linking is done with the "run_list" member of the task_struct:

    crash> struct task_struct.run_list
    struct task_struct {
        [60] struct list_head run_list;
    }

  Therefore, to view the list of task_struct addresses in the run queue,
  either of the following commands will work:

    crash> list task_struct.run_list -H runqueue_head
    f79ac000
    f7254000
    f7004000
    crash> list 60 -H runqueue_head
    f79ac000
    f7254000
    f7004000

  Lastly, in some kernel versions, the vfsmount structures of the mounted
  filesystems are linked by the LIST_HEAD "vfsmntlist", which uses the
  mnt_list list_head of each vfsmount structure in the list.  To dump each
  vfsmount structure in the list, append the -s option:

    crash> list -H vfsmntlist vfsmount.mnt_list -s vfsmount
    c3fc9e60
    struct vfsmount {
      mnt_hash = {
        next = 0xc3fc9e60,
        prev = 0xc3fc9e60
      },
      mnt_parent = 0xc3fc9e60,
      mnt_mountpoint = 0xc3fc5dc0,
      mnt_root = 0xc3fc5dc0,
      mnt_instances = {
        next = 0xc3f60a74,
        prev = 0xc3f60a74
      },
      mnt_sb = 0xc3f60a00,
      mnt_mounts = {
        next = 0xf7445e08,
        prev = 0xf7445f88
      },
      mnt_child = {
        next = 0xc3fc9e88,
        prev = 0xc3fc9e88
      },
      mnt_count = {
        counter = 209
      },
      mnt_flags = 0,
      mnt_devname = 0xc8465b20 "/dev/root",
      mnt_list = {
        next = 0xf7445f9c,
        prev = 0xc02eb828
      },
      mnt_owner = 0
    }
    f7445f60
    struct vfsmount {
    ...



 19. ) Documentation for crash command log:


NAME
  log - dump system message buffer

SYNOPSIS
  log [-m]

DESCRIPTION
  This command dumps the kernel log_buf contents in chronological order.
 
    -m  Display the message log level preceding each message.

EXAMPLES
  Dump the kernel message buffer:

    crash> log
    Linux version 2.2.5-15smp (root@mclinux1) (gcc version egcs-2.91.66 19990
    314/Linux (egcs-1.1.2 release)) #1 SMP Thu Aug 26 11:04:37 EDT 1999
    Intel MultiProcessor Specification v1.4
        Virtual Wire compatibility mode.
    OEM ID: DELL     Product ID: WS 410       APIC at: 0xFEE00000
    Processor #0 Pentium(tm) Pro APIC version 17
    Processor #1 Pentium(tm) Pro APIC version 17
    I/O APIC #2 Version 17 at 0xFEC00000.
    Processors: 2
    mapped APIC to ffffe000 (fee00000)
    mapped IOAPIC to ffffd000 (fec00000)
    Detected 447696347 Hz processor.
    Console: colour VGA+ 80x25
    Calibrating delay loop... 445.64 BogoMIPS
    ...
      8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
      MII transceiver found at address 24, status 782d.
      Enabling bus-master transmits and whole-frame receives.
    Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    nfsd_init: initialized fhcache, entries=256
    ...

  Do the same thing, but also show the log level preceding each message:

    crash> log -m
    <4>Linux version 2.2.5-15smp (root@mclinux1) (gcc version egcs-2.91.66 19990
    314/Linux (egcs-1.1.2 release)) #1 SMP Thu Aug 26 11:04:37 EDT 1999
    <4>Intel MultiProcessor Specification v1.4
    <4>    Virtual Wire compatibility mode.
    <4>OEM ID: DELL     Product ID: WS 410       APIC at: 0xFEE00000
    <4>Processor #0 Pentium(tm) Pro APIC version 17
    <4>Processor #1 Pentium(tm) Pro APIC version 17
    <4>I/O APIC #2 Version 17 at 0xFEC00000.
    <4>Processors: 2
    <4>mapped APIC to ffffe000 (fee00000)
    <4>mapped IOAPIC to ffffd000 (fec00000)
    <4>Detected 447696347 Hz processor.
    <4>Console: colour VGA+ 80x25
    <4>Calibrating delay loop... 445.64 BogoMIPS
    ...
    <6>  8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
    <6>  MII transceiver found at address 24, status 782d.
    <6>  Enabling bus-master transmits and whole-frame receives.
    <6>Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    <7>nfsd_init: initialized fhcache, entries=256
    ...



 20. ) Documentation for crash command mach:


NAME
  mach - machine specific data

SYNOPSIS
  mach [-cm]

DESCRIPTION
  This command displays data specific to a machine type.

    -c  Display each cpu's cpuinfo structure (x86, x86_64 and ia64 only).
        Display each cpu's x8664_pda structure (x86_64 only),
        Display the hwrpb_struct, and each cpu's percpu_struct (alpha only).
    -m  Display the physical memory map (x86, x86_64 and ia64 only).

EXAMPLES
    crash> mach
           MACHINE TYPE: i686
            MEMORY SIZE: 512 MB
                   CPUS: 2
        PROCESSOR SPEED: 1993 Mhz
                     HZ: 100
              PAGE SIZE: 4096
    KERNEL VIRTUAL BASE: c0000000
    KERNEL VMALLOC BASE: e0800000
      KERNEL STACK SIZE: 8192

  Display the system physical memory map:

    crash> mach -m
          PHYSICAL ADDRESS RANGE         TYPE
    0000000000000000 - 00000000000a0000  E820_RAM
    00000000000f0000 - 0000000000100000  E820_RESERVED
    0000000000100000 - 000000001ff75000  E820_RAM
    000000001ff75000 - 000000001ff77000  E820_NVS
    000000001ff77000 - 000000001ff98000  E820_ACPI
    000000001ff98000 - 0000000020000000  E820_RESERVED
    00000000fec00000 - 00000000fec90000  E820_RESERVED
    00000000fee00000 - 00000000fee10000  E820_RESERVED
    00000000ffb00000 - 0000000100000000  E820_RESERVED



 21. ) Documentation for crash command mod:


NAME
  mod - module information and loading of symbols and debugging data

SYNOPSIS
  mod -s module [objfile] | -d module | -S [directory] | -D | -r | -R | -o | -g

DESCRIPTION
  With no arguments, this command displays basic information of the currently
  installed modules, consisting of the module address, name, size, the
  object file name (if known), and whether the module was compiled with
  CONFIG_KALLSYMS.

  The arguments are concerned with with the loading or deleting of symbolic
  and debugging data from a module's object file.  A modules's object file
  always contains symbolic data (symbol names and addresses), but contains
  debugging data only if the module was compiled with the -g CFLAG.  In
  addition, the module may have compiled with CONFIG_KALLSYMS, which means
  that the module's symbolic data will have been loaded into the kernel's
  address space when it was installed.  If the module was not compiled with
  CONFIG_KALLSYMS, then only the module's exported symbols will be loaded
  into the kernel's address space.  Therefore, for the purpose of this
  command, it should noted that a kernel module may have been compiled in
  one of following manners:

  1. If the module was built without CONFIG_KALLSYMS and without the -g CFLAG,
     then the loading of the module's additional non-exported symbols can
     be accomplished with this command.
  2. If the module was built with CONFIG_KALLSYMS, but without the -g CFLAG,
     then there is no benefit in loading the symbols from the module object
     file, because all of the module's symbols will have been loaded into the
     kernel's address space when it was installed.
  3. If the module was built with CONFIG_KALLSYMS and with the the -g CFLAG,
     then the loading of the module's debugging data can be accomplished
     with this command.
  4. If the module was built without CONFIG_KALLSYMS but with the -g CFLAG,
     then the loading of the both module's symbolic and debugging data can
     be accomplished with this command.

  -s module [objfile]  Loads symbolic and debugging data from the object file
                       for the module specified.  If no objfile argument is
                       appended, a search will be made for an object file
                       consisting of the module name with a .o or .ko suffix,
                       starting at the /lib/modules/ directory on
                       the host system.  If an objfile argument is appended,
                       then that file will be used.
            -d module  Deletes the symbolic and debugging data of the module
                       specified.
       -S [directory]  Load symbolic and debugging data from the object file
                       for all loaded modules.  For each module, a search
                       will be made for an object file consisting of the
                       module name with a .o or.ko suffix, starting at the
                       /lib/modules/ directory of the host system.
                       If a directory argument is appended, then the search
                       will be restricted to that directory.
                   -D  Deletes the symbolic and debugging data of all modules.
                   -r  Passes the -readnow flag to the embedded gdb module,
                       which will override the two-stage strategy that it uses
                       for reading symbol tables from module object files.
                   -R  Reinitialize module data. All currently-loaded symbolic
                       and debugging data will be deleted, and the installed
                       module list will be updated (live system only).
                   -g  When used with -s or -S, add a module object's section
                       start and end addresses to its symbol list.
                   -o  Load module symbols with old mechanism.

  After symbolic and debugging data have been loaded, backtraces and text
  disassembly will be displayed appropriately.  Depending upon the processor
  architecture, data may also printed symbolically with the "p" command;
  at a minimum, the "rd" command may be used with module data symbols.

  If crash can recognize that the set of modules has changed while running a
  session on a live kernel, the module data will be reinitialized the next
  time this command is run; the -r option forces the reinitialization.

EXAMPLES
  Display the currently-installed modules:

    crash> mod
     MODULE   NAME         SIZE  OBJECT FILE
    c8019000  soundcore    2788  (not loaded)
    c801b000  soundlow      336  (not loaded)
    c801d000  sound       59864  (not loaded)
    c802d000  ad1848      15728  (not loaded)
    c8032000  uart401      6000  (not loaded)
    c8035000  cs4232       2472  (not loaded)
    c8043000  opl3        11048  (not loaded)
    c8047000  3c59x       18152  (not loaded)
    c804d000  sunrpc      53796  (not loaded)
    c805c000  lockd       31528  (not loaded)
    c8065000  nfsd       151896  (not loaded)
    c8092000  nfs         29752  (not loaded)

  Display the currently-installed modules on a system where all modules were
  compiled with CONFIG_KALLSYMS:

    crash> mod
     MODULE   NAME              SIZE  OBJECT FILE
    e080d000  jbd              57016  (not loaded)  [CONFIG_KALLSYMS]
    e081e000  ext3             92360  (not loaded)  [CONFIG_KALLSYMS]
    e0838000  usbcore          83168  (not loaded)  [CONFIG_KALLSYMS]
    e0850000  usb-uhci         27532  (not loaded)  [CONFIG_KALLSYMS]
    e085a000  ehci-hcd         20904  (not loaded)  [CONFIG_KALLSYMS]
    e0865000  input             6208  (not loaded)  [CONFIG_KALLSYMS]
    e086a000  hid              22404  (not loaded)  [CONFIG_KALLSYMS]
    e0873000  mousedev          5688  (not loaded)  [CONFIG_KALLSYMS]
    e0878000  keybdev           2976  (not loaded)  [CONFIG_KALLSYMS]
    e08fd000  cdrom            34144  (not loaded)  [CONFIG_KALLSYMS]
    e0909000  ide-cd           35776  (not loaded)  [CONFIG_KALLSYMS]
    e0915000  scsi_mod        117928  (not loaded)  [CONFIG_KALLSYMS]
    e0935000  ide-scsi         12752  (not loaded)  [CONFIG_KALLSYMS]
    e093c000  microcode         5248  (not loaded)  [CONFIG_KALLSYMS]
    e0943000  sr_mod           18136  (not loaded)  [CONFIG_KALLSYMS]
    e0956000  floppy           59056  (not loaded)  [CONFIG_KALLSYMS]
    e0966000  sg               38060  (not loaded)  [CONFIG_KALLSYMS]
    e0971000  ip_tables        16544  (not loaded)  [CONFIG_KALLSYMS]
    e097d000  iptable_filter    2412  (not loaded)  [CONFIG_KALLSYMS]
    e097f000  e1000            76096  (not loaded)  [CONFIG_KALLSYMS]
    e09ba000  autofs           13780  (not loaded)  [CONFIG_KALLSYMS]
    e09c1000  parport          39072  (not loaded)  [CONFIG_KALLSYMS]
    e09ce000  lp                9220  (not loaded)  [CONFIG_KALLSYMS]
    e09d4000  parport_pc       19204  (not loaded)  [CONFIG_KALLSYMS]
    e09e2000  agpgart          59128  (not loaded)  [CONFIG_KALLSYMS]
    e0a1a000  radeon          117156  (not loaded)  [CONFIG_KALLSYMS]
    e2dc7000  sunrpc           91996  (not loaded)  [CONFIG_KALLSYMS]
    e2de1000  lockd            60624  (not loaded)  [CONFIG_KALLSYMS]
    e2df3000  nfs              96880  (not loaded)  [CONFIG_KALLSYMS]

  Load the symbolic and debugging data of all modules:

    crash> mod -S
     MODULE   NAME         SIZE  OBJECT FILE
    c8019000  soundcore    2788  /lib/modules/2.2.5-15/misc/soundcore.o
    c801b000  soundlow      336  /lib/modules/2.2.5-15/misc/soundlow.o
    c801d000  sound       59864  /lib/modules/2.2.5-15/misc/sound.o
    c802d000  ad1848      15728  /lib/modules/2.2.5-15/misc/ad1848.o
    c8032000  uart401      6000  /lib/modules/2.2.5-15/misc/uart401.o
    c8035000  cs4232       2472  /lib/modules/2.2.5-15/misc/cs4232.o
    c8043000  opl3        11048  /lib/modules/2.2.5-15/misc/opl3.o
    c8047000  3c59x       18152  /lib/modules/2.2.5-15/net/3c59x.o
    c804d000  sunrpc      53796  /lib/modules/2.2.5-15/misc/sunrpc.o
    c805c000  lockd       31528  /lib/modules/2.2.5-15/fs/lockd.o
    c8065000  nfsd       151896  /lib/modules/2.2.5-15/fs/nfsd.o
    c8092000  nfs         29752  /lib/modules/2.2.5-15/fs/nfs.o
   
  Load the symbolic and debugging data of the soundcore module from its
  known location:

    crash> mod -s soundcore
     MODULE   NAME         SIZE  OBJECT FILE
    c8019000  soundcore    2788  /lib/modules/2.2.5-15/misc/soundcore.o
   
  Delete the current symbolic and debugging data of the soundcore module,
  and then re-load it from a specified object file:

    crash> mod -d soundcore
    crash> mod -s soundcore /tmp/soundcore.o
     MODULE   NAME         SIZE  OBJECT FILE
    c8019000  soundcore    2788  /tmp/soundcore.o

  After installing a new kernel module on a live system, reinitialize the
  installed module list:

    crash> !insmod mdacon
    crash> mod
    mod: NOTE: modules have changed on this system -- reinitializing
     MODULE   NAME         SIZE  OBJECT FILE
    c8019000  soundcore    2788  (not loaded)
    c801b000  soundlow      336  (not loaded)
    c801d000  sound       59864  (not loaded)
    c802d000  ad1848      15728  (not loaded)
    c8032000  uart401      6000  (not loaded)
    c8035000  cs4232       2472  (not loaded)
    c8043000  opl3        11048  (not loaded)
    c8047000  3c59x       18152  (not loaded)
    c804d000  sunrpc      53796  (not loaded)
    c805c000  lockd       31528  (not loaded)
    c8065000  nfs         29752  (not loaded)
    c806e000  autofs       9316  (not loaded)
    c8072000  nfsd       151896  (not loaded)
    c80a1000  mdacon       3556  (not loaded)



 22. ) Documentation for crash command mount:


NAME
  mount - mounted filesystem data

SYNOPSIS
  mount [-f] [-i] [-n pid|task] [vfsmount|superblock|devname|dirname|inode]

DESCRIPTION
  This command displays basic information about the currently-mounted
  filesystems.  The per-filesystem dirty inode list or list of open
  files for the filesystem may also be displayed.

     -f  dump dentries and inodes for open files in each filesystem.
     -i  dump all dirty inodes associated with each filesystem; only
         supported on kernels with super_block.s_dirty linked list.

  For kernels supporting namespaces, the -n option may be used to
  display the mounted filesystems with respect to the namespace of a
  specified task:

     -n pid   a process PID.
     -n task  a hexadecimal task_struct pointer.

  Specific filesystems may be selected using the following forms:

    vfsmount  hexadecimal address of filesystem vfsmount structure.
  superblock  hexadecimal address of filesystem super_block structure.
     devname  device name of filesystem.
     dirname  directory where filesystem is mounted.
       inode  hexadecimal address of an open inode of a filesystem.

EXAMPLES
  Display mounted filesystem data:

    crash> mount
    VFSMOUNT SUPERBLK TYPE   DEVNAME         DIRNAME
    c0089ea0 c0088a00 ext2   /dev/root       /   
    c0089cf0 c0088c00 proc   /proc           /proc
    c0089e10 c0088800 ext2   /dev/sda5       /boot
    c0089d80 c0088600 ext2   /dev/sda6       /usr
    c0089f30 c0088400 devpts none            /dev/pts
    c3f4b010 c0088200 ext2   /dev/sda1       /home
    c6bf3d10 c0088000 nfs    home:/home1     /home1
    c49b90a0 c43a2a00 nfs    home:/usr/local /usr/local

  Display the open files associated with each mounted filesystem:

    crash> mount -f
    VFSMOUNT SUPERBLK TYPE   DEVNAME         DIRNAME
    c7fb2b80 c7fb3200 ext2   /dev/root       /
    OPEN FILES:
     DENTRY    INODE    TYPE  PATH
    c6d02200  c6d0f7a0  REG   usr/X11R6/lib/libX11.so.6.1
    c6d02100  c6d0f9e0  REG   usr/X11R6/lib/libXext.so.6.3
    c6d02000  c6d0fc20  REG   usr/X11R6/lib/libICE.so.6.3
    c6d02680  c6d0f320  REG   usr/X11R6/bin/xfs
    c7106580  c70c5440  CHR   dev/psaux
    ...

  Display the dirty inodes associated with each mounted filesystem:

    crash> mount -i
    VFSMOUNT SUPERBLK TYPE   DEVNAME         DIRNAME
    c0089ea0 c0088a00 ext2   /dev/root       /           
    DIRTY INODES
    c7ad4008
    c2233438
    c72c4008
    c7d6b548
    c3af1a98
    c7d6b768
    c3c4e228
    ...

  Display the mounted filesystem containing inode c5000aa8:

    crash> mount c5000aa8
    VFSMOUNT SUPERBLK TYPE   DEVNAME         DIRNAME
    c0089f30 c0088600 ext2   /dev/sda6       /usr




 23. ) Documentation for crash command net:


NAME
  net - network command

SYNOPSIS
  net [-a] [[-s | -S] [-R ref] [pid | taskp]] [-n addr]

DESCRIPTION
  Display various network related data:

      -a  display the ARP cache.
      -s  display open network socket/sock addresses, their family and type,
          and for INET and INET6 families, their source and destination
          addresses and ports.
      -S  displays open network socket/sock addresses followed by a dump
          of both structures.
  -n addr translates an IPv4 address expressed as a decimal or hexadecimal
          value into a standard numbers-and-dots notation.
  -R ref  socket or sock address, or file descriptor.
     pid  a process PID.
   taskp  a hexadecimal task_struct pointer.

  If no arguments are entered, the list of network devices, names and IP
  addresses are displayed.  The -R option, typically invoked from "foreach net",
  and in conjunction with the -s or -S options, searches for references to a
  socket address, sock address, or a file descriptor; if found, only the
  referenced fd/socket/sock data will be displayed.

EXAMPLES
  Display the network device list:

    crash> net
     DEVICE   NAME   IP ADDRESS(ES)
    c0249f20  lo     127.0.0.1
    c7fe6d80  eth0   10.1.8.20

  Dump the ARP cache:

    crash> net -a
    IP ADDRESS      HW TYPE    HW ADDRESS         DEVICE  STATE
    0.0.0.0         UNKNOWN    00 00 00 00 00 00  lo      40 (NOARP)
    192.168.1.1     ETHER      00:50:54:fe:ef:23  eth0    04 (STALE)
    192.168.1.10    ETHER      00:90:27:9c:6c:79  eth0    02 (REACHABLE)
    192.168.1.118   ETHER      00:c0:4f:60:00:e2  eth0    02 (REACHABLE)
  
  Display the sockets for PID 2517, using both -s and -S output formats:

    crash> net -s 2517
    PID: 2517   TASK: c1598000  CPU: 1   COMMAND: "rlogin"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     3  c57375dc  c1ff1850  INET:STREAM      10.1.8.20-1023      10.1.16.62-513
   
    crash> net -S 2517
    PID: 2517   TASK: c1598000  CPU: 1   COMMAND: "rlogin"
    FD   SOCKET     SOCK
     3  c57375dc  c1ff1850
   
    struct socket {
      state = SS_CONNECTED,
      flags = 131072,
      ops = 0xc023f820,
      inode = 0xc5737540,
      fasync_list = 0x0,
      file = 0xc58892b0,
      sk = 0xc1ff1850,
      wait = 0xc14d9ed4,
      type = 1,
      passcred = 0 '\000',
      tli = 0 '\000'
    }
    struct sock {
      sklist_next = 0xc1ff12f0,
      sklist_prev = 0xc216bc00,
      bind_next = 0x0,
      bind_pprev = 0xc0918448,
      daddr = 1041236234,
      rcv_saddr = 336068874,
      dport = 258,
      num = 1023,
      bound_dev_if = 0,
      next = 0x0,
      pprev = 0xc0286dd4,
      state = 1 '\001',
      zapped = 0 '\000',
      sport = 65283,
      family = 2,
      reuse = 0 '\000',
      ...
   Translate the rcv_saddr from above into dotted-decimal notation:

    crash> net -n 1041236234
    10.1.16.62

  From "foreach", find all tasks with references to socket c08ea3cc:

    crash> foreach net -s -R c08ea3cc
    PID: 2184   TASK: c7026000  CPU: 1   COMMAND: "klines.kss"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2200   TASK: c670a000  CPU: 1   COMMAND: "kpanel"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2201   TASK: c648a000  CPU: 1   COMMAND: "kbgndwm"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 19294  TASK: c250a000  CPU: 0   COMMAND: "prefdm"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2194   TASK: c62dc000  CPU: 1   COMMAND: "kaudioserver"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2195   TASK: c6684000  CPU: 1   COMMAND: "maudio"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2196   TASK: c6b58000  CPU: 1   COMMAND: "kwmsound"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2197   TASK: c6696000  CPU: 0   COMMAND: "kfm"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2199   TASK: c65ec000  CPU: 0   COMMAND: "krootwm"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 694    TASK: c1942000  CPU: 0   COMMAND: "prefdm"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 698    TASK: c6a2c000  CPU: 1   COMMAND: "X"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   
    PID: 2159   TASK: c4a5a000  CPU: 1   COMMAND: "kwm"
    FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
     5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0
   



 24. ) Documentation for crash command p:


NAME
  p - print the value of an expression

SYNOPSIS
  p [-x|-d][-u] expression

DESCRIPTION
  This command passes its arguments on to gdb "print" command for evaluation.

    expression   The expression to be evaluated.
            -x  override default output format with hexadecimal format.
            -d  override default output format with decimal format.
            -u  the expression evaluates to a user address reference.

  The default output format is decimal, but that can be changed at any time
  with the two built-in aliases "hex" and "dec".  Alternatively, there
  are two other built-in aliases, "px" and "pd", which force the command
  output to be displayed in hexadecimal or decimal, without changing the
  default mode.

EXAMPLES
  Print the contents of jiffies:

    crash> p jiffies
    jiffies = $6 = 166532620
    crash> px jiffies
    jiffies = $7 = 0x9ed174b
    crash> pd jiffies
    jiffies = $8 = 166533160

  Print the contents of the vm_area_struct "init_mm":

    crash> p init_mm
    init_mm = $5 = {
      mmap = 0xc022d540,
      mmap_avl = 0x0,
      mmap_cache = 0x0,
      pgd = 0xc0101000,
      count = {
        counter = 0x6
      },
      map_count = 0x1,
      mmap_sem = {
        count = {
          counter = 0x1
        },
        waking = 0x0,
        wait = 0x0
      },
      context = 0x0,
      start_code = 0xc0000000,
      end_code = 0xc022b4c8,
      start_data = 0x0,
      end_data = 0xc0250388,
      start_brk = 0x0,
      brk = 0xc02928d8,
      start_stack = 0x0,
      arg_start = 0x0,
      arg_end = 0x0,
      env_start = 0x0,
      env_end = 0x0,
      rss = 0x0,
      total_vm = 0x0,
      locked_vm = 0x0,
      def_flags = 0x0,
      cpu_vm_mask = 0x0,
      swap_cnt = 0x0,
      swap_address = 0x0,
      segments = 0x0
    }



 25. ) Documentation for crash command ps:


NAME
  ps - display process status information

SYNOPSIS
  ps [-k|-u|-G][-s][-p|-c|-t|-l|-a|-g|-r] [pid | taskp | command] ...

DESCRIPTION
  This command displays process status for selected, or all, processes
  in the system.  If no arguments are entered, the process data is
  is displayed for all processes.  Specific processes may be selected
  by using the following identifier formats:

       pid  a process PID.
     taskp  a hexadecimal task_struct pointer.
   command  a command name.  If a command name is made up of letters that
            are all numerical values, precede the name string with a "\".

  The process list may be further restricted by the following options:

        -k  restrict the output to only kernel threads.
        -u  restrict the output to only user tasks.
        -G  display only the thread group leader in a thread group.

  The process identifier types may be mixed.  For each task, the following
  items are displayed:

    1. the process PID.
    2. the parent process PID.
    3. the CPU number that the task ran on last.
    4. the task_struct address or the kernel stack pointer of the process.
       (see -s option below)
    5. the task state (RU, IN, UN, ZO, ST, DE, SW).
    6. the percentage of physical memory being used by this task.
    7. the virtual address size of this task in kilobytes.
    8. the resident set size of this task in kilobytes.
    9. the command name.

  The default output shows the task_struct address of each process under a
  column titled "TASK".  This can be changed to show the kernel stack
  pointer under a column titled "KSTACKP".

       -s  replace the TASK column with the KSTACKP column.

  On SMP machines, the active task on each CPU will be highlighted by an
  angle bracket (">") preceding its information.

  Alternatively, information regarding parent-child relationships,
  per-task time usage data, argument/environment data, thread groups,
  or resource limits may be displayed:

       -p  display the parental hierarchy of selected, or all, tasks.
       -c  display the children of selected, or all, tasks.
       -t  display the task run time, start time, and cumulative user
           and system times.
       -l  display the task last_run or timestamp value, whichever applies,
           of selected, or all, tasks; the list is sorted with the most
           recently-run task (largest last_run/timestamp) shown first.
       -a  display the command line arguments and environment strings of
           selected, or all, user-mode tasks.
       -g  display tasks by thread group, of selected, or all, tasks.
       -r  display resource limits (rlimits) of selected, or all, tasks.

EXAMPLES
  Show the process status of all current tasks:

    crash> ps
       PID    PPID  CPU   TASK    ST  %MEM   VSZ   RSS  COMM
    >     0      0   3  c024c000  RU   0.0     0     0  [swapper]
    >     0      0   0  c0dce000  RU   0.0     0     0  [swapper]
          0      0   1  c0fa8000  RU   0.0     0     0  [swapper]
    >     0      0   2  c009a000  RU   0.0     0     0  [swapper]
          1      0   1  c0098000  IN   0.0  1096   476  init
          2      1   1  c0090000  IN   0.0     0     0  [kflushd]
          3      1   1  c000e000  IN   0.0     0     0  [kpiod]
          4      1   3  c000c000  IN   0.0     0     0  [kswapd]
          5      1   1  c0008000  IN   0.0     0     0  [mdrecoveryd]
        253      1   2  fbc4c000  IN   0.0  1088   376  portmap
        268      1   2  fbc82000  IN   0.1  1232   504  ypbind
        274    268   2  fa984000  IN   0.1  1260   556  ypbind
        321      1   1  fabf6000  IN   0.1  1264   608  syslogd
        332      1   1  fa9be000  RU   0.1  1364   736  klogd
        346      1   2  fae88000  IN   0.0  1112   472  atd
        360      1   2  faeb2000  IN   0.1  1284   592  crond
        378      1   2  fafd6000  IN   0.1  1236   560  inetd
        392      1   0  fb710000  IN   0.1  2264  1468  named
        406      1   3  fb768000  IN   0.1  1284   560  lpd
        423      1   1  fb8ac000  IN   0.1  1128   528  rpc.statd
        434      1   2  fb75a000  IN   0.0  1072   376  rpc.rquotad
        445      1   2  fb4a4000  IN   0.0  1132   456  rpc.mountd
        460      1   1  fa938000  IN   0.0     0     0  [nfsd]
        461      1   1  faa86000  IN   0.0     0     0  [nfsd]
        462      1   0  fac48000  IN   0.0     0     0  [nfsd]
        463      1   0  fb4ca000  IN   0.0     0     0  [nfsd]
        464      1   0  fb4c8000  IN   0.0     0     0  [nfsd]
        465      1   2  fba6e000  IN   0.0     0     0  [nfsd]
        466      1   1  fba6c000  IN   0.0     0     0  [nfsd]
        467      1   2  fac04000  IN   0.0     0     0  [nfsd]
        468    461   2  fa93a000  IN   0.0     0     0  [lockd]
        469    468   2  fa93e000  IN   0.0     0     0  [rpciod]
        486      1   0  fab54000  IN   0.1  1596   880  amd
        523      1   2  fa84e000  IN   0.1  1884  1128  sendmail
        538      1   0  fa82c000  IN   0.0  1112   416  gpm
        552      1   3  fa70a000  IN   0.1  2384  1220  httpd
        556    552   3  fa776000  IN   0.1  2572  1352  httpd
        557    552   2  faba4000  IN   0.1  2572  1352  httpd
        558    552   1  fa802000  IN   0.1  2572  1352  httpd
        559    552   3  fa6ee000  IN   0.1  2572  1352  httpd
        560    552   3  fa700000  IN   0.1  2572  1352  httpd
        561    552   0  fa6f0000  IN   0.1  2572  1352  httpd
        562    552   3  fa6ea000  IN   0.1  2572  1352  httpd
        563    552   0  fa67c000  IN   0.1  2572  1352  httpd
        564    552   3  fa674000  IN   0.1  2572  1352  httpd
        565    552   3  fa66a000  IN   0.1  2572  1352  httpd
        582      1   2  fa402000  IN   0.2  2968  1916  xfs
        633      1   2  fa1ec000  IN   0.2  5512  2248  innd
        636      1   3  fa088000  IN   0.1  2536   804  actived
        676      1   0  fa840000  IN   0.0  1060   384  mingetty
        677      1   1  fa590000  IN   0.0  1060   384  mingetty
        678      1   2  fa3b8000  IN   0.0  1060   384  mingetty
        679      1   0  fa5b8000  IN   0.0  1060   384  mingetty
        680      1   1  fa3a4000  IN   0.0  1060   384  mingetty
        681      1   2  fa30a000  IN   0.0  1060   384  mingetty
        683      1   3  fa5d8000  IN   0.0  1052   280  update
        686    378   1  fa3aa000  IN   0.1  2320  1136  in.rlogind
        687    686   2  f9e52000  IN   0.1  2136  1000  login
        688    687   0  f9dec000  IN   0.1  1732   976  bash
    >   700    688   1  f9d62000  RU   0.0  1048   256  gen12

  Display the parental hierarchy of the "crash" process on a live system:

    crash> ps -p 4249
    PID: 0      TASK: c0252000  CPU: 0   COMMAND: "swapper"
     PID: 1      TASK: c009a000  CPU: 1   COMMAND: "init"
      PID: 632    TASK: c73b6000  CPU: 1   COMMAND: "prefdm"
       PID: 637    TASK: c5a4a000  CPU: 1   COMMAND: "prefdm"
        PID: 649    TASK: c179a000  CPU: 0   COMMAND: "kwm"
         PID: 683    TASK: c1164000  CPU: 0   COMMAND: "kfm"
          PID: 1186   TASK: c165a000  CPU: 0   COMMAND: "xterm"
           PID: 1188   TASK: c705e000  CPU: 1   COMMAND: "bash"
            PID: 4249   TASK: c6b9a000  CPU: 0   COMMAND: "crash"

  Display all children of the "kwm" window manager:

    crash> ps -c kwm
      PID: 649    TASK: c179a000  CPU: 0   COMMAND: "kwm"
      PID: 682    TASK: c2d58000  CPU: 1   COMMAND: "kwmsound"
      PID: 683    TASK: c1164000  CPU: 1   COMMAND: "kfm"
      PID: 685    TASK: c053c000  CPU: 0   COMMAND: "krootwm"
      PID: 686    TASK: c13fa000  CPU: 0   COMMAND: "kpanel"
      PID: 687    TASK: c13f0000  CPU: 1   COMMAND: "kbgndwm"

  Display all threads in a firefox session:

    crash> ps firefox
       PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
      21273  21256   6  ffff81003ec15080  IN  46.3 1138276 484364  firefox
      21276  21256   6  ffff81003f49e7e0  IN  46.3 1138276 484364  firefox
      21280  21256   0  ffff81003ec1d7e0  IN  46.3 1138276 484364  firefox
      21286  21256   6  ffff81000b0d1820  IN  46.3 1138276 484364  firefox
      21287  21256   2  ffff81000b0d10c0  IN  46.3 1138276 484364  firefox
      26975  21256   5  ffff81003b5c1820  IN  46.3 1138276 484364  firefox
      26976  21256   5  ffff810023232820  IN  46.3 1138276 484364  firefox
      26977  21256   4  ffff810021a11820  IN  46.3 1138276 484364  firefox
      26978  21256   5  ffff810003159040  IN  46.3 1138276 484364  firefox
      26979  21256   5  ffff81003a058820  IN  46.3 1138276 484364  firefox

  Display only the thread group leader in the firefox session:

    crash> ps -G firefox
       PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
      21273  21256   0  ffff81003ec15080  IN  46.3 1138276 484364  firefox

  Show the time usage data for pid 10318:

    crash> ps -t 10318
    PID: 10318  TASK: f7b85550  CPU: 5   COMMAND: "bash"
        RUN TIME: 1 days, 01:35:32
      START TIME: 5209
           UTIME: 95
           STIME: 57

  Show the process status of PID 1, task f9dec000, and all nfsd tasks:

    crash> ps 1 f9dec000 nfsd
       PID    PPID  CPU   TASK    ST  %MEM   VSZ   RSS  COMM
          1      0   1  c0098000  IN   0.0  1096   476  init
        688    687   0  f9dec000  IN   0.1  1732   976  bash
        460      1   1  fa938000  IN   0.0     0     0  [nfsd]
        461      1   1  faa86000  IN   0.0     0     0  [nfsd]
        462      1   0  fac48000  IN   0.0     0     0  [nfsd]
        463      1   0  fb4ca000  IN   0.0     0     0  [nfsd]
        464      1   0  fb4c8000  IN   0.0     0     0  [nfsd]
        465      1   2  fba6e000  IN   0.0     0     0  [nfsd]
        466      1   1  fba6c000  IN   0.0     0     0  [nfsd]
        467      1   2  fac04000  IN   0.0     0     0  [nfsd]

  Show all kernel threads:

    crash> ps -k
       PID    PPID  CPU   TASK    ST  %MEM   VSZ   RSS  COMM
          0      0   1  c0fac000  RU   0.0     0     0  [swapper]
          0      0   0  c0252000  RU   0.0     0     0  [swapper]
          2      1   1  c0fa0000  IN   0.0     0     0  [kflushd]
          3      1   1  c03de000  IN   0.0     0     0  [kpiod]
          4      1   1  c03dc000  IN   0.0     0     0  [kswapd]
          5      1   0  c0092000  IN   0.0     0     0  [mdrecoveryd]
        336      1   0  c4a9a000  IN   0.0     0     0  [rpciod]
        337      1   0  c4830000  IN   0.0     0     0  [lockd]
        487      1   1  c4ba6000  IN   0.0     0     0  [nfsd]
        488      1   0  c18c6000  IN   0.0     0     0  [nfsd]
        489      1   0  c0cac000  IN   0.0     0     0  [nfsd]
        490      1   0  c056a000  IN   0.0     0     0  [nfsd]
        491      1   0  c0860000  IN   0.0     0     0  [nfsd]
        492      1   1  c0254000  IN   0.0     0     0  [nfsd]
        493      1   0  c0a86000  IN   0.0     0     0  [nfsd]
        494      1   0  c0968000  IN   0.0     0     0  [nfsd]

  Show all tasks sorted by their task_struct's last_run or timestamp value,
  whichever applies:

    crash> ps -l
    [280195]  PID: 2      TASK: c1468000  CPU: 0   COMMAND: "keventd"
    [280195]  PID: 1986   TASK: c5af4000  CPU: 0   COMMAND: "sshd"
    [280195]  PID: 2039   TASK: c58e6000  CPU: 0   COMMAND: "sshd"
    [280195]  PID: 2044   TASK: c5554000  CPU: 0   COMMAND: "bash"
    [280195]  PID: 2289   TASK: c70c0000  CPU: 0   COMMAND: "s"
    [280190]  PID: 1621   TASK: c54f8000  CPU: 0   COMMAND: "cupsd"
    [280184]  PID: 5      TASK: c154c000  CPU: 0   COMMAND: "kswapd"
    [280184]  PID: 6      TASK: c7ff6000  CPU: 0   COMMAND: "kscand"
    [280170]  PID: 0      TASK: c038e000  CPU: 0   COMMAND: "swapper"
    [280166]  PID: 2106   TASK: c0c0c000  CPU: 0   COMMAND: "sshd"
    [280166]  PID: 2162   TASK: c03a4000  CPU: 0   COMMAND: "vmstat"
    [280160]  PID: 1      TASK: c154a000  CPU: 0   COMMAND: "init"
    [280131]  PID: 3      TASK: c11ce000  CPU: 0   COMMAND: "kapmd"
    [280117]  PID: 1568   TASK: c5a8c000  CPU: 0   COMMAND: "smartd"
    [280103]  PID: 1694   TASK: c4c66000  CPU: 0   COMMAND: "ntpd"
    [280060]  PID: 8      TASK: c7ff2000  CPU: 0   COMMAND: "kupdated"
    [279767]  PID: 1720   TASK: c4608000  CPU: 0   COMMAND: "sendmail"
    [279060]  PID: 13     TASK: c69f4000  CPU: 0   COMMAND: "kjournald"
    [278657]  PID: 1523   TASK: c5ad4000  CPU: 0   COMMAND: "ypbind"
    [277712]  PID: 2163   TASK: c06e0000  CPU: 0   COMMAND: "sshd"
    [277711]  PID: 2244   TASK: c4cdc000  CPU: 0   COMMAND: "ssh"
    [277261]  PID: 1391   TASK: c5d8e000  CPU: 0   COMMAND: "syslogd"
    [276837]  PID: 1990   TASK: c58d8000  CPU: 0   COMMAND: "bash"
    [276802]  PID: 1853   TASK: c3828000  CPU: 0   COMMAND: "atd"
    [276496]  PID: 1749   TASK: c4480000  CPU: 0   COMMAND: "cannaserver"
    [274931]  PID: 1760   TASK: c43ac000  CPU: 0   COMMAND: "crond"
    [246773]  PID: 1844   TASK: c38d8000  CPU: 0   COMMAND: "xfs"
    [125620]  PID: 2170   TASK: c48dc000  CPU: 0   COMMAND: "bash"
    [119059]  PID: 1033   TASK: c64be000  CPU: 0   COMMAND: "kjournald"
    [110916]  PID: 1663   TASK: c528a000  CPU: 0   COMMAND: "sshd"
    [ 86122]  PID: 2112   TASK: c0da6000  CPU: 0   COMMAND: "bash"
    [ 13637]  PID: 1891   TASK: c67ae000  CPU: 0   COMMAND: "sshd"
    [ 13636]  PID: 1894   TASK: c38ec000  CPU: 0   COMMAND: "bash"
    [  7662]  PID: 1885   TASK: c6478000  CPU: 0   COMMAND: "mingetty"
    [  7662]  PID: 1886   TASK: c62da000  CPU: 0   COMMAND: "mingetty"
    [  7662]  PID: 1887   TASK: c5f8c000  CPU: 0   COMMAND: "mingetty"
    [  7662]  PID: 1888   TASK: c5f88000  CPU: 0   COMMAND: "mingetty"
    [  7662]  PID: 1889   TASK: c5f86000  CPU: 0   COMMAND: "mingetty"
    [  7662]  PID: 1890   TASK: c6424000  CPU: 0   COMMAND: "mingetty"
    [  7661]  PID: 4      TASK: c154e000  CPU: 0   COMMAND: "ksoftirqd/0"
    [  7595]  PID: 1872   TASK: c2e7e000  CPU: 0   COMMAND: "inventory.pl"
    [  6617]  PID: 1771   TASK: c435a000  CPU: 0   COMMAND: "jserver"
    [  6307]  PID: 1739   TASK: c48f8000  CPU: 0   COMMAND: "gpm"
    [  6285]  PID: 1729   TASK: c4552000  CPU: 0   COMMAND: "sendmail"
    [  6009]  PID: 1395   TASK: c6344000  CPU: 0   COMMAND: "klogd"
    [  5820]  PID: 1677   TASK: c4d74000  CPU: 0   COMMAND: "xinetd"
    [  5719]  PID: 1422   TASK: c5d04000  CPU: 0   COMMAND: "portmap"
    [  4633]  PID: 1509   TASK: c5ed4000  CPU: 0   COMMAND: "apmd"
    [  4529]  PID: 1520   TASK: c5d98000  CPU: 0   COMMAND: "ypbind"
    [  4515]  PID: 1522   TASK: c5d32000  CPU: 0   COMMAND: "ypbind"
    [  4373]  PID: 1441   TASK: c5d48000  CPU: 0   COMMAND: "rpc.statd"
    [  4210]  PID: 1352   TASK: c5b30000  CPU: 0   COMMAND: "dhclient"
    [  1184]  PID: 71     TASK: c65b6000  CPU: 0   COMMAND: "khubd"
    [   434]  PID: 9      TASK: c11de000  CPU: 0   COMMAND: "mdrecoveryd"
    [    48]  PID: 7      TASK: c7ff4000  CPU: 0   COMMAND: "bdflush"

  Show the kernel stack pointer of each user task:

    crash> ps -us
       PID    PPID  CPU  KSTACKP  ST  %MEM   VSZ   RSS  COMM
          1      0   0  c009bedc  IN   0.0  1096    52  init
        239      1   0  c15e7ed8  IN   0.2  1332   224  pump
        280      1   1  c7cbdedc  IN   0.2  1092   208  portmap
        295      1   0  c7481edc  IN   0.0  1232     0  ypbind
        301    295   0  c7c7bf28  IN   0.1  1260   124  ypbind
        376      1   1  c5053f28  IN   0.0  1316    40  automount
        381      1   0  c34ddf28  IN   0.2  1316   224  automount
        391      1   1  c2777f28  IN   0.2  1316   224  automount
    ...

  Display the argument and environment data for the automount task:

    crash> ps -a automount
    PID: 3948   TASK: f722ee30  CPU: 0   COMMAND: "automount"
    ARG: /usr/sbin/automount --timeout=60 /net program /etc/auto.net
    ENV: SELINUX_INIT=YES
         CONSOLE=/dev/console
         TERM=linux
         INIT_VERSION=sysvinit-2.85
         PATH=/sbin:/usr/sbin:/bin:/usr/bin
         LC_MESSAGES=en_US
         RUNLEVEL=3
         runlevel=3
         PWD=/
         LANG=ja_JP.UTF-8
         PREVLEVEL=N
         previous=N
         HOME=/
         SHLVL=2
         _=/usr/sbin/automount

  Display the tasks in the thread group containing task c20ab0b0:

    crash> ps -g c20ab0b0
    PID: 6425   TASK: f72f50b0  CPU: 0   COMMAND: "firefox-bin"
      PID: 6516   TASK: f71bf1b0  CPU: 0   COMMAND: "firefox-bin"
      PID: 6518   TASK: d394b930  CPU: 0   COMMAND: "firefox-bin"
      PID: 6520   TASK: c20aa030  CPU: 0   COMMAND: "firefox-bin"
      PID: 6523   TASK: c20ab0b0  CPU: 0   COMMAND: "firefox-bin"
      PID: 6614   TASK: f1f181b0  CPU: 0   COMMAND: "firefox-bin"

  Display the tasks in the thread group for each instance of the
  program named "multi-thread":

    crash> ps -g multi-thread
    PID: 2522   TASK: 1003f0dc7f0       CPU: 1   COMMAND: "multi-thread"
      PID: 2523   TASK: 10037b13030       CPU: 1   COMMAND: "multi-thread"
      PID: 2524   TASK: 1003e064030       CPU: 1   COMMAND: "multi-thread"
      PID: 2525   TASK: 1003e13a7f0       CPU: 1   COMMAND: "multi-thread"
   
    PID: 2526   TASK: 1002f82b7f0       CPU: 1   COMMAND: "multi-thread"
      PID: 2527   TASK: 1003e1737f0       CPU: 1   COMMAND: "multi-thread"
      PID: 2528   TASK: 10035b4b7f0       CPU: 1   COMMAND: "multi-thread"
      PID: 2529   TASK: 1003f0c37f0       CPU: 1   COMMAND: "multi-thread"
      PID: 2530   TASK: 10035597030       CPU: 1   COMMAND: "multi-thread"
      PID: 2531   TASK: 100184be7f0       CPU: 1   COMMAND: "multi-thread"

  Display the resource limits of "bash" task 13896:

    crash> ps -r 13896
    PID: 13896  TASK: cf402000  CPU: 0   COMMAND: "bash"
       RLIMIT     CURRENT       MAXIMUM
          CPU   (unlimited)   (unlimited)
        FSIZE   (unlimited)   (unlimited)
         DATA   (unlimited)   (unlimited)
        STACK    10485760     (unlimited)
         CORE   (unlimited)   (unlimited)
          RSS   (unlimited)   (unlimited)
        NPROC      4091          4091
       NOFILE      1024          1024
      MEMLOCK      4096          4096
           AS   (unlimited)   (unlimited)
        LOCKS   (unlimited)   (unlimited)



 26. ) Documentation for crash command pte:


NAME
  pte - translate a page table entry

SYNOPSIS
  pte contents ...

DESCRIPTION
  This command translates the hexadecimal contents of a PTE into its physical
  page address and page bit settings.  If the PTE references a swap location,
  the swap device and offset are displayed.

EXAMPLES

    crash> pte d8e067
     PTE    PHYSICAL  FLAGS
    d8e067   d8e000   (PRESENT|RW|USER|ACCESSED|DIRTY)

    crash> pte 13f600
     PTE      SWAP     OFFSET
    13f600  /dev/hda2   5104



 27. ) Documentation for crash command ptob:


NAME
  ptob - page to bytes

SYNOPSIS
  ptob page_number ...

DESCRIPTION
  This command translates a page frame number to its byte value.

EXAMPLES
    crash> ptob 512a
    512a: 512a000



 28. ) Documentation for crash command ptov:


NAME
  ptov - physical to virtual

SYNOPSIS
  ptov address ...

DESCRIPTION
  This command translates a hexadecimal physical address into a kernel
  virtual address.

EXAMPLES
  Translate physical address 56e000 into a kernel virtual address:

    crash> ptov 56e000
    VIRTUAL   PHYSICAL
    c056e000  56e000



 29. ) Documentation for crash command rd:


NAME
  rd - read memory

SYNOPSIS
  rd [-adDsSupxmfN][-8|-16|-32|-64][-o offs][-e addr][-r file][address|symbol]
     [count]

DESCRIPTION
  This command displays the contents of memory, with the output formatted
  in several different manners.  The starting address may be entered either
  symbolically or by address.  The default output size is the size of a long
  data type, and the default output format is hexadecimal.  When hexadecimal
  output is used, the output will be accompanied by an ASCII translation.

       -p  address argument is a physical address.
       -u  address argument is a user virtual address; only required on
           processors with common user and kernel virtual address spaces.
       -m  address argument is a xen host machine address.
       -f  address argument is a dumpfile offset.
       -d  display output in signed decimal format (default is hexadecimal).
       -D  display output in unsigned decimal format (default is hexadecimal).
       -s  displays output symbolically when appropriate.
       -S  displays output symbolically when appropriate; if the address
           references a slab cache object, the name of the slab cache will
           be displayed in brackets.
       -x  do not display ASCII translation at end of each line.
       -8  display output in 8-bit values.
      -16  display output in 16-bit values.
      -32  display output in 32-bit values (default on 32-bit machines).
      -64  display output in 64-bit values (default on 64-bit machines).
       -a  display output in ASCII characters if the memory contains printable
           ASCII characters; if no count argument is entered, stop at the first
           non-printable character.
       -N  display output in network byte order (only valid for 16- and 32-bit
           values)
  -o offs  offset the starting address by offs.
  -e addr  display memory until reaching specified ending hexadecimal address.
  -r file  dumps raw data to the specified output file; the number of bytes that
           are copied to the file must be specified either by a count argument
           or by the -e option.
  address  starting hexadecimal address:
             1  the default presumes a kernel virtual address.
             2. -p specifies a physical address.
             3. -u specifies a user virtual address, but is only necessary on
                processors with common user and kernel virtual address spaces.
   symbol  symbol of starting address to read.
    count  number of memory locations to display; if entered, it must be the
           last argument on the command line; if not entered, the count defaults
           to 1, or unlimited for -a; when used with the -r option, it is the
           number of bytes to be written to the file.

EXAMPLES
  Display the kernel's version string:

    crash> rd -a linux_banner
    c082a020:  Linux version 2.6.32-119.el6.i686 (mockbuild@hs20-bc2-4.buil
    c082a05c:  d.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13)
    c082a098:   (GCC) ) #1 SMP Tue Mar 1 18:16:57 EST 2011

  Display the same block of memory, first without symbols, again
  with symbols, and then with symbols and slab cache references:

    crash> rd dff12e80 36
    dff12e80:  dff12e94 00000000 c05a363a dff12ed0   ........:6Z.....
    dff12e90:  00000001 dff12e98 0041fe3f ffffffff   ........?.A.....
    dff12ea0:  00000001 d5147800 00000000 def8abc0   .....x..........
    dff12eb0:  dff12ebc c05a4aa0 00000000 dff12ed0   .....JZ.........
    dff12ec0:  00000001 00000000 00000000 00000000   ................
    dff12ed0:  0808b353 00000000 dff12efc c0698220   S........... .i.
    dff12ee0:  dff12efc df7c6480 00000001 c046f99b   .....d|.......F.
    dff12ef0:  00000000 00000000 0808b352 dff12f68   ........R...h/..
    dff12f00:  c155a128 00000000 00000001 ffffffff   (.U.............
    crash> rd -s dff12e80 36
    dff12e80:  dff12e94 00000000 sock_aio_write+83 dff12ed0
    dff12e90:  00000001 dff12e98 0041fe3f ffffffff
    dff12ea0:  00000001 d5147800 00000000 def8abc0
    dff12eb0:  dff12ebc sys_recvfrom+207 00000000 dff12ed0
    dff12ec0:  00000001 00000000 00000000 00000000
    dff12ed0:  0808b353 00000000 dff12efc socket_file_ops
    dff12ee0:  dff12efc df7c6480 00000001 do_sync_write+182
    dff12ef0:  00000000 00000000 0808b352 dff12f68
    dff12f00:  c155a128 00000000 00000001 ffffffff
    crash> rd -S dff12e80 36
    dff12e80:  [size-4096] 00000000 sock_aio_write+83 [size-4096]
    dff12e90:  00000001 [size-4096] 0041fe3f ffffffff
    dff12ea0:  00000001 [sock_inode_cache] 00000000 [filp]  
    dff12eb0:  [size-4096] sys_recvfrom+207 00000000 [size-4096]
    dff12ec0:  00000001 00000000 00000000 00000000
    dff12ed0:  0808b353 00000000 [size-4096] socket_file_ops
    dff12ee0:  [size-4096] [filp]   00000001 do_sync_write+182
    dff12ef0:  00000000 00000000 0808b352 [size-4096]
    dff12f00:  [vm_area_struct] 00000000 00000001 ffffffff

  Read jiffies in hexadecimal and decimal format:

    crash> rd jiffies
    c0213ae0:  0008cc3a                              :...

    crash> rd -d jiffies
    c0213ae0:        577376

  Access the same memory in different sizes:

    crash> rd -64 kernel_version
    c0226a6c:  35312d352e322e32                    2.2.5-15

    crash> rd -32 kernel_version 2
    c0226a6c:  2e322e32 35312d35                     2.2.5-15

    crash> rd -16 kernel_version 4
    c0226a6c:  2e32 2e32 2d35 3531                       2.2.5-15

    crash> rd -8 kernel_version 8
    c0226a6c:  32 2e 32 2e 35 2d 31 35                           2.2.5-15

  Read the range of memory from c009bf2c to c009bf60:

    crash> rd c009bf2c -e c009bf60
    c009bf2c:  c009bf64 c01328c3 c009bf64 c0132838   d....(..d...8(..
    c009bf3c:  0000002a 00000004 c57d77e8 00000104   *........w}.....
    c009bf4c:  0000000b c009a000 7fffffff 00000000   ................
    c009bf5c:  00000000                              ....



 30. ) Documentation for crash command repeat:


NAME
  repeat - repeat a command

SYNOPSIS
  repeat [-seconds] command

DESCRIPTION
  This command repeats a command indefinitely, optionally delaying a given
  number of seconds between each command execution.

    -seconds   The number of seconds to delay between command executions.
               This option must precede the command name to be executed.

  Command execution may be stopped with CTRL-C, or if scrolling is in effect,
  by entering "q".  This command is meant for use on a live system; it is
  hard to conceive of a reason to use it when debugging a crash dump.

EXAMPLES
  Display the value of jiffies once per second:

    crash> repeat -1 p jiffies
    jiffies = $1 = 155551079
    jiffies = $2 = 155551180
    jiffies = $3 = 155551281
    jiffies = $4 = 155551382
    jiffies = $5 = 155551483
    jiffies = $6 = 155551584
    jiffies = $7 = 155551685
    jiffies = $8 = 155551786
    jiffies = $9 = 155551887
    jiffies = $10 = 155551988
    jiffies = $11 = 155552089
    jiffies = $12 = 155552190
    jiffies = $13 = 155552291
    jiffies = $14 = 155552392
    jiffies = $15 = 155552493
    jiffies = $16 = 155552594
    jiffies = $17 = 155552695
    jiffies = $18 = 155552796
    ...



 31. ) Documentation for crash command runq:


NAME
  runq - run queue

SYNOPSIS
  runq 

DESCRIPTION
  This command displays the tasks on the run queues of each cpu.

EXAMPLES
    crash> runq
    CPU 0 RUNQUEUE: ffff880001cdb460
      CURRENT: PID: 2739   TASK: ffff8800320fa7e0  COMMAND: "bash"
      ACTIVE PRIO_ARRAY: ffff880001cdb4d8
         [115] PID: 2739   TASK: ffff8800320fa7e0  COMMAND: "bash"
               PID: 1776   TASK: ffff88003217d820  COMMAND: "syslogd"
      EXPIRED PRIO_ARRAY: ffff880001cdbdb8
         [no tasks queued]
   
    CPU 1 RUNQUEUE: ffff880001ce3460
      CURRENT: PID: 1779   TASK: ffff88003207a860  COMMAND: "klogd"
      ACTIVE PRIO_ARRAY: ffff880001ce34d8
         [115] PID: 1779   TASK: ffff88003207a860  COMMAND: "klogd"
      EXPIRED PRIO_ARRAY: ffff880001ce3db8
         [no tasks queued]



 32. ) Documentation for crash command search:


NAME
  search - search memory

SYNOPSIS
  search [-s start] [ -[kKV] | -u | -p ] [-e end | -l length] [-m mask]
         [-x count] -[cwh] value ...

DESCRIPTION
  This command searches for a given value within a range of user virtual, kernel
  virtual, or physical memory space.  If no end nor length value is entered,
  then the search stops at the end of user virtual, kernel virtual, or physical
  address space, whichever is appropriate.

  An optional mask value may be entered to mask off "don't care" bits.

   -s start  Start the search at this hexadecimal user or kernel virtual
             address, physical address, or kernel symbol.  The start address
             must be appropriate for the memory type specified; if no memory
             type is specified, the default is kernel virtual address space.
         -k  If no start address is specified, start the search at the base
             of kernel virtual address space.  This option is the default.
         -K  Same as -k, except that mapped kernel virtual memory that was
             allocated by vmalloc(), module memory, or virtual mem_map regions
             will not be searched.
         -V  Same as -k, except that unity-mapped kernel virtual memory and
             mapped kernel-text/static-data (x86_64 and ia64) will not be
             searched.
         -u  If no start address is specified, start the search at the base
             of the current context's user virtual address space.  If a start
             address is specified, then this option specifies that the start
             address is a user virtual address.
         -p  If no start address is specified, start the search at the base
             of physical address space.  If a start address is specified, then
             this option specifies that the start address is a physical address.
     -e end  Stop the search at this hexadecimal user or kernel virtual address,
             kernel symbol, or physical address.  The end address must be
             appropriate for the memory type specified.
  -l length  Length in bytes of address range to search.
    -m mask  Ignore the bits that are set in the hexadecimal mask value.
         -c  Search for character string values instead of unsigned longs.  If
             the string contains any space(s), it must be encompassed by double
             quotes.
         -w  Search for unsigned hexadecimal ints instead of unsigned longs.
             This is only meaningful on 64-bit systems in order to search both
             the upper and lower 32-bits of each 64-bit long for the value.
         -h  Search for unsigned hexadecimal shorts instead of unsigned longs.
   -x count  Display the memory contents before and after any found value.  The
             before and after memory context will consist of "count" memory
             items of the same size as the "value" argument.  This option is
             not applicable with the -c option.
      value  Search for this hexadecimal long, unless modified by -c, -w, or -h.

  If -k, -K, -V, -u or -p are not used, then the search defaults to kernel
  virtual address space.  The starting address must be long-word aligned.
  Address ranges that start in user space and end in kernel space are not
  accepted.

EXAMPLES
  Search the current context's address space for all instances of 0xdeadbeef:

    crash> search -u deadbeef
    81aba5c: deadbeef
    81abaa8: deadbeef
    bfffc698: deadbeef
    bffff390: deadbeef

  Search all kernel memory above the kernel text space for all instances
  of 0xabcd occuring in the lower 16-bits of each 32-bit word:

    crash> search -s _etext -m ffff0000 abcd
    c071481c: abcd
    c0c2b0fc: 804abcd
    c0cf5e74: 7489abcd
    c17c0b44: c012abcd
    c1dac730: 3dbeabcd
    c226d0e8: ffffabcd
    c23ed5dc: abcd
    c3022544: 3dbeabcd
    c3069b58: 3dbeabcd
    c3e86e84: aabcd
    c3e88ed0: aabcd
    c3e8ee5c: aabcd
    c3e9df50: aabcd
    c3e9e930: aabcd
    c440a778: 804abcd
    c486eb44: 3dbeabcd
    c578f0fc: 804abcd
    c6394f90: 8ababcd
    c65219f0: 3abcd
    c661399c: abcd
    c68514ac: 8abcd
    c7e036bc: 3dbeabcd
    c7e12568: 5abcd
    c7e1256c: 5abcd

  Search the 4K page at c532c000 for all instances of 0xffffffff:

    crash> search -s c532c000 -l 4096 ffffffff
    c532c33c: ffffffff
    c532c3fc: ffffffff

  Search the static kernel data area for all instances of c2d400eb:

    crash> search -s _etext -e _edata c2d400eb
    c022b550: c2d400eb
    c022b590: c2d400eb
    c022b670: c2d400eb
    c022b6e0: c2d400eb
    c022b7b0: c2d400eb
    c022b7e0: c2d400eb
    c022b8b0: c2d400eb

  Search physical memory for all instances of 0xbabe occuring in the
  upper 16 bits of each 32-bit word:

    crash> search -p babe0000 -m ffff
    2a1dc4: babe671e
    2b6928: babe3de1
    2f99ac: babe0d54
    31843c: babe70b9
    3ba920: babeb5d7
    413ce4: babe7540
    482747c: babe2600
    48579a4: babe2600
    4864a68: babe2600
    ...

  Search physical memory for all instances of 0xbabe occuring in the
  upper 16 bits of each 32-bit word on a 64-bit system:

    crash> search -p babe0000 -m ffff -w
    102e248: babe1174
    11d2f90: babe813d
    122d3ad70: babe6b27
    124d8cd30: babe3dc8
    124d8eefc: babef981
    124d8f060: babe3dc8
    124d8f17c: babefc81
    ...

  Search kernel memory for all instances of 32-bit value 0xbabe1174
  on a 64-bit system:

    crash> search -k -w babe1174
    ffff88000102e248: babe1174
    ffffffff8102e248: babe1174

  Search kernel memory for two strings:

    crash> search -k -c "can't allocate memory" "Failure to"
    ffff8800013ddec1: can't allocate memory for key lists..<3>%s %s: error con
    ffff8801258be748: Failure to install fence: %d..<3>[drm:%s] *ERROR* Failed
    ffff880125f07ec9: can't allocate memory..<3>ACPI: Invalid data..Too many d
    ffffffff813ddec1: can't allocate memory for key lists..<3>%s %s: error con



 33. ) Documentation for crash command set:


NAME
  set - set a process context or internal crash variable

SYNOPSIS
  set [[-a] [pid | taskp] | [-c cpu] | -p] | [crash_variable [setting]] | -v

DESCRIPTION
  This command either sets a new context, or gets the current context for
  display.  The context can be set by the use of:

      pid  a process PID.
    taskp  a hexadecimal task_struct pointer.
       -a  sets the pid or task as the active task on its cpu (dumpfiles only).
   -c cpu  sets the context to the active task on a cpu (dumpfiles only).
       -p  sets the context to the panic task, or back to the crash task on
           a live system.
       -v  display the current state of internal crash variables.

  If no argument is entered, the current context is displayed.  The context
  consists of the PID, the task pointer, the CPU, and task state.
 
  This command may also be used to set internal crash variables.  If no value
  argument is entered, the current value of the crash variable is shown.  These
  are the crash variables, acceptable arguments, and purpose:

          scroll  on | off     controls output scrolling.
          scroll  less         /usr/bin/less as the output scrolling program.
          scroll  more         /bin/more as the output scrolling program.
          scroll  CRASHPAGER   use CRASHPAGER environment variable as the
                               output scrolling program.
           radix  10 | 16      sets output radix to 10 or 16.
         refresh  on | off     controls internal task list refresh.
       print_max  number       set maximum number of array elements to print.
         console  device-name  sets debug console device.
           debug  number       sets crash debug level.
            core  on | off     if on, drops core when the next error message
                               is displayed.
            hash  on | off     controls internal list verification.
          silent  on | off     turns off initialization messages; turns off
                               crash prompt during input file execution.
                               (scrolling is turned off if silent is on)
            edit  vi | emacs   set line editing mode (from .crashrc file only).
        namelist  filename     name of kernel (from .crashrc file only).
   zero_excluded  on | off     controls whether excluded pages from a dumpfile
                               should return zero-filled memory.
       null-stop  on | off     if on, gdb's printing of character arrays will
                               stop at the first NULL encountered.

  Internal variables may be set in four manners:

    1. entering the set command in $HOME/.crashrc.
    2. entering the set command in .crashrc in the current directory.
    3. executing an input file containing the set command.
    4. during runtime with this command.

  During initialization, $HOME/.crashrc is read first, followed by the
  .crashrc file in the current directory.  Set commands in the .crashrc file
  in the current directory override those in $HOME/.crashrc.  Set commands
  entered with this command or by runtime input file override those
  defined in either .crashrc file.  Multiple set command arguments or argument
  pairs may be entered in one command line.

EXAMPLES
  Set the current context to task c2fe8000:

    crash> set c2fe8000
         PID: 15917
     COMMAND: "bash"
        TASK: c2fe8000 
         CPU: 0
       STATE: TASK_INTERRUPTIBLE

  Set the context back to the panicking task:

    crash> set -p
         PID: 698
     COMMAND: "gen12"
        TASK: f9d78000
         CPU: 2
       STATE: TASK_RUNNING (PANIC)

  Turn off output scrolling:

    crash> set scroll off
    scroll: off (/usr/bin/less)

  Show the current state of crash internal variables:

    crash> set -v
            scroll: on (/usr/bin/less)
             radix: 10 (decimal)
           refresh: on
         print_max: 256
           console: /dev/pts/2
             debug: 0
              core: off
              hash: on
            silent: off
              edit: vi
          namelist: vmlinux
     zero_excluded: off
         null-stop: on

  Show the current context:

    crash> set
         PID: 1525
     COMMAND: "bash"
        TASK: c1ede000
         CPU: 0
       STATE: TASK_INTERRUPTIBLE




 34. ) Documentation for crash command sig:


NAME
  sig - task signal handling

SYNOPSIS
  sig [[-l] | [-s sigset]] | [-g] [pid | taskp] ...

DESCRIPTION
  This command displays signal-handling data of one or more tasks.  Multiple
  task or PID numbers may be entered; if no arguments are entered, the signal
  handling data of the current context will be displayed.  The default display
  shows:

    1.  A formatted dump of the "sig" signal_struct structure referenced by
        the task_struct.  For each defined signal, it shows the sigaction
        structure address, the signal handler, the signal sigset_t mask
        (also expressed as a 64-bit hexadecimal value), and the flags.
    2.  Whether the task has an unblocked signal pending.
    3.  The contents of the "blocked" and "signal" sigset_t structures
        from the task_struct/signal_struct, both of which are represented
        as a 64-bit hexadecimal value.
    4.  For each queued signal, private and/or shared, if any, its signal
        number and associated siginfo structure address.

  The -l option lists the signal numbers and their name(s).  The -s option
  translates a 64-bit hexadecimal value representing the contents of a
  sigset_t structure into the signal names whose bits are set.

        pid  a process PID.
      taskp  a hexadecimal task_struct pointer.
         -g  displays signal information for all threads in a task's
             thread group.
         -l  displays the defined signal numbers and names.
  -s sigset  translates a 64-bit hexadecimal value representing a sigset_t
             into a list of signal names associated with the bits set.

EXAMPLES
  Dump the signal-handling data of PID 8970:

    crash> sig 8970
    PID: 8970   TASK: f67d8560  CPU: 1   COMMAND: "procsig"
    SIGNAL_STRUCT: f6018680  COUNT: 1
     SIG SIGACTION  HANDLER       MASK       FLAGS  
     [1]  f7877684  SIG_DFL 0000000000000000 0
     [2]  f7877698  SIG_DFL 0000000000000000 0
    ...
     [8]  f7877710  SIG_DFL 0000000000000000 0
     [9]  f7877724  SIG_DFL 0000000000000000 0
    [10]  f7877738  804867a 0000000000000000 80000000 (SA_RESETHAND)
    [11]  f787774c  SIG_DFL 0000000000000000 0
    [12]  f7877760  804867f 0000000000000000 10000004 (SA_SIGINFO|SA_RESTART)
    [13]  f7877774  SIG_DFL 0000000000000000 0
    ...
    [31]  f78778dc  SIG_DFL 0000000000000000 0
    [32]  f78778f0  SIG_DFL 0000000000000000 0
    [33]  f7877904  SIG_DFL 0000000000000000 0
    [34]  f7877918  804867f 0000000000000000 10000004 (SA_SIGINFO|SA_RESTART)
    [35]  f787792c  SIG_DFL 0000000000000000 0
    [36]  f7877940  SIG_DFL 0000000000000000 0
    ...
    [58]  f7877af8  SIG_DFL 0000000000000000 0
    [59]  f7877b0c  SIG_DFL 0000000000000000 0
    [60]  f7877b20  SIG_DFL 0000000000000000 0
    [61]  f7877b34  SIG_DFL 0000000000000000 0
    [62]  f7877b48  SIG_DFL 0000000000000000 0
    [63]  f7877b5c  SIG_DFL 0000000000000000 0
    [64]  f7877b70  804867f 0000000000000000 10000004 (SA_SIGINFO|SA_RESTART)
   SIGPENDING: no
      BLOCKED: 8000000200000800
   PRIVATE_PENDING
       SIGNAL: 0000000200000800
     SIGQUEUE:  SIG  SIGINFO
                 12  f51b9c84
                 34  f51b9594
   SHARED_PENDING
       SIGNAL: 8000000000000800
     SIGQUEUE:  SIG  SIGINFO
                 12  f51b9188
                 64  f51b9d18
                 64  f51b9500
   
  Dump the signal-handling data for all tasks in the thread group containing
  PID 2578:

    crash> sig -g 2578
    PID: 2387   TASK: f617d020  CPU: 0   COMMAND: "slapd"
    SIGNAL_STRUCT: f7dede00  COUNT: 6
    SIG SIGACTION  HANDLER       MASK       FLAGS
    [1]  c1f60c04   a258a7 0000000000000000 10000000 (SA_RESTART)
    [2]  c1f60c18   a258a7 0000000000000000 10000000 (SA_RESTART)
    [3]  c1f60c2c  SIG_DFL 0000000000000000 0
    [4]  c1f60c40  SIG_DFL 0000000000000000 0
    [5]  c1f60c54   a258a7 0000000000000000 10000000 (SA_RESTART)
    [6]  c1f60c68  SIG_DFL 0000000000000000 0
    [7]  c1f60c7c  SIG_DFL 0000000000000000 0
    [8]  c1f60c90  SIG_DFL 0000000000000000 0
    [9]  c1f60ca4  SIG_DFL 0000000000000000 0
   [10]  c1f60cb8   a25911 0000000000000000 10000000 (SA_RESTART)
   ...
   [64]  c1f610f0  SIG_DFL 0000000000000000 0
   SHARED_PENDING
       SIGNAL: 0000000000000000
     SIGQUEUE: (empty)
    
     PID: 2387   TASK: f617d020  CPU: 0   COMMAND: "slapd"
     SIGPENDING: no
        BLOCKED: 0000000000000000
     PRIVATE_PENDING
         SIGNAL: 0000000000000000
       SIGQUEUE: (empty)
   
     PID: 2392   TASK: f6175aa0  CPU: 0   COMMAND: "slapd"
     SIGPENDING: no
        BLOCKED: 0000000000000000
     PRIVATE_PENDING
         SIGNAL: 0000000000000000
       SIGQUEUE: (empty)
   
     PID: 2523   TASK: f7cd4aa0  CPU: 1   COMMAND: "slapd"
     SIGPENDING: no
        BLOCKED: 0000000000000000
     PRIVATE_PENDING
         SIGNAL: 0000000000000000
       SIGQUEUE: (empty)
   
     ...
   
  Translate the sigset_t mask value, cut-and-pasted from the signal handling
  data from signals 1 and 10 above:

    crash> sig -s 800A000000000201
    SIGHUP SIGUSR1 SIGRTMAX-14 SIGRTMAX-12 SIGRTMAX

  List the signal numbers and their names:

    crash> sig -l
     [1] SIGHUP
     [2] SIGINT
     [3] SIGQUIT
     [4] SIGILL
     [5] SIGTRAP
     [6] SIGABRT/SIGIOT
     [7] SIGBUS
     [8] SIGFPE
     [9] SIGKILL
    [10] SIGUSR1
    [11] SIGSEGV
    [12] SIGUSR2
    [13] SIGPIPE
    [14] SIGALRM
    [15] SIGTERM
    [16] SIGSTKFLT
    [17] SIGCHLD/SIGCLD
    [18] SIGCONT
    [19] SIGSTOP
    [20] SIGTSTP
    [21] SIGTTIN
    [22] SIGTTOU
    [23] SIGURG
    [24] SIGXCPU
    [25] SIGXFSZ
    [26] SIGVTALRM
    [27] SIGPROF
    [28] SIGWINCH
    [29] SIGIO/SIGPOLL
    [30] SIGPWR
    [31] SIGSYS
    [32] SIGRTMIN
    [33] SIGRTMIN+1
    [34] SIGRTMIN+2
    [35] SIGRTMIN+3
    [36] SIGRTMIN+4
    [37] SIGRTMIN+5
    [38] SIGRTMIN+6
    [39] SIGRTMIN+7
    [40] SIGRTMIN+8
    [41] SIGRTMIN+9
    [42] SIGRTMIN+10
    [43] SIGRTMIN+11
    [44] SIGRTMIN+12
    [45] SIGRTMIN+13
    [46] SIGRTMIN+14
    [47] SIGRTMIN+15
    [48] SIGRTMIN+16
    [49] SIGRTMAX-15
    [50] SIGRTMAX-14
    [51] SIGRTMAX-13
    [52] SIGRTMAX-12
    [53] SIGRTMAX-11
    [54] SIGRTMAX-10
    [55] SIGRTMAX-9
    [56] SIGRTMAX-8
    [57] SIGRTMAX-7
    [58] SIGRTMAX-6
    [59] SIGRTMAX-5
    [60] SIGRTMAX-4
    [61] SIGRTMAX-3
    [62] SIGRTMAX-2
    [63] SIGRTMAX-1
    [64] SIGRTMAX



 35. ) Documentation for crash command struct:


NAME
  struct - structure contents

SYNOPSIS
  struct struct_name[.member[,member]][-o][-l offset][-rfuxdp][address | symbol]
         [count | -c count]

DESCRIPTION
  This command displays either a structure definition, or a formatted display
  of the contents of a structure at a specified address.  When no address is
  specified, the structure definition is shown along with the structure size.
  A structure member may be appended to the structure name in order to limit
  the scope of the data displayed to that particular member; when no address
  is specified, the member's offset and definition are shown.

    struct_name  name of a C-code structure used by the kernel.
        .member  name of a structure member; to display multiple members of a
                 structure, use a comma-separated list of members.
             -o  show member offsets when displaying structure definitions.
      -l offset  if the address argument is a pointer to a structure member that
                 is contained by the target data structure, typically a pointer
                 to an embedded list_head, the offset to the embedded member may
                 be entered in either of the following manners:
                   1. in "structure.member" format.
                   2. a number of bytes.
             -r  raw dump of structure data.
             -f  address argument is a dumpfile offset.
             -u  address argument is a user virtual address in the current
                 context.
             -x  override default output format with hexadecimal format.
             -d  override default output format with decimal format.
             -p  if a structure member is a pointer value, show the member's
                 data type on the output line; and on the subsequent line(s),
                 dereference the pointer, display the pointer target's symbol
                 value in brackets if appropriate, and if possible, display the
                 target data; requires an address argument.
        address  hexadecimal address of a structure; if the address points
                 to an embedded list_head structure contained within the
                 target data structure, then the "-l" option must be used.
         symbol  symbolic reference to the address of a structure.
          count  count of structures to dump from an array of structures;
                 if used, this must be the last argument entered.
       -c count  "-c" is only required if "count" is not the last argument
                 entered or if a negative number is entered; if a negative
                 value is entered, the (positive) "count" structures that
                 lead up to and include the target structure will be displayed.

  Structure data, sizes, and member offsets are shown in the current output
  radix unless the -x or -d option is specified.

  Please note that in the vast majority of cases, the "struct" command
  name may be dropped; if the structure name does not conflict with any crash
  or gdb command name, then the "struct_name[.member]" argument will be
  recognized as a structure name, and this command automatically executed.
  See the NOTE below.

EXAMPLES
  Display the vm_area_struct at address c1e44f10:

    crash> struct vm_area_struct c1e44f10
    struct vm_area_struct {
      vm_mm = 0xc2857750,
      vm_start = 0x8048000,
      vm_end = 0x80a5000,
      vm_next = 0xc1e44a10,
      vm_page_prot = {
        pgprot = 0x25     
      },
      vm_flags = 0x1875,
      vm_avl_height = 0x2,  
      vm_avl_left = 0xc30fe200,
      vm_avl_right = 0xc30fed00,
      vm_next_share = 0x0,      
      vm_pprev_share = 0xc1e44a30,
      vm_ops = 0xc0215ca0,
      vm_offset = 0x0,      
      vm_file = 0xc0bfdc70,
      vm_pte = 0  
    }

  Display the definition and size of a vm_area_struct structure.  This first
  example below displays just the structure and size.  The second example
  uses the -o option to also display member offsets.  Both examples were
  run with the output radix set to 10 (decimal):

    crash> struct vm_area_struct
    struct vm_area_struct {
        struct mm_struct *vm_mm;
        long unsigned int vm_start;
        long unsigned int vm_end;
        struct vm_area_struct *vm_next;
        pgprot_t vm_page_prot;
        short unsigned int vm_flags;
        short int vm_avl_height;
        struct vm_area_struct *vm_avl_left;
        struct vm_area_struct *vm_avl_right;
        struct vm_area_struct *vm_next_share;
        struct vm_area_struct **vm_pprev_share;
        struct vm_operations_struct *vm_ops;
        long unsigned int vm_offset;
        struct file *vm_file;
        long unsigned int vm_pte;
    }
    SIZE: 56

    crash> struct vm_area_struct -o
    struct vm_area_struct {
       [0] struct mm_struct *vm_mm;
       [4] long unsigned int vm_start;
       [8] long unsigned int vm_end;
      [12] struct vm_area_struct *vm_next;
      [16] pgprot_t vm_page_prot;
      [20] short unsigned int vm_flags;
      [22] short int vm_avl_height;
      [24] struct vm_area_struct *vm_avl_left;
      [28] struct vm_area_struct *vm_avl_right;
      [32] struct vm_area_struct *vm_next_share;
      [36] struct vm_area_struct **vm_pprev_share;
      [40] struct vm_operations_struct *vm_ops;
      [44] long unsigned int vm_offset;
      [48] struct file *vm_file;
      [52] long unsigned int vm_pte;
    }
    SIZE: 56

  Display the definition and offset of the pgd member of an mm_struct:

    crash> struct mm_struct.pgd
    struct mm_struct {
       [80] pgd_t *pgd;
    }

  Display the pgd member of the mm_struct at address ffff810022e7d080:

    crash> struct mm_struct.pgd ffff810022e7d080
      pgd = 0xffff81000e3ac000

  Display the pgd_t pointed to by the mm_struct.pgd pointer above, forcing
  the output to be expressed in hexadecimal:

    crash> mm_struct.pgd ffff810022e7d080 -px
      pgd_t *pgd = 0xffff81000e3ac000
      -> {
           pgd = 0x2c0a6067
         }

  Display the thread_info structure pointed to by the thread_info
  member of the task_struct at ffff8100181190c0:

    crash> task_struct.thread_info ffff8100181190c0 -p
      struct thread_info *thread_info = 0xffff810023c06000
      -> {
           task = 0xffff8100181190c0,
           exec_domain = 0xffffffff802f78e0,
           flags = 128,
           status = 1,
           cpu = 3,
           preempt_count = 0,
           addr_limit = {
             seg = 18446604435732824064
           },
           restart_block = {
             fn = 0xffffffff80095a52 ,
             arg0 = 0,
             arg1 = 0,
             arg2 = 0,
             arg3 = 0
           }
         }

  Display the flags and virtual members of 4 contigous page structures
  in the mem_map page structure array:

    crash> page.flags,virtual c101196c 4
      flags = 0x8000,
      virtual = 0xc04b0000
   
      flags = 0x8000,
      virtual = 0xc04b1000
   
      flags = 0x8000,
      virtual = 0xc04b2000
   
      flags = 0x8000,
      virtual = 0xc04b3000

  Display the array of tcp_sl_timer structures declared by tcp_slt_array[]:

    crash> struct tcp_sl_timer tcp_slt_array 4
    struct tcp_sl_timer {
      count = {
        counter = 0x0      
      },
      period = 0x32,     
      last = 0x1419e4, 
      handler = 0xc0164854 
    }
    struct tcp_sl_timer {
      count = {
        counter = 0x2      
      },
      period = 0x753,    
      last = 0x14a6df, 
      handler = 0xc01645b0 
    }
    struct tcp_sl_timer {
      count = {
        counter = 0x0      
      },
      period = 0x2ee,    
      last = 0x143134, 
      handler = 0xc016447c 
    }
    struct tcp_sl_timer {
      count = {
        counter = 0x0      
      },
      period = 0x64,     
      last = 0x143198, 
      handler = 0xc0164404 
    }

  Without using the "struct" command name, display the the "d_child"
  list_head member from a dentry structure:

    crash> dentry.d_child 0xe813cb4
      d_child = {
        next = 0x3661344,
        prev = 0xdea4bc4
      },

  Display the child dentry structure referenced by the "next" pointer above.
  Since the "next" address of 0x3661344 above is a pointer to an embedded
  list_head structure within the child dentry structure, the -l option
  is required:

    crash> dentry -l dentry.d_child 0x3661344
    struct dentry {
      d_count = {
        counter = 1
      },
      d_flags = 0,
      d_inode = 0xf9aa604,
      d_parent = 0x11152b1c,
      d_hash = {
        next = 0x11fb3fc0,
        prev = 0x11fb3fc0
      },
      d_lru = {
        next = 0x366133c,
        prev = 0x366133c
      },
      d_child = {
        next = 0x36613cc,
        prev = 0xe813cd4
      },
      d_subdirs = {
        next = 0x366134c,
        prev = 0x366134c
      },
      d_alias = {
        next = 0xf9aa614,
        prev = 0xf9aa614
      },
      d_mounted = 0,
      d_name = {
        name = 0x3661384 "boot.log",
        len = 8,
        hash = 1935169207
      },
      d_time = 1515870810,
      d_op = 0x0,
      d_sb = 0x11fc9c00,
      d_vfs_flags = 0,
      d_fsdata = 0x0,
      d_extra_attributes = 0x0,
      d_iname = "boot.log\000"
    }

NOTE
  If the structure name does not conflict with any crash command name, the
  "struct" command may be dropped.  Accordingly, the examples above could
  also have been accomplished like so:

    crash> vm_area_struct c1e44f10
    crash> vm_area_struct
    crash> vm_area_struct -o
    crash> mm_struct.pgd ffff810022e7d080
    crash> mm_struct.pgd
    crash> tcp_sl_timer tcp_slt_array 4

  Lastly, the short-cut "*" pointer-to command may also be used to negate
  the need to enter the "struct" command name (enter "help *" for details).



 36. ) Documentation for crash command swap:


NAME
  swap - swap device information

SYNOPSIS
  swap 

DESCRIPTION
  This command displays information for each configured swap device.

EXAMPLES
    crash> swap
    FILENAME           TYPE         SIZE       USED   PCT  PRIORITY
    /dev/sda8        PARTITION     136516k    47896k  35%     -1



 37. ) Documentation for crash command sym:


NAME
  sym - translate a symbol to its virtual address, or vice-versa

SYNOPSIS
  sym [-l] | [-M] | [-m module] | [-p|-n] | [-q string] | [symbol | vaddr]

DESCRIPTION
  This command translates a symbol to its virtual address, or a static
  kernel virtual address to its symbol -- or to a symbol-plus-offset value,
  if appropriate.  Additionally, the symbol type is shown in parentheses,
  and if the symbol is a known text value, the file and line number are shown.

              -l  dumps all symbols and their values.
              -M  dumps the current set of module symbols.
       -m module  dumps the current set of symbols for a specified module.
              -p  display the target symbol and the previous symbol.
              -n  display the target symbol and the next symbol.
       -q string  searches for all symbols containing "string".
          symbol  a kernel text or data symbol.
           vaddr  a kernel virtual address.

  If the "symbol", "vaddr" or "string" argument resolves to a module
  symbol, then the module name will be displayed in brackets following the
  symbol value.

EXAMPLES
  Translate data symbol jiffies to its value, and vice-versa:

    crash> sym jiffies
    c0213ae0 (D) jiffies

    crash> sym c0213ae0
    c0213ae0 (D) jiffies

  Translate a text address to its symbolic value and source file:

    crash> sym c0109944
    c0109944 (T) system_call+0x34  ../linux-2.2.5/arch/i386/kernel/signal.c: 723

  Dump the whole symbol table:

    crash> sym -l
    c0100000 (T) _stext
    c0100000 (A) _text
    c0100000 (t) startup_32
    c0100000 (T) stext
    c01000a4 (t) checkCPUtype
    c0100139 (t) is486
    c0100148 (t) is386
    c01001b1 (t) L6
    c01001b3 (t) ready
    c01001b4 (t) check_x87
    c01001da (t) setup_idt
    c01001f7 (t) rp_sidt
    c0100204 (T) stack_start
    c010020c (t) int_msg
    c0100220 (t) ignore_int
    c0100242 (t) idt_descr
    c0100244 (T) idt
    c010024a (t) gdt_descr
    c010024c (T) gdt
    c0101000 (T) swapper_pg_dir
    c0102000 (T) pg0
    c0103000 (T) empty_bad_page
    c0104000 (T) empty_bad_page_table
    c0105000 (T) empty_zero_page
    ...

  Find all symbols containing the string "pipe":

    crash> sym -q pipe
    c010ec60 (T) sys_pipe 
    c012f660 (t) pipe_read 
    c012f7b8 (t) pipe_write 
    c012f9c0 (t) pipe_lseek 
    c012f9d0 (t) bad_pipe_r 
    c012f9dc (t) bad_pipe_w 
    c012f9e8 (t) pipe_ioctl 
    c012fa18 (t) pipe_poll 
    c012fb00 (t) pipe_release 
    c012fb48 (t) pipe_read_release 
    c012fb5c (t) pipe_write_release 
    c012fb70 (t) pipe_rdwr_release 
    c012fba0 (t) pipe_read_open 
    c012fbb0 (t) pipe_write_open 
    c012fbc0 (t) pipe_rdwr_open 
    c012fbec (t) get_pipe_inode 
    c012fcc4 (T) do_pipe 
    c023a920 (D) read_pipe_fops 
    c023a960 (D) write_pipe_fops 
    c023a9a0 (D) rdwr_pipe_fops 
    c023a9e0 (D) pipe_inode_operations 

  Dump the symbols of the uart401 module, both before, and then after,
  the complete set of symbols are loaded with the "mod -s" command:

    crash> sym -m uart401
    c8032000 MODULE START: uart401
    c8032138 (?) uart401intr 
    c803235c (?) attach_uart401 
    c8032638 (?) probe_uart401 
    c80326d4 (?) unload_uart401 
    c8033770 MODULE END: uart401
    crash> mod -s uart401
     MODULE   NAME         SIZE  OBJECT FILE
    c8032000  uart401      6000  /lib/modules/2.2.14/misc/uart401.o
    crash> sym -m uart401
    c8032000 MODULE START: uart401
    c8032050 (t) my_notifier_call 
    c8032084 (t) uart401_status 
    c8032098 (t) uart401_cmd 
    c80320a8 (t) uart401_read 
    c80320bc (t) uart401_write 
    c80320cc (t) uart401_input_loop 
    c8032138 (T) uart401intr 
    c8032168 (t) uart401_open 
    c80321c8 (t) uart401_close 
    c80321f4 (t) uart401_out 
    c80322ac (t) uart401_start_read 
    c80322b4 (t) uart401_end_read 
    c80322bc (t) uart401_kick 
    c80322c4 (t) uart401_buffer_status 
    c80322cc (t) enter_uart_mode 
    c803235c (T) attach_uart401 
    c803259c (t) reset_uart401 
    c8032638 (T) probe_uart401 
    c80326d4 (T) unload_uart401 
    c8032760 (T) init_module
    c80327cc (T) cleanup_module 
    c8032b00 (d) sound_notifier 
    c8032b0c (d) detected_devc 
    c8032b20 (d) std_synth_info 
    c8032bc0 (d) std_midi_synth 
    c8033600 (d) uart401_operations 
    c80336c4 (D) io 
    c80336c8 (D) irq 
    c80336e0 (b) hw_info.508 
    c8033770 MODULE END: uart401

  Display the value of jiffies, along with the next and previous symbols:

    crash> sym -np jiffies
    c023027c (D) prof_shift
    c0230280 (D) jiffies
    c02302a0 (D) task

  Translate a symbol value to its name and module:

    crash> sym f88878d1
    f88878d1 (t) ext3_readdir [ext3]
    crash>



 38. ) Documentation for crash command sys:


NAME
  sys - system data

SYNOPSIS
  sys [-c [name|number]] config

DESCRIPTION
  This command displays system-specific data.  If no arguments are entered,
  the same system data shown during crash invocation is shown.

    -c [name|number]  If no name or number argument is entered, dump all
                      sys_call_table entries.  If a name string is entered,
                      search the table for all entries containing the string.
                      If a number is entered, the table entry associated with
                      that number is displayed.  If the current output radix
                      has been set to 16, the system call numbers will be
                      displayed in hexadecimal.
    config            If the kernel was configured with CONFIG_IKCONFIG, then
                      dump the in-kernel configuration data.
    -panic            Panic a live system.  Requires write permission to
                      /dev/mem.  Results in the crash context causing an
                      "Attempted to kill the idle task!" panic.  (The dump
                      will indicate that the crash context has a PID of 0).

EXAMPLES
  Display essential system information:

    crash> sys
          KERNEL: vmlinux.4
        DUMPFILE: lcore.cr.4
            CPUS: 4
            DATE: Mon Oct 11 18:48:55 1999
          UPTIME: 10 days, 14:14:39
    LOAD AVERAGE: 0.74, 0.23, 0.08
           TASKS: 77
        NODENAME: test.mclinux.com
         RELEASE: 2.2.5-15smp
         VERSION: #24 SMP Mon Oct 11 17:41:40 CDT 1999
         MACHINE: i686  (500 MHz)
          MEMORY: 1 GB

  Dump the system configuration data (if CONFIG_IKCONFIG):

    crash> sys config
    #
    # Automatically generated make config: don't edit
    # Linux kernel version: 2.6.16
    # Mon Apr 10 07:58:06 2006
    #
    CONFIG_X86_64=y
    CONFIG_64BIT=y
    CONFIG_X86=y
    CONFIG_SEMAPHORE_SLEEPERS=y
    CONFIG_MMU=y
    CONFIG_RWSEM_GENERIC_SPINLOCK=y
    CONFIG_GENERIC_CALIBRATE_DELAY=y
    CONFIG_X86_CMPXCHG=y
    CONFIG_EARLY_PRINTK=y
    CONFIG_GENERIC_ISA_DMA=y
    CONFIG_GENERIC_IOMAP=y
    CONFIG_ARCH_MAY_HAVE_PC_FDC=y
    CONFIG_DMI=y
    ...

  Dump the system call table:

    crash> sys -c
    NUM  SYSTEM CALL                FILE AND LINE NUMBER
      0  sys_ni_syscall             ../kernel/sys.c: 48
      1  sys_exit                   ../kernel/exit.c: 404
      2  sys_fork                   ../arch/i386/kernel/process.c: 771
      3  sys_read                   ../fs/read_write.c: 117
      4  sys_write                  ../fs/read_write.c: 146
      5  sys_open                   ../fs/open.c: 754
      6  sys_close                  ../fs/open.c: 839
      7  sys_waitpid                ../kernel/exit.c: 503
      8  sys_creat                  ../fs/open.c: 789
      9  sys_link                   ../fs/namei.c: 1213
     10  sys_unlink                 ../fs/namei.c: 1074
     11  sys_execve                 ../arch/i386/kernel/process.c: 806
    ...

  Find the system call number of the select system call:

    crash> sys -c select
    NUM  SYSTEM CALL                FILE AND LINE NUMBER
     65  sys_select                 ../fs/select.c: 259

    If the current output radix has been set to 16, the system call numbers
    will be displayed in hexadecimal.



 39. ) Documentation for crash command task:


NAME
  task - task_struct contents

SYNOPSIS
  task [-R member[,member]] [-dx] [pid | taskp] ...

DESCRIPTION
  This command dumps a formatted display of the contents of a task_struct.
  Multiple task or PID numbers may be entered; if no arguments are entered,
  the task_struct of the current context is displayed.  The -R option,
  typically invoked indirectly via "foreach task", pares the output down
  to one or more structure members.

        pid  a process PID.
      taskp  a hexadecimal task_struct pointer.
  -R member  a comma-separated list of one or more task_struct members.
         -x  override default output format with hexadecimal format.
         -d  override default output format with decimal format.

EXAMPLES
  Dump the task_struct structure of the current context:

    crash> task
    PID: 18138  TASK: c7d12000  CPU: 1   COMMAND: "crash"
    struct task_struct {
      state = 0,
      flags = 0,
      sigpending = 0,
      addr_limit = {
        seg = 3221225472
      },
      exec_domain = 0xc02386c0,
      need_resched = 0,
      counter = 2,
      priority = 20,
      avg_slice = 0,
      has_cpu = 1,
      processor = 1,
      last_processor = 0,
      lock_depth = 0,
      next_task = 0xc7808000,
      prev_task = 0xc6d1c000,
      next_run = 0xc0252000,
      prev_run = 0xc0252000,
      binfmt = 0xc023abd0,
      exit_code = 0,
      exit_signal = 17,
         .
         .
         .
      files = 0xc09a7d60,
      mm = 0xc753fb50,
      sigmask_lock = {
        lock = 0
      },
      sig = 0xc4745800,
      signal = {
        sig = {65536, 0}
      },
      blocked = {
        sig = {65536, 0}
      },
      sigqueue = 0x0,
      sigqueue_tail = 0xc7d124ac,
      sas_ss_sp = 0,
      sas_ss_size = 0
    }

  Display the ngroups and groups task_struct members for PID 2958:

    crash> task -R ngroups,groups 2958
    PID: 2958   TASK: c6718000  CPU: 0   COMMAND: "bash"
      ngroups = 6,
      groups = {504, 8, 9, 1000, 1007, 1006, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},

  NOTE: When this command is invoked directly (i.e., not from "foreach"), it
  is not necessary to include the "-R" before the task_struct member name(s).



 40. ) Documentation for crash command timer:


NAME
  timer - timer queue data

SYNOPSIS
  timer 

DESCRIPTION
  This command displays the timer queue entries, both old- and new-style,
  in chronological order.  In the case of the old-style timers, the
  timer_table array index is shown; in the case of the new-style timers,
  the timer_list address is shown.  On later kernels, the timer data is
  per-cpu.

EXAMPLES
    crash> timer
    JIFFIES
      68102
    EXPIRES  TIMER_LIST/TABLE  FUNCTION
      68346      c0241934      c01775d4 
      68379      c0241204      c01696d8 
      68523      c7fcdfc0      c0112d6c 
      68718      c7fd8edc      c018719c 
      68723   timer_table[2]   c01c707c 
      68742      c20c1f7c      c0112d6c 
      68742      c20c1f7c      c0112d6c 
      68742      c20c1f7c      c0112d6c 
      68752      c7fd1fc4      c0112d6c 
      68752      c7fd1fc4      c0112d6c 
      68989      c0241d40      c0168060 
      69028      c2533f7c      c0112d6c 
      69134      c22dd868      c0181948 
      71574      c0241430      c0169ea4 
      72179      c7fb1c48      c01cb9a0 
      73144      c1b17f10      c0112d6c 
      73259      c17a5f10      c0112d6c 
     112929      c203ff10      c0112d6c 
     372010      c2323f7c      c0112d6c 
     372138      c2191f10      c0112d6c 
    8653052      c1f13f10      c0112d6c 

  Display the timer queue on a 2-cpu system:

    crash> timer
    TVEC_BASES[0]: c1299be0
     JIFFIES
    18256298
     EXPIRES  TIMER_LIST  FUNCTION
    18256406   cd5ddec0   c01232bb 
    18256677   ceea93e0   c011e3cc 
    18256850   ceea7f64   c01232bb 
    18258751   cd1d4f64   c01232bb 
    18258792   cf5782f0   c011e3cc 
    18261266   c03c9f80   c022fad5 
    18262196   c02dc2e0   c0233329 
    18270518   ceb8bf1c   c01232bb 
    18271327   c03c9120   c0222074 
    18271327   c03ca580   c0233ace 
    18272532   c02d1e18   c0129946 
    18276518   c03c9fc0   c022fd40 
    18332334   ceea9970   c011e3cc 
    18332334   cfb6a840   c011e3cc 
    18665378   cec25ec0   c01232bb 
    TVEC_BASES[1]: c12a1be0
     JIFFIES
    18256298
     EXPIRES  TIMER_LIST  FUNCTION
    18256493   c02c7d00   c013dad5 
    18256499   c12a2db8   c0129946 
    18277900   ceebaec0   c01232bb 
    18283769   cf739f64   c01232bb 
    18331902   cee8af64   c01232bb 



 41. ) Documentation for crash command union:


NAME
  union - union contents

SYNOPSIS
  union union_name[.member[,member]] [-o][-l offset][-rfuxdp] [address | symbol]
                                     [count | -c count]

DESCRIPTION
  This command displays either a union definition, or a formatted display
  of the contents of a union at a specified address.  When no address is
  specified, the union definition is shown along with the union size.
  A union member may be appended to the structure name in order to limit
  the scope of the data displayed to that particular member; when no address
  is specified, the member's offset (always 0) and definition are shown.

     union_name  name of a C-code union used by the kernel.
        .member  name of a union member; to display multiple members of a
                 union, use a comma-separated list of members.
             -o  show member offsets when displaying union definitions.
                 (always 0)
      -l offset  if the address argument is a pointer to a list_head structure
                 that is embedded in the target union structure, the offset
                 to the list_head member may be entered in either of the
                 following manners:
                   1. in "structure.member" format.
                   2. a number of bytes.
             -r  raw dump of union data.
             -f  address argument is a dumpfile offset.
             -x  override default output format with hexadecimal format.
             -d  override default output format with decimal format.
             -p  if a union member is a pointer value, show the member's
                 data type on the output line; and on the subsequent line(s),
                 dereference the pointer, display the pointer target's symbol
                 value in brackets if appropriate, and if possible, display the
                 target data; requires an address argument.
             -u  address argument is a user virtual address in the current
                 context.
        address  hexadecimal address of a union; if the address points
                 to an embedded list_head structure contained within the
                 target union structure, then the "-l" option must be used.
         symbol  symbolic reference to the address of a union.
          count  count of unions to dump from an array of unions; if used,
                 this must be the last argument entered.
       -c count  "-c" is only required if "count" is not the last argument
                 entered or if a negative number is entered; if a negative
                 value is entered, the (positive) "count" structures that
                 lead up to and include the target structure will be displayed.

  Union data, sizes, and member offsets are shown in the current output radix
  unless the -x or -d option is specified.

  Please note that in the vast majority of cases, the "union" command
  name may be dropped; if the union name does not conflict with any crash
  or gdb command name, then the "union_name[.member]" argument will be
  recognized as a union name, and this command automatically executed.
  See the NOTE below.

EXAMPLES

  Display the bdflush_param union definition, and then an instance of it:

    crash> union bdflush_param
    union bdflush_param {
        struct {
            int nfract;
            int ndirty;
            int nrefill;
            int nref_dirt;
            int dummy1;
            int age_buffer;
            int age_super;
            int dummy2;
            int dummy3;
        } b_un;
        unsigned int data[9];
    }
   
    SIZE: 36  (0x24)

    crash> union bdflush_param bdf_prm
    union bdflush_param {
      b_un = {
        nfract = 40,
        ndirty = 500,
        nrefill = 64,
        nref_dirt = 256,
        dummy1 = 15,
        age_buffer = 3000,
        age_super = 500,
        dummy2 = 1884,
        dummy3 = 2
      },
      data = {40, 500, 64, 256, 15, 3000, 500, 1884, 2}
    }

NOTE
  If the union name does not conflict with any crash command name, the
  "union" command may be dropped.  Accordingly, the examples above could
  also have been accomplished like so:

    crash> bdflush_param
    crash> bdflush_param bdf_prm

  Lastly, the short-cut "*" (pointer-to) command may also be used to negate
  the need to enter the "union" command name (enter "help *" for details).



 42. ) Documentation for crash command vm:


NAME
  vm - virtual memory

SYNOPSIS
  vm [-p | -v | -m | [-R reference] | [-f vm_flags]] [pid | taskp] ...

DESCRIPTION
  This command displays basic virtual memory information of a context,
  consisting of a pointer to its mm_struct and page dirctory, its RSS and
  total virtual memory size; and a list of pointers to each vm_area_struct,
  its starting and ending address, vm_flags value, and file pathname.  If no
  arguments are entered, the current context is used.  Additionally, the -p
  option translates each virtual page of each VM area to its physical address.
  The -R option, typically invoked from "foreach vm", searches for references
  to a supplied number, address, or filename argument, and prints only the
  essential information leading up to and including the reference. 
  Alternatively, the -m or -v options may be used to dump the task's mm_struct
  or all of its vm_area_structs respectively.  The -p, -v, -m, -R and -f
  options are all mutually exclusive.

            -p  translate each virtual page to its physical address, or if
                the page is not mapped, its swap device and offset, or
                filename and offset.
  -R reference  search for references to this number or filename.
            -m  dump the mm_struct assocated with the task.
            -v  dump all of the vm_area_structs associated with the task.
   -f vm_flags  translate the bits of a FLAGS (vm_flags) value.
           pid  a process PID.
         taskp  a hexadecimal task_struct pointer.

EXAMPLES
  Display the virtual memory data of the current context:

    crash> vm
    PID: 30986  TASK: c0440000  CPU: 0   COMMAND: "bash"
       MM       PGD       RSS    TOTAL_VM
    c303fe20  c4789000    88k      1728k
      VMA      START      END     FLAGS  FILE
    c0d1f540   8048000   80ad000  1875   /bin/bash
    c0d1f400   80ad000   80b3000  1873   /bin/bash
    c0d1f880   80b3000   80ec000    77
    c0d1f0c0  40000000  40012000   875   /lib/ld-2.1.1.so
    c0d1f700  40012000  40013000   873   /lib/ld-2.1.1.so
    c0d1fe00  40013000  40014000    77
    c0d1f580  40014000  40016000    73
    c0d1f280  4001a000  4004b000    75   /usr/lib/libncurses.so.4.2
    c0d1f100  4004b000  40054000    73   /usr/lib/libncurses.so.4.2
    c0d1f600  40054000  40057000    73
    c0d1f9c0  40057000  40059000    75   /lib/libdl-2.1.1.so
    c0d1f800  40059000  4005a000    73   /lib/libdl-2.1.1.so
    c0d1fd00  4005a000  40140000    75   /lib/libc-2.1.1.so
    c0d1fe40  40140000  40145000    73   /lib/libc-2.1.1.so
    c0d1f780  40145000  40148000    73
    c0d1f140  40148000  40150000    75   /lib/libnss_files-2.1.1.so
    c0d1fa80  40150000  40151000    73   /lib/libnss_files-2.1.1.so
    c0d1fb00  40151000  4015a000    75   /lib/libnss_nisplus-2.1.1.so
    c5f754e0  4015a000  4015b000    73   /lib/libnss_nisplus-2.1.1.so
    c0d1fec0  4015b000  4016d000    75   /lib/libnsl-2.1.1.so
    c5f75460  4016d000  4016e000    73   /lib/libnsl-2.1.1.so
    c5f75420  4016e000  40170000    73
    c5f753e0  40170000  40178000    75   /lib/libnss_nis-2.1.1.so
    c5f753a0  40178000  40179000    73   /lib/libnss_nis-2.1.1.so
    c0d1f240  bfffc000  c0000000   177

  Display the virtual memory data along with page translations for PID 386:

    crash> vm -p 386
    PID: 386    TASK: c11cc000  CPU: 0   COMMAND: "atd"
       MM       PGD       RSS    TOTAL_VM
    c7e30560  c10e5000    104k     1112k
      VMA      START      END     FLAGS  FILE
    c0fbe6a0   8048000   804b000  1875   /usr/sbin/atd
     VIRTUAL  PHYSICAL
     8048000  20e1000
     8049000  17c6000
     804a000  1f6f000
      VMA      START      END     FLAGS  FILE
    c61e0ba0   804b000   804d000  1873   /usr/sbin/atd
     VIRTUAL  PHYSICAL
     804b000  254d000
     804c000  6a9c000
      VMA      START      END     FLAGS  FILE
    c61e04e0   804d000   8050000    77  
     VIRTUAL  PHYSICAL
     804d000  219d000
     804e000  2617000
     804f000  SWAP: /dev/sda8  OFFSET: 24225
      VMA      START      END     FLAGS  FILE
    c61e0720  40000000  40012000   875   /lib/ld-2.1.1.so
     VIRTUAL  PHYSICAL
    40000000  FILE: /lib/ld-2.1.1.so  OFFSET: 0
    40001000  FILE: /lib/ld-2.1.1.so  OFFSET: 1000
    40002000  FILE: /lib/ld-2.1.1.so  OFFSET: 2000
    40003000  FILE: /lib/ld-2.1.1.so  OFFSET: 3000
    40004000  FILE: /lib/ld-2.1.1.so  OFFSET: 4000
    40005000  FILE: /lib/ld-2.1.1.so  OFFSET: 5000
    ...

  Although the -R option is typically invoked from "foreach vm", it can be
  executed directly.  This example displays all VM areas with vm_flags of 75:

    crash> vm -R 75
    PID: 694    TASK: c0c76000  CPU: 1   COMMAND: "crash"
       MM       PGD      RSS    TOTAL_VM
    c6c43110  c0fe9000  8932k    10720k
      VMA       START      END   FLAGS  FILE
    c322c0d0  40019000  4004a000    75  /usr/lib/libncurses.so.4.2
    c67537c0  40056000  40071000    75  /lib/libm-2.1.1.so
    c6753d00  40072000  40074000    75  /lib/libdl-2.1.1.so
    c6753540  40075000  40081000    75  /usr/lib/libz.so.1.1.3
    c6753740  40085000  4016b000    75  /lib/libc-2.1.1.so

  One reason to use -R directly is to pare down the output associated with
  the -p option on a task with a huge address space.  This example displays
  the page data associated with virtual address 40121000:

    crash> vm -R 40121000
    PID: 694    TASK: c0c76000  CPU: 0   COMMAND: "crash"
       MM       PGD      RSS    TOTAL_VM
    c6c43110  c0fe9000  8928k    10720k
      VMA       START      END   FLAGS  FILE
    c6753740  40085000  4016b000    75  /lib/libc-2.1.1.so
    VIRTUAL   PHYSICAL
    40121000  FILE: /lib/libc-2.1.1.so  OFFSET: 9c000

  Display the mm_struct for PID 4777:

    crash> vm -m 4777
    PID: 4777   TASK: c0896000  CPU: 0   COMMAND: "bash"
    struct mm_struct {
      mmap = 0xc6caa1c0,
      mmap_avl = 0x0,
      mmap_cache = 0xc6caabc0,
      pgd = 0xc100a000,
      count = {
        counter = 0x1
      },
      map_count = 0x14,
      mmap_sem = {
        count = {
          counter = 0x1
        },
        waking = 0x0,
        wait = 0x0
      },
      context = 0x0,
      start_code = 0x8048000,
      end_code = 0x809c6f7,
      start_data = 0x0,
      end_data = 0x80a2090,
      start_brk = 0x80a5420,
      brk = 0x80b9000,
      start_stack = 0xbffff9d0,
      arg_start = 0xbffffad1,
      arg_end = 0xbffffad7,
      env_start = 0xbffffad7,
      env_end = 0xbffffff2,
      rss = 0xf6,
      total_vm = 0x1a3,
      locked_vm = 0x0,
      def_flags = 0x0,
      cpu_vm_mask = 0x0,
      swap_cnt = 0x23d,
      swap_address = 0x0,
      segments = 0x0
    }

  Display all of the vm_area_structs for task c47d4000:

    crash> vm -v c47d4000
    PID: 4971   TASK: c47d4000  CPU: 1   COMMAND: "login"
    struct vm_area_struct {
      vm_mm = 0xc4b0d200,
      vm_start = 0x8048000,
      vm_end = 0x804d000,
      vm_next = 0xc3e3abd0,
      vm_page_prot = {
        pgprot = 0x25
      },
      vm_flags = 0x1875,
      vm_avl_height = 0x1,
      vm_avl_left = 0x0,
      vm_avl_right = 0x0,
      vm_next_share = 0x0,
      vm_pprev_share = 0xc3e3abf0,
      vm_ops = 0xc02392a0,
      vm_offset = 0x0,
      vm_file = 0xc1e23660,
      vm_pte = 0x0
    }
    struct vm_area_struct {
      vm_mm = 0xc4b0d200,
      vm_start = 0x804d000,
      vm_end = 0x804e000,
      vm_next = 0xc3e3a010,
      vm_page_prot = {
        pgprot = 0x25
      },
      vm_flags = 0x1873,
      vm_avl_height = 0x2,
      vm_avl_left = 0xc3e3a810,
      vm_avl_right = 0xc3e3a010,
      vm_next_share = 0xc3e3a810,
      vm_pprev_share = 0xc3699c14
      ...

  Translate a FLAGS value:

    crash> vm -f 3875
    3875: (READ|EXEC|MAYREAD|MAYWRITE|MAYEXEC|DENYWRITE|EXECUTABLE|LOCKED)



 43. ) Documentation for crash command vtop:


NAME
  vtop - virtual to physical

SYNOPSIS
  vtop [-c [pid | taskp]] [-u|-k] address ...

DESCRIPTION
  This command translates a user or kernel virtual address to its physical
  address.  Also displayed is the PTE translation, the vm_area_struct data
  for user virtual addresses, the mem_map page data associated with the
  physical page, and the swap location or file location if the page is
  not mapped.  The -u and -k options specify that the address is a user
  or kernel virtual address; -u and -k are not necessary on processors whose
  virtual addresses self-define themselves as user or kernel.  User addresses
  are translated with respect to the current context unless the -c option
  is used.  Kernel virtual addresses are translated using the swapper_pg_dir
  as the base page directory unless the -c option is used.

   -u                 The address is a user virtual address; only required
                      on processors with overlapping user and kernel virtual
                      address spaces.
   -k                 The address is a kernel virtual address; only required
                      on processors with overlapping user and kernel virtual
                      address spaces.
   -c [pid | taskp]   Translate the virtual address from the page directory
                      of the specified PID or hexadecimal task_struct pointer.
                      However, if this command is invoked from "foreach vtop",
                      the pid or taskp argument should NOT be entered; the
                      address will be translated using the page directory of
                      each task specified by "foreach".
   address            A hexadecimal user or kernel virtual address.

EXAMPLES
  Translate user virtual address 80b4000:

    crash> vtop 80b4000
    VIRTUAL   PHYSICAL
    80b4000   660f000
   
    PAGE DIRECTORY: c37f0000
      PGD: c37f0080 => e0d067
      PMD: c37f0080 => e0d067
      PTE: c0e0d2d0 => 660f067
     PAGE: 660f000

      PTE    PHYSICAL  FLAGS
    660f067   660f000  (PRESENT|RW|USER|ACCESSED|DIRTY)
   
      VMA      START      END      FLAGS  FILE
    c773daa0   80b4000   810c000    77
   
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c0393258   660f000         0     17000  1  uptodate
   
  Translate kernel virtual address c806e000, first using swapper_pg_dir
  as the page directory base, and secondly, using the page table base
  of PID 1359:

    crash> vtop c806e000
    VIRTUAL   PHYSICAL
    c806e000  2216000
   
    PAGE DIRECTORY: c0101000
      PGD: c0101c80 => 94063
      PMD: c0101c80 => 94063
      PTE: c00941b8 => 2216063
     PAGE: 2216000

      PTE    PHYSICAL  FLAGS
    2216063   2216000  (PRESENT|RW|ACCESSED|DIRTY)
   
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c02e9370   2216000         0         0  1 
   
    crash> vtop -c 1359 c806e000
    VIRTUAL   PHYSICAL
    c806e000  2216000
   
    PAGE DIRECTORY: c5caf000
      PGD: c5cafc80 => 94063
      PMD: c5cafc80 => 94063
      PTE: c00941b8 => 2216063
     PAGE: 2216000
   
      PTE    PHYSICAL  FLAGS
    2216063   2216000  (PRESENT|RW|ACCESSED|DIRTY)

      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c02e9370   2216000         0         0  1 
   
  Determine swap location of user virtual address 40104000:

    crash> vtop 40104000
    VIRTUAL   PHYSICAL
    40104000  (not mapped)
   
    PAGE DIRECTORY: c40d8000
      PGD: c40d8400 => 6bbe067
      PMD: c40d8400 => 6bbe067
      PTE: c6bbe410 => 58bc00 

     PTE      SWAP     OFFSET
    58bc00  /dev/sda8   22716
   
      VMA      START      END     FLAGS  FILE
    c7200ae0  40104000  40b08000    73  

    SWAP: /dev/sda8  OFFSET: 22716



 44. ) Documentation for crash command waitq:


NAME
  waitq - list tasks queued on a wait queue

SYNOPSIS
  waitq  [ symbol ] | [ struct.member struct_addr ] | [ address ]

DESCRIPTION
  This command walks the wait queue list displaying the tasks which
  are blocked on the specified wait queue.  The command differentiates
  between the old- and new-style wait queue structures used by the kernel.
  It can be invoked with the following argument types:

                     symbol  a global symbol of a wait queue.
  struct.member struct_addr  a structure name and wait queue member combination
                             followed by the structure's hexadecimal address.
                    address  a hexadecimal wait queue pointer.

EXAMPLES

  Find out if any tasks are blocked on the "buffer_wait" wait queue:

    crash> waitq buffer_wait
    wait queue "buffer_wait" (c02927f0) is empty

  See who is blocked on the "wait_chldexit" queue of task c5496000:

    crash> waitq task_struct.wait_chldexit c5496000
    PID: 30879  TASK: c5496000  CPU: 0   COMMAND: "bash"

  Display the task list waiting on a known task queue:

    crash> waitq c3534098
    PID: 13691  TASK: c3534000  CPU: 1   COMMAND: "bash"



 45. ) Documentation for crash command whatis:


NAME
  whatis - search symbol table for data or type information

SYNOPSIS
  whatis [struct | union | typedef | symbol]

DESCRIPTION
  This command displays the definition of structures, unions, typedefs or
  text/data symbols.

    struct  a structure name. The output is the same as if the "struct"
            command was used.
     union  a union name. The output is the same as if the "union" command
            was used.
   typedef  a typedef name. If the typedef translates to a structure or union
            the output is the same as if the "struct" or "union" command
            was used. If the typedef is a primitive datatype, the one-line
            declaration is displayed.
    symbol  a kernel symbol. 

EXAMPLES
   Display the definition of a linux_binfmt structure:

    crash> whatis linux_binfmt
    struct linux_binfmt {
      struct linux_binfmt  *next;
      struct module  *module;
      int (*load_binary) ();
      int (*load_shlib) ();
      int (*core_dump) ();
    };
   
  Since a kmem_bufctl_t is typedef'd to be a kmem_bufctl_s structure, the
  output of the following two commands is identical:

    crash> whatis kmem_bufctl_s
    struct kmem_bufctl_s {
      union {
        struct kmem_bufctl_s  *buf_nextp;
        kmem_slab_t *buf_slabp;
        void *buf_objp;
      } u;
    };
   
    crash> whatis kmem_bufctl_t
    struct kmem_bufctl_s {
      union {
        struct kmem_bufctl_s *buf_nextp;
        kmem_slab_t *buf_slabp;
        void *buf_objp;
      } u;
    };
    SIZE: 4  (0x4)
   
  Display the type data of sys_read() and jiffies text and data symbols:

    crash> whatis sys_read
    ssize_t sys_read(unsigned int, char *, size_t);

    crash> whatis jiffies
    long unsigned int jiffies;

  Display definition of a kdev_t typedef:

    crash> whatis kdev_t
    typedef short unsigned int kdev_t;
    SIZE: 2  (0x2)



 46. ) Documentation for crash command wr:


NAME
  wr - write memory

SYNOPSIS
  wr [-u|-k|-p] [-8|-16|-32|-64] [address|symbol] value

DESCRIPTION
  This command modifies the contents of memory.  The starting address may be
  entered either symbolically or by address.  The default modification size
  is the size of a long data type.  Write permission must exist on the
  /dev/mem.  When writing to memory on a live system, this command should
  obviously be used with great care.

       -u  address argument is a user virtual address.
       -k  address argument is a kernel virtual address.
       -p  address argument is a physical address.
       -8  write data in an 8-bit value.
      -16  write data in a 16-bit value.
      -32  write data in a 32-bit values (default on 32-bit machines).
      -64  write data in a 64-bit values (default on 64-bit machines).
  address  address to write.  The address is considered virtual unless the
           -p option is used.  If a virtual address is specified, the
           -u or -k options are necessary only if the address space cannot
           be determined from the address value itself.  If a user virtual
           address is specified, the address space of the current context
           implied.  The address must be expressed in hexadecimal format.
   symbol  symbol of starting address to write.
    value  the value of the data to write.

EXAMPLES
  Turn on a debug flag:

    crash> wr my_debug_flag 1



 47. ) Documentation for crash command q:


NAME
  q - exit this session

SYNOPSIS
  q 

DESCRIPTION
  Bail out of the current crash session.

NOTE
  This command is equivalent to the "exit" command.



 48. ) Documentation for crash command itself:


USAGE:

  crash [OPTION]... NAMELIST MEMORY-IMAGE  (dumpfile form)
  crash [OPTION]... [NAMELIST]             (live system form)

OPTIONS:

  NAMELIST
    This is a pathname to an uncompressed kernel image (a vmlinux
    file), or a Xen hypervisor image (a xen-syms file) which has
    been compiled with the "-g" option.  If using the dumpfile form,
    a vmlinux file may be compressed in either gzip or bzip2 formats.

  MEMORY-IMAGE
    A kernel core dump file created by the netdump, diskdump, LKCD
    kdump, xendump or kvmdump facilities.

    If a MEMORY-IMAGE argument is not entered, the session will be
    invoked on the live system, which typically requires root privileges
    because of the device file used to access system RAM.  By default,
    /dev/crash will be used if it exists.  If it does not exist, then
    /dev/mem will be used; but if the kernel has been configured with
    CONFIG_STRICT_DEVMEM, then /proc/kcore will be used.  It is permissible
    to explicitly enter /dev/crash, /dev/mem or /proc/kcore.

  mapfile
    If the NAMELIST file is not the same kernel that is running
    (live system form), or the kernel that was running when the system
    crashed (dumpfile form), then the System.map file of the original
    kernel should be entered on the command line.

  -h [option]
  --help [option]
    Without an option argument, display a crash usage help message.
    If the option argument is a crash command name, the help page
    for that command is displayed.  If it is the string "input", a
    page describing the various crash command line input options is
    displayed.  If it is the string "output", a page describing command
    line output options is displayed.  If it is the string "all", then
    all of the possible help messages are displayed.  After the help
    message is displayed, crash exits.

  -s    
    Proceed directly to the "crash>" prompt without displaying any
    version, GPL, or crash initialization data during startup.

  -i file
    Execute the command(s) contained in "file" prior to displaying
    the "crash>" prompt for interactive user input.

  -d num
    Set the internal debug level.  The higher the number, the more
    debugging data will be printed when crash initializes and runs.

  -S    
    Use /boot/System.map as the mapfile.

  -e  vi | emacs
    Set the readline(3) command  line editing mode to "vi" or "emacs". 
    The default editing mode is "vi".

  -f    
    Force the usage of a compressed vmlinux file if its original
    name does not start with "vmlinux".

  -k    
    Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo file.

  -t    
    Display the system-crash timestamp and exit.

  -L    
    Attempt to lock all of its virtual address space into memory by
    calling mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
    If the system call fails, an error message will be displayed,
    but the session continues.

  -c tty-device
    Open the tty-device as the console used for debug messages.

  -p page-size
    If a processor's page size cannot be determined by the dumpfile,
    and the processor default cannot be used, use page-size.

  -m option=value
  --machdep option=value
    Pass an option and value pair to machine-dependent code.  These
    architecture-specific option/pairs should only be required in
    very rare circumstances:

    X86_64:
      physbase=
      irq_eframe_link=
      max_physmem_bits=
      vm=orig       (pre-2.6.11 virtual memory address ranges)
      vm=2.6.11     (2.6.11 and later virtual memory address ranges)
      vm=xen        (Xen kernel virtual memory address ranges)
      vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
    PPC64:
      vm=orig
      vm=2.6.14     (4-level page tables)
    IA64:
      phys_start=
      init_stack_size=
      vm=4l         (4-level page tables)
    ARM:
      physbase=

  -x    
    Automatically load extension modules from a particular directory.
    The directory is determined by the following order of precedence:

    (1) the directory specified in the CRASH_EXTENSIONS shell
        environment variable
    (2) /usr/lib64/crash/extensions  (64-bit  architectures)
    (3) /usr/lib/crash/extensions  (32-bit architectures)
    (4) the ./extensions subdirectory of the current directory

  --memory_module modname
    Use the modname as an alternative kernel module to the crash.ko
    module that creates the /dev/crash device.

  --memory_device device
    Use device as an alternative device to the /dev/crash, /dev/mem
    or /proc/kcore devices.

  --no_kallsyms
    Do not use kallsyms-generated symbol information contained within
    kernel module object files.

  --no_modules
    Do not access or display any kernel module related information.

  --no_ikconfig
    Do not attempt to read configuration data that was built into
    kernels configured with CONFIG_IKCONFIG.

  --no_data_debug
    Do not verify the validity of all structure member offsets and
    structure sizes that it uses.

  --no_kmem_cache
    Do not initialize the kernel's slab cache infrastructure, and
    commands that use kmem_cache-related data will not work.

  --no_elf_notes
    Do not use the registers from the ELF NT_PRSTATUS notes saved
    in a compressed kdump header for backtraces.

  --kmem_cache_delay
    Delay the initialization of the kernel's slab cache infrastructure
    until it is required by a run-time command.

  --readnow
    Pass this flag to the embedded gdb module, which will override
    the two-stage strategy that it uses for reading symbol tables
    from the NAMELIST.  If module symbol tables are loaded during
    runtime with the "mod" command, the same override will occur.

  --smp 
    Specify that the system being analyzed is an SMP kernel.

  -v
  --version
    Display the version of the crash utility, the version of the
    embedded gdb module, GPL information, and copyright notices.

  --cpus number
    Specify the number of cpus in the SMP system being analyzed.

  --osrelease dumpfile
    Display the OSRELEASE vmcoreinfo string from a kdump dumpfile
    header.

  --hyper
    Force the session to be that of a Xen hypervisor.

  --p2m_mfn pfn
    When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile
    is typically analyzed with either the Xen hypervisor or the dom0
    kernel.  It is also possible to analyze any of the guest domU
    kernels if the pfn_to_mfn_list_list pfn value of the guest kernel
    is passed on the command line along with its NAMELIST and the
    dumpfile.

  --xen_phys_start physical-address
    Supply the base physical address of the Xen hypervisor's text
    and static data for older xendump dumpfiles that did not pass
    that information in the dumpfile header.

  --zero_excluded
    If a kdump dumpfile has been filtered to exclude various types of
    non-essential  pages, any attempt to read them will fail.  With
    this flag, reads from any of those pages will return zero-filled
    memory.

  --no_panic
    Do not attempt to find the task that was running when the kernel
    crashed.  Set the initial context to that of the "swapper"  task
    on cpu 0.

  --more
    Use /bin/more as the command output scroller, overriding the
    default of /usr/bin/less and any settings in either ./.crashrc
    or $HOME/.crashrc.

  --less
    Use /usr/bin/less as the command output scroller, overriding any
    settings in either ./.crashrc or $HOME/.crashrc.

  --CRASHPAGER
    Use the output paging command defined in the CRASHPAGER shell
    environment variable, overriding any settings in either ./.crashrc
    or $HOME/.crashrc.

  --no_scroll
    Do not pass run-time command output to any scrolling command.

  --no_crashrc
    Do not execute the commands in either $HOME/.crashrc or ./.crashrc.

  --mod directory
    When loading the debuginfo data of kernel modules with the "mod -S"
    command, search for their object files in directory instead of in
    the standard location.

  --reloc size
    When analyzing live x86 kernels configured with a CONFIG_PHYSICAL_START
    value that is larger than its CONFIG_PHYSICAL_ALIGN value, then it will
    be necessary to enter a relocation size equal to the difference between
    the two values.

  --minimal
    Bring up a session that is restricted to the log, dis, rd, sym,
    eval, set and exit commands.  This option may provide a way to
    extract some minimal/quick information from a corrupted or truncated
    dumpfile, or in situations where one of the several kernel subsystem
    initialization routines would abort the crash session.

  --kvmhost [32|64]
    When examining an x86 KVM guest dumpfile, this option specifies
    that the KVM host that created the dumpfile was an x86 (32-bit)
    or an x86_64 (64-bit) machine, overriding the automatically
    determined value.

  --kvmio
    override the automatically-calculated KVM guest I/O hole size.

FILES:

  .crashrc
    Initialization commands.  The file can be located in the user's
    HOME directory and/or the current directory.  Commands found in
    the .crashrc file in the HOME directory are executed before
    those in the current directory's .crashrc file.

ENVIRONMENT VARIABLES:

  EDITOR
    Command input is read using readline(3).  If EDITOR is set to
    emacs or vi then suitable keybindings are used.  If EDITOR is
    not set, then vi is used.  This can be overridden by "set vi" or
    "set emacs" commands located in a .crashrc file, or by entering
    "-e emacs" on the crash command line.

  CRASHPAGER
    If CRASHPAGER is set, its value is used as the name of the program
    to which command output will be sent.  If not, then command output
    output is sent to "/usr/bin/less -E -X" by default.

  CRASH_MODULE_PATH
    Specifies an alternative directory tree to search for kernel
    module object files.

  CRASH_EXTENSIONS
    Specifies a directory containing extension modules that will be
    loaded automatically if the -x command line option is used.