×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Contact US

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Troubleshooting

How do I analize a system dump ? by MoshiachNow
Posted: 7 Feb 02 (Edited 14 Apr 02)

1.    System Dump
1.1    Overview
During system crash (flashing 888 on the LCD) dump process is invoked.
Dump process will store the entire kernel segment that resides in real memory (the kernel segment is segment 0) to a disk for future debugging. It is actually creates snap shot core image of the machine in the moment of a crash and save it to a disk.
The core will include also the memory resident user data (such as u-blocks).
If the dump process was successful it will be indicated by 0c0 on the LCD.

1.2    Dump Status Codes

The following dump progress indicators, or dump status codes, are part of a Type 102
message.
Note: When a lowercase c is listed, it displays in the lower half of the character
position. Some systems produce 4-digit codes, the two leftmost positions can
have a blanks or zeros. Use the two rightmost digits.
0c0 The dump completed successfully.
0c1 The dump failed due to an I/O error.
0c2 A dump, requested by the user, is started.
0c3 The dump is inhibited.
0c4 The dump device is not large enough.
0c5 The dump did not start, or the dump crashed.
0c6 Dumping to a secondary dump device.
0c7 Reserved.
0c8 The dump function is disabled.
0c9 A dump is in progress.
0cc Unknown dump failure
1.3    Crash Codes

Note: Some systems may produce 4-digit codes. If the leftmost digit of a 4-digit code is
0, use the three rightmost digits.
The following crash codes are part of a Type 102 message.

000 Unexpected system interrupt.
200 Machine check because of a memory bus error.
201 Machine check because of a memory timeout.
202 Machine check because of a memory card failure.
203 Machine check because of a out of range address.
204 Machine check because of an attempt to write to ROS.
205 Machine check because of an uncorrectable address parity.
206 Machine check because of an uncorrectable ECC error.
207 Machine check because of an unidentified error.
208 Machine check due to an L2 uncorrectable ECC.
300 Data storage interrupt from the processor.
32x Data storage interrupt because of an I/O exception from IOCC.
38x Data storage interrupt because of an I/O exception from SLA.
400 Instruction storage interrupt.
500 External interrupt because of a scrub memory bus error.
501 External interrupt because of an unidentified error.
51x External interrupt because of a DMA memory bus error.
52x External interrupt because of an IOCC channel check.
53x External interrupt from an IOCC bus timeout; x represents the IOCC number.
54x External interrupt because of an IOCC keyboard check.
558 There is not enough memory to continue the IPL.
700 Program interrupt.
800 Floating point is not available.

1.4    Enabling system dump
We can check the current dump settings via "smitty dump" or as following :

# sysdumpdev -l

primary              /dev/hd6
secondary            /dev/sysdumpnull
copy directory       /var/adm/ras
forced copy flag     TRUE
always allow dump    TRUE
dump compression     OFF

This means that the system dump is enabled (always allow dump= TRUE) and  it will be copied to /var/adm/ras.
If  always allow dump=FALSE, then the core dump will not be generated.
To change this, change the settings in smit:
Always ALLOW System Dump = true
/var filesystem must have enough space to accommodate a couple of system dumps.
Its size can be increased by, say, 100MB as following:

# chfs  -a  size=+200000   /var
1.5    Analyzing system dump
If the customer complains that his system had frozen with 888 on the display, check errpt for the entry like this:
C0AA5338   0614145601 U S SYSDUMP        SYSTEM DUMP

This means that the system dump have occurred on 14 of June at 14:56.

Run the following command to verify the status of the last system dump:

# sysdumpdev -L

0453-039

Device name:         /dev/hd6
Major device number: 10
Minor device number: 2
Size:                63952384 bytes
Date/Time:           Thu Jun 14 14:43:11 CST 2001
Dump status:         0
dump completed successfully
Dump copy filename: /var/adm/ras/vmcore.0

Run  the crash command in order to get a basic idea on the possible reasons of the system dump.
The crash subcommands (trace -k, thread -r, status 0) are used to provide a hint on the problem origin:

#cd /var/adm/ras
#crash vmcore.0

Using /unix as the default namelist file.

> trace -k
STACK TRACE:
0x2ff3b400 (excpt=edffff54:40000000:00001004:edffff54:00000106) (intpri=0)
        IAR:      .remove_e_list+38 (00032888):   tweqi   r7,0x0
        LR:       .e_block_thread+40c (00034424)
        2ff3b010: .e_sleep_thread+4c (0003497c)
        2ff3b060: .[nspdd]+4144 (016ba4e4)
        2ff3b100: .[nspdd]+2de4 (016b9184)
        2ff3b170: .[nspdd]+7e8 (016b6b88)
        2ff3b1f0: .rdevioctl+140 (001b4344)
        2ff3b260: .vnop_ioctl+1c (001c01d4)
        2ff3b2a0: .vno_ioctl+144 (001d81d8)
        2ff3b360: .common_ioctl+b0 (001e7894)
        2ff3b3c0: .sys_call_ret+0 (00003a90)
IAR not in kernel segment.

> status 0

CPU     TID  TSLOT     PID  PSLOT  STOPPED  PROC_NAME
  0    700f    112    6db0    109      yes  pltDc

> thread -r

SLT ST    TID      PID    CPUID  POLICY PRI CPU    EVENT  PROCNAME     FLAGS
  2 r     205      204        0    FIFO  7f  78               wait
        t_flags:  sig_avail funnel kthread
  3 r     307      306        1    FIFO  7f  78               wait
        t_flags:  sig_avail funnel kthread
  4 r     409      408        2    FIFO  7f  78               wait
        t_flags:  sig_avail funnel kthread
  5 r     50b      50a        3    FIFO  7f  78               wait
        t_flags:  sig_avail funnel kthread
112 r    700f     6db0        0      RR  40   0              pltDc
        t_flags:  local cdefer funnel


> proc -r

SLT ST    PID   PPID   PGRP   UID  EUID  TCNT  NAME
  2 a     204      0      0     0     0     1  wait  
        FLAGS: swapped_in no_swap fixed_pri kproc
  3 a     306      0      0     0     0     1  wait  
        FLAGS: swapped_in no_swap fixed_pri kproc
 55 a    37b8   2282   2282   200   200     1  X  
        FLAGS: swapped_in execed
112 a    7054   571a   25c8   200   200     1  expose  
        FLAGS: swapped_in no_swap fixed_pri ppnocldstop execed
122 a    7a14      1   744c   200   200     1  plateExp_dlg35  
        FLAGS: swapped_in orphanpgrp ppnocldstop execed

>q                ;quits the crash command
=================================================================
In this case trace -k shows a problem with nspdd process, which is part of the TSP driver.
 thread -r and status 0  both hint on the application process pltDc as responsible for the core dump (it's the last process that run).
The core file can be copied on a CD and sent to IBM for further analyzing.

 

Back to IBM: AIX FAQ Index
Back to IBM: AIX Forum

My Archive

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close