Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Register Log in
  • Congratulations derfloh on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

FSCK problems

Status
Not open for further replies.
mrn

mrn

MIS
Joined
Apr 27, 2001
Messages
3,993
Location
GB
Hello,

Were running a HAGEO system and I'm having weird problems with disk corruption on both sides.

A Filesystem became corrupt. I stopped HAGEO and ran an fsck on the filesystem. This appeared to work a second fsck reported no errors. I restarted HAGEO and all looked well for around 12 hours, then the same thing happened again. I failed the systems over this time and the errors on the disk moved across. Again an fsck was done but this time didn't fix the errors.

I've run diag's against both boxes with no errors found.

I'm considering running a certify on each ssa disk.

Any other ideas on tracing this problem?

Mike

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

 
Sort by date Sort by votes
HI,

1.What does "errpt |head -30" and "errpt -a|head -90" say ?
2.What does fsck promt ?any errors/warnings ?

Long live king Moshiach !
 
Upvote 0 Downvote
errpt | head -30

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
BC3BE5A3 0907112905 P S SRC SOFTWARE PROGRAM ERROR
AFA89905 0907112805 I O grpsvcs Group Services daemon started
97419D60 0907112805 I O topsvcs Topology Services daemon started
A6DF45AA 0907112105 I O RMCdaemon The daemon is started.
2BFA76F6 0907095005 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0907112005 T O errdemon ERROR LOGGING TURNED ON
192AC071 0907094805 T O errdemon ERROR LOGGING TURNED OFF
AA8AB241 0907094805 T O OPERATOR OPERATOR NOTIFICATION
BC3BE5A3 0907031305 P S SRC SOFTWARE PROGRAM ERROR
AFA89905 0907031205 I O grpsvcs Group Services daemon started
97419D60 0907031205 I O topsvcs Topology Services daemon started
6D19271E 0907025905 I O topsvcs Topology Services daemon stopped
28854E81 0907025905 I O grpsvcs Group Services daemon stopped
99FA80C7 0907025905 U S haemd SOFTWARE
173C787F 0907005405 I S topsvcs Possible malfunction on local adapter
BC3BE5A3 0906140905 P S SRC SOFTWARE PROGRAM ERROR
AFA89905 0906140905 I O grpsvcs Group Services daemon started
97419D60 0906140805 I O topsvcs Topology Services daemon started
E85C5C4C 0906134205 P S KBDDD SOFTWARE PROGRAM ERROR
E85C5C4C 0906132905 P S KBDDD SOFTWARE PROGRAM ERROR
E85C5C4C 0906131705 P S KBDDD SOFTWARE PROGRAM ERROR
E85C5C4C 0906131305 P S KBDDD SOFTWARE PROGRAM ERROR
E85C5C4C 0906130805 P S KBDDD SOFTWARE PROGRAM ERROR
A6DF45AA 0906125205 I O RMCdaemon The daemon is started.
C0AA5338 0906124905 U S SYSDUMP SYSTEM DUMP
684A365B 0906123105 U U SYSPFS FILE SYSTEM CORRUPTION
9DBCFDEE 0906125005 T O errdemon ERROR LOGGING TURNED ON
684A365B 0906071705 U U SYSPFS FILE SYSTEM CORRUPTION
CD546B25 0906071705 I O SYSPFS FILE SYSTEM RECOVERY REQUIRED

errpt -a|head -90

---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Wed 7 Sep 11:29:12 2005
Sequence Number: 18286
Machine Id: 004214CA4C00
Node Id: one
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Wed 7 Sep 11:28:21 2005
Sequence Number: 18285
Machine Id: 004214CA4C00
Node Id: one
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej031g51/3OH1/2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_61.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Wed 7 Sep 11:28:18 2005
Sequence Number: 18284
Machine Id: 004214CA4C00
Node Id: one
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation

fsck

** Checking /dev/rdhod414l_gmd (/data/)
** Phase 0 - Check Log
log redo processing for /dev/rdhod414l_gmd
** Phase 1 - Check Blocks and Sizes
Block 8392704: duplicate Inode=707996 fragment=100200
Block 8392712: duplicate Inode=707996 fragment=100201
Block 8392720: duplicate Inode=707996 fragment=100202
Block 8392728: duplicate Inode=707996 fragment=100203
Block 8392736: duplicate Inode=707996 fragment=100204
Block 8392744: duplicate Inode=707996 fragment=100205
Block 8392752: duplicate Inode=707996 fragment=100206
Block 8392760: duplicate Inode=707996 fragment=100207
Block 8392768: duplicate Inode=707996 fragment=100208
Block 8392776: duplicate Inode=707996 fragment=100209
Excessive duplicate blocks Inode=707996 (CONTINUED)
** Phase 1b - Rescan For More Duplicate Blocks
Block 8392704: duplicate Inode=1873 fragment=80100200
Block 8392712: duplicate Inode=707996 fragment=100201
Block 8392720: duplicate Inode=707996 fragment=100202
Block 8392728: duplicate Inode=707996 fragment=100203
Block 8392736: duplicate Inode=707996 fragment=100204
Block 8392744: duplicate Inode=707996 fragment=100205
Block 8392752: duplicate Inode=707996 fragment=100206
Block 8392760: duplicate Inode=707996 fragment=100207
Block 8392768: duplicate Inode=707996 fragment=100208
** Phase 2 - Check Pathnames
Duplicate or invalid I=707996 owner=hodElvo mode=100664
size=80640 mtime=06 Sep 15:02 2005
file=/inter/COLROT/out/work/INP9015SUBA.sf (REMOVED)
Unallocated I=710527 owner=leiess mode=100774
size=1571 mtime=06 Sep 15:13 2005
file=/ESS_INT/LEI/arch/LE000007385.UNL (REMOVED)
Unallocated I=704834 owner=leiess mode=100664
size=27440 mtime=06 Sep 15:13 2005
file=/ESS_INT/LEI/log/ESS_INTERFACE.log (REMOVED)
Unallocated I=9811 owner=rotlhu mode=100644
size=0 mtime=06 Sep 15:15 2005
file=/COMMON_193682.dat (REMOVED)
Unallocated I=9809 owner=cerda mode=100664
size=208 mtime=06 Sep 15:12 2005
file=/COMMON_30766_15125688.dat (REMOVED)
Unallocated I=9810 owner=leimbu mode=100644
size=208 mtime=06 Sep 15:15 2005
file=/COMMON_50064_15151161.dat (REMOVED)
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
Unreferenced file I=418 owner=hodarc mode=100664
size=32638976 mtime=06 Sep 14:23 2005 (RECONNECTED)
Unreferenced file I=482 owner=hodarc mode=100664
size=3152896 mtime=06 Sep 14:23 2005 (RECONNECTED)
Unreferenced file I=1015 owner=hodarc mode=100664
size=360223744 mtime=06 Sep 15:15 2005 (RECONNECTED)
Unreferenced file I=1016 owner=hodarc mode=100664
size=987571200 mtime=06 Sep 15:15 2005 (RECONNECTED)
Unreferenced file I=1033 owner=hodarc mode=100664
size=84429824 mtime=06 Sep 15:15 2005 (RECONNECTED)
Unreferenced file I=1034 owner=hodarc mode=100664
size=192548864 mtime=06 Sep 15:15 2005 (RECONNECTED)
Unreferenced file I=1066 owner=hodarc mode=100664
size=125083648 mtime=06 Sep 15:15 2005 (RECONNECTED)
Unreferenced file I=1067 owner=hodarc mode=100664
size=101093376 mtime=06 Sep 15:15 2005 (RECONNECTED)
Duplicate or invalid file I=1873 owner=hodarc mode=100664
size=693016576 mtime=06 Sep 15:15 2005 (CLEARED)
Unreferenced file I=2431 owner=hodarc mode=100664
size=1904856064 mtime=06 Sep 15:15 2005 (RECONNECTED)
Unreferenced file I=2530 owner=hodarc mode=100664
size=1538904064 mtime=06 Sep 15:15 2005 (RECONNECTED)
Unreferenced file I=9505 owner=rotchm mode=100644
size=208 mtime=06 Sep 09:01 2005 (RECONNECTED)
Unreferenced file I=9509 owner=leimbu mode=100644
size=0 mtime=06 Sep 14:00 2005 (RECONNECTED)
Unreferenced file I=9511 owner=rotchm mode=100644
size=208 mtime=06 Sep 09:57 2005 (RECONNECTED)
Unreferenced file I=9514 owner=rotchm mode=100644
size=208 mtime=06 Sep 09:02 2005 (RECONNECTED)
Unreferenced file I=9517 owner=leimbu mode=100644
size=208 mtime=06 Sep 15:15 2005 (RECONNECTED)
NO SPACE LEFT IN lost+found (EXPANDED)
Unreferenced file I=9529 owner=leinc mode=100644
size=208 mtime=06 Sep 14:35 2005 (RECONNECTED)
Unreferenced file I=9530 owner=rotlhu mode=100644
size=208 mtime=06 Sep 14:14 2005 (RECONNECTED)
Unreferenced file I=9533 owner=rotchm mode=100644
size=208 mtime=06 Sep 09:09 2005 (RECONNECTED)
Unreferenced file I=9674 owner=leimbu mode=100644
size=208 mtime=06 Sep 14:05 2005 (RECONNECTED)
Unreferenced file I=9675 owner=hodElvi mode=100664
size=0 mtime=06 Sep 14:10 2005 (RECONNECTED)
Unreferenced file I=9676 owner=rotlhu mode=100644
size=208 mtime=06 Sep 14:14 2005 (RECONNECTED)
Unreferenced file I=704812 owner=hodElvo mode=100664
size=7048 mtime=06 Sep 14:59 2005 (RECONNECTED)
Unreferenced file I=704896 owner=hodElvo mode=100664
size=1664 mtime=06 Sep 15:00 2005 (RECONNECTED)
link count file I=707863 owner=leiess mode=100664
size=8 mtime=06 Sep 15:13 2005
count 0 should be -1 (ADJUSTED)
Duplicate or invalid file I=707996 owner=hodElvo mode=100664
size=80640 mtime=06 Sep 15:02 2005 (CLEARED)
link count file I=708803 owner=hodElvo mode=100664
size=1600 mtime=06 Sep 14:59 2005
count 1 should be 2 (ADJUSTED)
link count file I=708819 owner=hodElvo mode=100664
size=0 mtime=06 Sep 14:59 2005
count 1 should be 2 (ADJUSTED)
link count file I=709019 owner=hodElvo mode=100664
size=2752 mtime=06 Sep 14:59 2005
count 1 should be 2 (ADJUSTED)
Unreferenced file I=709042 owner=hodElvo mode=100664
size=0 mtime=06 Sep 14:59 2005 (RECONNECTED)
Unreferenced file I=710328 owner=hodElvo mode=100664
size=876 mtime=06 Sep 14:59 2005 (RECONNECTED)
** Phase 5 - Check Inode Map
Bad Inode Map (SALVAGED)
** Phase 5b - Salvage Inode Map
** Phase 6 - Check Block Map
Bad Block Map (SALVAGED)
** Phase 6b - Salvage Block Map
-1 blocks missing
-1 blocks missing
94388 files 28083040 blocks 8528968 free
***** Filesystem was modified *****

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

 
Upvote 0 Downvote
HI,

Looking more backwards into "errpt",can you identify what recorded errors preceeded the FS corruptions?
Was it anything to do with a power failure/UPS issues?
Are both machines sitting on the same power system ?
This kind of severe FS corruption (from my experience)I saw caused either by power issues,or by severe HW (disk/cable/ssa/scsi) problems.However these should have showen up in errpt as well - maybe we did not get enough errpt lines above ?

Long live king Moshiach !
 
Upvote 0 Downvote
No Systems are about 15 miles apart. No power problems either side.

full errpt


---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Wed 7 Sep 11:29:12 2005
Sequence Number: 18286
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Wed 7 Sep 11:28:21 2005
Sequence Number: 18285
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej031g51/3OH1/2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_61.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Wed 7 Sep 11:28:18 2005
Sequence Number: 18284
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL001g51/pfi//2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.07.112817.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Wed 7 Sep 11:21:54 2005
Sequence Number: 18283
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.46,209
ERROR ID
6eKora00xf51/rm8//2E.5/...................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: REBOOT_ID
IDENTIFIER: 2BFA76F6

Date/Time: Wed 7 Sep 09:50:30 2005
Sequence Number: 18281
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: TEMP
Resource Name: SYSPROC

Description
SYSTEM SHUTDOWN BY USER

Probable Causes
SYSTEM SHUTDOWN

Detail Data
USER ID
0
0=SOFT IPL 1=HALT 2=TIME REBOOT
1
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
0
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE

Date/Time: Wed 7 Sep 11:20:36 2005
Sequence Number: 18280
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

Recommended Actions
NONE

---------------------------------------------------------------------------
LABEL: ERRLOG_OFF
IDENTIFIER: 192AC071

Date/Time: Wed 7 Sep 09:48:51 2005
Sequence Number: 18279
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED OFF

Probable Causes
ERRSTOP COMMAND

User Causes
ERRSTOP COMMAND

Recommended Actions
RUN ERRDEAD COMMAND
TURN ERROR LOGGING ON

---------------------------------------------------------------------------
LABEL: OPMSG
IDENTIFIER: AA8AB241

Date/Time: Wed 7 Sep 09:48:41 2005
Sequence Number: 18278
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: OPERATOR

Description
OPERATOR NOTIFICATION

User Causes
ERRLOGGER COMMAND

Recommended Actions
REVIEW DETAILED DATA

Detail Data
MESSAGE FROM ERRLOGGER COMMAND
WARNING: clstop called while cluster is not stable. Current state is ( ST_RP_FAILED)
---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Wed 7 Sep 03:13:14 2005
Sequence Number: 18277
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Wed 7 Sep 03:12:23 2005
Sequence Number: 18276
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej05mY51/TdE0/2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_60.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Wed 7 Sep 03:12:21 2005
Sequence Number: 18275
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL03mY51/JWI1/2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.07.031221.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: TS_STOP_ST
IDENTIFIER: 6D19271E

Date/Time: Wed 7 Sep 02:59:43 2005
Sequence Number: 18274
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,comm.C,1.137,593
ERROR ID
6SQG4h/DaY51/bZd0/2E.5/...................
REFERENCE CODE
6UpNEL0eHN51/WIl./2E.5/...................
Topology Services daemon stopped by:
Signal SIGTERM
---------------------------------------------------------------------------
LABEL: GS_STOP_ST
IDENTIFIER: 28854E81

Date/Time: Wed 7 Sep 02:59:42 2005
Sequence Number: 18273
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped manually by user

Recommended Actions
Check that Group Services daemon is stopped

Detail Data
DETECTING MODULE
RSCT,SRCSocket.C,1.53,443
ERROR ID
6/uIVc.CaY51/QIL//2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
Received signal[SIGTERM]. Converted to normal stop
---------------------------------------------------------------------------
LABEL: HA001_TR
IDENTIFIER: 99FA80C7

Date/Time: Wed 7 Sep 02:59:41 2005
Sequence Number: 18272
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: UNKN
Resource Name: haemd

Description
SOFTWARE

Probable Causes
SUBSYSTEM

Failure Causes
SUBSYSTEM

Recommended Actions
NONE

Detail Data
DETECTING MODULE
LPP=PSSP,Fn=emd.c,SID=1.47,L#=478,
DIAGNOSTIC EXPLANATION
haemd: Received termination signal SIGTERM, program is exiting.

---------------------------------------------------------------------------
LABEL: TS_LOC_DOWN_ST
IDENTIFIER: 173C787F

Date/Time: Wed 7 Sep 00:54:58 2005
Sequence Number: 18271
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: INFO
Resource Name: topsvcs

Description
Possible malfunction on local adapter

Probable Causes
Local adapter mal-functioned
Local adapter lost connection to network
Local adapter mis-configured

Failure Causes
Local adapter mal-functioned
Local adapter lost connection to network
Local adapter mis-configured

Recommended Actions
Verify adapter configuration
Verify network connectivity

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.38,4801
ERROR ID
6zV5DL.GlW51/V8k0/2E.5/...................
REFERENCE CODE

Adapter interface name
en2
Adapter offset
2
Adapter IP address
71.1.3.1
---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Tue 6 Sep 14:09:51 2005
Sequence Number: 18270
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Tue 6 Sep 14:09:01 2005
Sequence Number: 18269
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej0hHN51/ORJ1/2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_59.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Tue 6 Sep 14:08:58 2005
Sequence Number: 18268
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL0eHN51/WIl./2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.06.140857.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: LFTERR
IDENTIFIER: E85C5C4C

Date/Time: Tue 6 Sep 13:42:13 2005
Sequence Number: 18267
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: KBDDD

Description
SOFTWARE PROGRAM ERROR

Probable Causes
SOFTWARE PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
IF PROBLEM CONTINUES TO OCCUR REPEATEDLY THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
REPORT DETAILED DATA

Detail Data
DETECTED FAILED RC ERROR LOCATION
kbdintr 03 00 11 AA
---------------------------------------------------------------------------
LABEL: LFTERR
IDENTIFIER: E85C5C4C

Date/Time: Tue 6 Sep 13:29:28 2005
Sequence Number: 18266
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: KBDDD

Description
SOFTWARE PROGRAM ERROR

Probable Causes
SOFTWARE PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
IF PROBLEM CONTINUES TO OCCUR REPEATEDLY THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
REPORT DETAILED DATA

Detail Data
DETECTED FAILED RC ERROR LOCATION
kbdintr 03 00 11 AA
---------------------------------------------------------------------------
LABEL: LFTERR
IDENTIFIER: E85C5C4C

Date/Time: Tue 6 Sep 13:17:03 2005
Sequence Number: 18265
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: KBDDD

Description
SOFTWARE PROGRAM ERROR

Probable Causes
SOFTWARE PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
IF PROBLEM CONTINUES TO OCCUR REPEATEDLY THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
REPORT DETAILED DATA

Detail Data
DETECTED FAILED RC ERROR LOCATION
kbdintr 03 00 11 AA
---------------------------------------------------------------------------
LABEL: LFTERR
IDENTIFIER: E85C5C4C

Date/Time: Tue 6 Sep 13:13:58 2005
Sequence Number: 18264
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: KBDDD

Description
SOFTWARE PROGRAM ERROR

Probable Causes
SOFTWARE PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
IF PROBLEM CONTINUES TO OCCUR REPEATEDLY THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
REPORT DETAILED DATA

Detail Data
DETECTED FAILED RC ERROR LOCATION
kbdintr 03 00 11 AA
---------------------------------------------------------------------------
LABEL: LFTERR
IDENTIFIER: E85C5C4C

Date/Time: Tue 6 Sep 13:08:53 2005
Sequence Number: 18263
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: KBDDD

Description
SOFTWARE PROGRAM ERROR

Probable Causes
SOFTWARE PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
IF PROBLEM CONTINUES TO OCCUR REPEATEDLY THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
REPORT DETAILED DATA

Detail Data
DETECTED FAILED RC ERROR LOCATION
kbdintr 03 00 11 AA
---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Tue 6 Sep 12:52:20 2005
Sequence Number: 18262
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.46,209
ERROR ID
6eKora0o9M51/39H//2E.5/...................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: DUMP_STATS
IDENTIFIER: C0AA5338

Date/Time: Tue 6 Sep 12:49:01 2005
Sequence Number: 18260
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: UNKN
Resource Name: SYSDUMP

Description
SYSTEM DUMP

Probable Causes
UNEXPECTED SYSTEM HALT

User Causes
SYSTEM DUMP REQUESTED BY USER

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Failure Causes
UNEXPECTED SYSTEM HALT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
DUMP DEVICE
/dev/primarydump
DUMP SIZE
191839744
TIME
Tue Sep 6 12:31:54 2005
DUMP TYPE (1 = PRIMARY, 2 = SECONDARY)
1
DUMP STATUS
0
ERROR CODE
0
FILE NAME

PROCESSOR ID
0
---------------------------------------------------------------------------
LABEL: JFS_META_CORRUPTION
IDENTIFIER: 684A365B

Date/Time: Tue 6 Sep 12:31:54 2005
Sequence Number: 18259
Machine Id: 004214CA4C00
Node Id: liv
Class: U
Type: UNKN
Resource Name: SYSPFS
Resource Class: NONE
Resource Type: NONE
Location:
VPD:

Description
FILE SYSTEM CORRUPTION

Probable Causes
INVALID FILE SYSTEM CONTROL DATA

Recommended Actions
PERFORM FULL FILE SYSTEM RECOVERY USING FSCK UTILITY
OBTAIN DUMP
CHECK ERROR LOG FOR ADDITIONAL RELATED ENTRIES

Failure Causes
ADAPTER HARDWARE OR MICROCODE
DISK DRIVE HARDWARE OR MICROCODE
SOFTWARE PROGRAM
STORAGE CABLE LOOSE, DEFECTIVE, OR UNTERMINATED

Recommended Actions
CHECK CABLES AND THEIR CONNECTIONS
INSTALL LATEST ADAPTER AND DRIVE MICROCODE
INSTALL LATEST STORAGE DEVICE DRIVERS
IF PROBLEM PERSISTS, CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
FILE NAME
comsubs.c
LINE NO.
373
MAJOR/MINOR DEVICE NUMBER
0035 0019
ADDITIONAL INFORMATION
4A46 5345 431D 7E2A 0000 001E 0000 002D 0003 1140 0000 0000 0000 0000 0000 0009
0003 8747 0002 E96A 0000 0001 D000 D528 FFFF FFFF F7C5 01DF 0000 0000 0020 0000
2FF3 ABC8 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE

Date/Time: Tue 6 Sep 12:50:04 2005
Sequence Number: 18258
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

Recommended Actions
NONE

---------------------------------------------------------------------------
LABEL: JFS_META_CORRUPTION
IDENTIFIER: 684A365B

Date/Time: Tue 6 Sep 07:17:43 2005
Sequence Number: 18257
Machine Id: 004214CA4C00
Node Id: liv
Class: U
Type: UNKN
Resource Name: SYSPFS
Resource Class: NONE
Resource Type: NONE
Location:
VPD:

Description
FILE SYSTEM CORRUPTION

Probable Causes
INVALID FILE SYSTEM CONTROL DATA

Recommended Actions
PERFORM FULL FILE SYSTEM RECOVERY USING FSCK UTILITY
OBTAIN DUMP
CHECK ERROR LOG FOR ADDITIONAL RELATED ENTRIES

Failure Causes
ADAPTER HARDWARE OR MICROCODE
DISK DRIVE HARDWARE OR MICROCODE
SOFTWARE PROGRAM
STORAGE CABLE LOOSE, DEFECTIVE, OR UNTERMINATED

Recommended Actions
CHECK CABLES AND THEIR CONNECTIONS
INSTALL LATEST ADAPTER AND DRIVE MICROCODE
INSTALL LATEST STORAGE DEVICE DRIVERS
IF PROBLEM PERSISTS, CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
FILE NAME
isubs.c
LINE NO.
2211
MAJOR/MINOR DEVICE NUMBER
0035 0016
ADDITIONAL INFORMATION
4A46 5345 431D 3487 0000 001B 0000 002A 0003 0310 0000 0000 0000 0000 0000 0004
1DB0 0D28 0000 8046 0000 0001 0000 81B4 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
---------------------------------------------------------------------------
LABEL: JFS_FSCK_REQUIRED
IDENTIFIER: CD546B25

Date/Time: Tue 6 Sep 07:17:43 2005
Sequence Number: 18256
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: SYSPFS

Description
FILE SYSTEM RECOVERY REQUIRED

Recommended Actions
PERFORM FULL FILE SYSTEM RECOVERY USING FSCK UTILITY

Detail Data
MAJOR/MINOR DEVICE NUMBER
0035 0016
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/dhod414l_gmd, /data/cert/live-414/hod
---------------------------------------------------------------------------
LABEL: CORE_DUMP
IDENTIFIER: B6048838

Date/Time: Mon 5 Sep 22:07:49 2005
Sequence Number: 18255
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

Recommended Actions
CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
6
USER'S PROCESS ID:
23316
FILE SYSTEM SERIAL NUMBER
30
INODE NUMBER
489490
PROCESSOR ID
-1
CORE FILE NAME
/home/hodtemp1/core
PROGRAM NAME
qutil
ADDITIONAL INFORMATION
??
??
Unable to generate symptom string.
---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Mon 5 Sep 21:24:46 2005
Sequence Number: 18254
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Mon 5 Sep 21:23:53 2005
Sequence Number: 18253
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej0NZ851/XmA//2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_58.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Mon 5 Sep 21:23:50 2005
Sequence Number: 18252
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL0KZ851/qEj//2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.05.212349.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Mon 5 Sep 21:22:14 2005
Sequence Number: 18251
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.46,209
ERROR ID
6eKora0qX851/IyA0/2E.5/...................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: REBOOT_ID
IDENTIFIER: 2BFA76F6

Date/Time: Mon 5 Sep 20:56:50 2005
Sequence Number: 18249
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: TEMP
Resource Name: SYSPROC

Description
SYSTEM SHUTDOWN BY USER

Probable Causes
SYSTEM SHUTDOWN

Detail Data
USER ID
0
0=SOFT IPL 1=HALT 2=TIME REBOOT
0
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
0
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE

Date/Time: Mon 5 Sep 21:20:34 2005
Sequence Number: 18248
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

Recommended Actions
NONE

---------------------------------------------------------------------------
LABEL: ERRLOG_OFF
IDENTIFIER: 192AC071

Date/Time: Mon 5 Sep 20:54:34 2005
Sequence Number: 18247
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED OFF

Probable Causes
ERRSTOP COMMAND

User Causes
ERRSTOP COMMAND

Recommended Actions
RUN ERRDEAD COMMAND
TURN ERROR LOGGING ON

---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Mon 5 Sep 12:39:26 2005
Sequence Number: 18246
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Mon 5 Sep 12:38:35 2005
Sequence Number: 18245
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej0vs051/I.C//2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_57.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Mon 5 Sep 12:38:31 2005
Sequence Number: 18244
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL0rs051/lva1/2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.05.123830.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Mon 5 Sep 12:34:08 2005
Sequence Number: 18243
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.46,209
ERROR ID
6eKora0ko051/9rl//2E.5/...................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: REBOOT_ID
IDENTIFIER: 2BFA76F6

Date/Time: Mon 5 Sep 12:18:59 2005
Sequence Number: 18241
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: TEMP
Resource Name: SYSPROC

Description
SYSTEM SHUTDOWN BY USER

Probable Causes
SYSTEM SHUTDOWN

Detail Data
USER ID
0
0=SOFT IPL 1=HALT 2=TIME REBOOT
0
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
0
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE

Date/Time: Mon 5 Sep 12:32:49 2005
Sequence Number: 18240
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

Recommended Actions
NONE

---------------------------------------------------------------------------
LABEL: ERRLOG_OFF
IDENTIFIER: 192AC071

Date/Time: Mon 5 Sep 12:17:43 2005
Sequence Number: 18239
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED OFF

Probable Causes
ERRSTOP COMMAND

User Causes
ERRSTOP COMMAND

Recommended Actions
RUN ERRDEAD COMMAND
TURN ERROR LOGGING ON

---------------------------------------------------------------------------
LABEL: OPMSG
IDENTIFIER: AA8AB241

Date/Time: Mon 5 Sep 12:17:33 2005
Sequence Number: 18238
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: OPERATOR

Description
OPERATOR NOTIFICATION

User Causes
ERRLOGGER COMMAND

Recommended Actions
REVIEW DETAILED DATA

Detail Data
MESSAGE FROM ERRLOGGER COMMAND
WARNING: clstop called while cluster is not stable. Current state is ( ST_BARRIER)
---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Mon 5 Sep 12:07:45 2005
Sequence Number: 18237
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Mon 5 Sep 12:06:55 2005
Sequence Number: 18236
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej0DP051/p.4//2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_56.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Mon 5 Sep 12:06:53 2005
Sequence Number: 18235
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL0BP051/6F90/2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.05.120653.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: TS_STOP_ST
IDENTIFIER: 6D19271E

Date/Time: Mon 5 Sep 11:53:57 2005
Sequence Number: 18234
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,comm.C,1.137,593
ERROR ID
6SQG4h/3D051/Six//2E.5/...................
REFERENCE CODE
6UpNEL0Lm/51/MEr./2E.5/...................
Topology Services daemon stopped by:
Signal SIGTERM
---------------------------------------------------------------------------
LABEL: GS_STOP_ST
IDENTIFIER: 28854E81

Date/Time: Mon 5 Sep 11:53:56 2005
Sequence Number: 18233
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped manually by user

Recommended Actions
Check that Group Services daemon is stopped

Detail Data
DETECTING MODULE
RSCT,SRCSocket.C,1.53,443
ERROR ID
6/uIVc.2D051/0Jt./2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
Received signal[SIGTERM]. Converted to normal stop
---------------------------------------------------------------------------
LABEL: HA001_TR
IDENTIFIER: 99FA80C7

Date/Time: Mon 5 Sep 11:53:55 2005
Sequence Number: 18232
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: UNKN
Resource Name: haemd

Description
SOFTWARE

Probable Causes
SUBSYSTEM

Failure Causes
SUBSYSTEM

Recommended Actions
NONE

Detail Data
DETECTING MODULE
LPP=PSSP,Fn=emd.c,SID=1.47,L#=478,
DIAGNOSTIC EXPLANATION
haemd: Received termination signal SIGTERM, program is exiting.

---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Mon 5 Sep 11:24:13 2005
Sequence Number: 18231
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Mon 5 Sep 11:23:22 2005
Sequence Number: 18230
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej0Om/51/pEo./2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_55.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Mon 5 Sep 11:23:19 2005
Sequence Number: 18229
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL0Lm/51/MEr./2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.05.112318.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Mon 5 Sep 11:07:48 2005
Sequence Number: 18228
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.46,209
ERROR ID
6eKora0oX/51/uIS1/2E.5/...................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: EPOW_SUS_CHRP
IDENTIFIER: BE0A03E5

Date/Time: Mon 5 Sep 10:51:18 2005
Sequence Number: 18226
Machine Id: 004214CA4C00
Node Id: liv
Class: H
Type: PERM
Resource Name: sysplanar0
Resource Class: planar
Resource Type: sysplanar_rspc
Location:

Description
ENVIRONMENTAL PROBLEM

Probable Causes
Power Turned Off Without a Shutdown
POWER OR FAN COMPONENT

Recommended Actions
RUN SYSTEM DIAGNOSTICS.
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
POWER STATUS REGISTER
0000 0005
PROBLEM DATA
0254 0040 0000 003E C600 9500 0951 1800 2005 0905 0000 0005 4010 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 4942 4D00 5631 2050 3200 0000 0008 3031
0045 0000 0002 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE

Date/Time: Mon 5 Sep 11:05:34 2005
Sequence Number: 18225
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

Recommended Actions
NONE

---------------------------------------------------------------------------
LABEL: ERRLOG_OFF
IDENTIFIER: 192AC071

Date/Time: Mon 5 Sep 03:10:45 2005
Sequence Number: 18224
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED OFF

Probable Causes
ERRSTOP COMMAND

User Causes
ERRSTOP COMMAND

Recommended Actions
RUN ERRDEAD COMMAND
TURN ERROR LOGGING ON

---------------------------------------------------------------------------
LABEL: TS_STOP_ST
IDENTIFIER: 6D19271E

Date/Time: Mon 5 Sep 03:10:01 2005
Sequence Number: 18223
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,comm.C,1.137,593
ERROR ID
6SQG4h/tXu41/K8P./2E.5/...................
REFERENCE CODE
6UpNEL0SQu41/J79//2E.5/...................
Topology Services daemon stopped by:
Signal SIGTERM
---------------------------------------------------------------------------
LABEL: GS_STOP_ST
IDENTIFIER: 28854E81

Date/Time: Mon 5 Sep 03:09:59 2005
Sequence Number: 18222
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped manually by user

Recommended Actions
Check that Group Services daemon is stopped

Detail Data
DETECTING MODULE
RSCT,SRCSocket.C,1.53,443
ERROR ID
6/uIVc.rXu41/EhD1/2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
Received signal[SIGTERM]. Converted to normal stop
---------------------------------------------------------------------------
LABEL: HA001_TR
IDENTIFIER: 99FA80C7

Date/Time: Mon 5 Sep 03:09:59 2005
Sequence Number: 18221
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: UNKN
Resource Name: haemd

Description
SOFTWARE

Probable Causes
SUBSYSTEM

Failure Causes
SUBSYSTEM

Recommended Actions
NONE

Detail Data
DETECTING MODULE
LPP=PSSP,Fn=emd.c,SID=1.47,L#=478,
DIAGNOSTIC EXPLANATION
haemd: Received termination signal SIGTERM, program is exiting.

---------------------------------------------------------------------------
LABEL: OPMSG
IDENTIFIER: AA8AB241

Date/Time: Mon 5 Sep 03:09:38 2005
Sequence Number: 18220
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: OPERATOR

Description
OPERATOR NOTIFICATION

User Causes
ERRLOGGER COMMAND

Recommended Actions
REVIEW DETAILED DATA

Detail Data
MESSAGE FROM ERRLOGGER COMMAND
WARNING: clstop called while cluster is not stable. Current state is ( ST_BARRIER)
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Mon 5 Sep 03:02:08 2005
Sequence Number: 18219
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej0UQu41/45B./2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_54.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Mon 5 Sep 03:02:06 2005
Sequence Number: 18218
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL0SQu41/J79//2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.05.030206.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: TS_STOP_ST
IDENTIFIER: 6D19271E

Date/Time: Mon 5 Sep 02:58:47 2005
Sequence Number: 18217
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,comm.C,1.137,593
ERROR ID
6SQG4h/LNu41/KTF1/2E.5/...................
REFERENCE CODE
6UpNEL0jAu41/Tih//2E.5/...................
Topology Services daemon stopped by:
Signal SIGTERM
---------------------------------------------------------------------------
LABEL: GS_STOP_ST
IDENTIFIER: 28854E81

Date/Time: Mon 5 Sep 02:58:46 2005
Sequence Number: 18216
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon stopped

Probable Causes
Daemon stopped by SRC
Daemon stopped by signal

User Causes
Daemon stopped manually by user

Recommended Actions
Check that Group Services daemon is stopped

Detail Data
DETECTING MODULE
RSCT,SRCSocket.C,1.53,443
ERROR ID
6/uIVc.KNu41/JR30/2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
Received signal[SIGTERM]. Converted to normal stop
---------------------------------------------------------------------------
LABEL: HA001_TR
IDENTIFIER: 99FA80C7

Date/Time: Mon 5 Sep 02:58:46 2005
Sequence Number: 18215
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: UNKN
Resource Name: haemd

Description
SOFTWARE

Probable Causes
SUBSYSTEM

Failure Causes
SUBSYSTEM

Recommended Actions
NONE

Detail Data
DETECTING MODULE
LPP=PSSP,Fn=emd.c,SID=1.47,L#=478,
DIAGNOSTIC EXPLANATION
haemd: Received termination signal SIGTERM, program is exiting.

---------------------------------------------------------------------------
LABEL: OPMSG
IDENTIFIER: AA8AB241

Date/Time: Mon 5 Sep 02:58:32 2005
Sequence Number: 18214
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: OPERATOR

Description
OPERATOR NOTIFICATION

User Causes
ERRLOGGER COMMAND

Recommended Actions
REVIEW DETAILED DATA

Detail Data
MESSAGE FROM ERRLOGGER COMMAND
WARNING: clstop called while cluster is not stable. Current state is ( ST_BARRIER)
---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Mon 5 Sep 02:46:12 2005
Sequence Number: 18213
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: PERM
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
256
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
krpcmond
---------------------------------------------------------------------------
LABEL: GS_START_ST
IDENTIFIER: AFA89905

Date/Time: Mon 5 Sep 02:45:23 2005
Sequence Number: 18212
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: grpsvcs

Description
Group Services daemon started

Probable Causes
Daemon started during system startup
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Check that Group Services daemon is running

Detail Data
DETECTING MODULE
RSCT,pgsd.C,1.54,554
ERROR ID
63Y7ej0nAu41/AG6//2E.5/...................
REFERENCE CODE

DIAGNOSTIC EXPLANATION
HAGS daemon started by SRC. Log file is /var/ha/log/grpsvcs_1_53.cert.
---------------------------------------------------------------------------
LABEL: TS_START_ST
IDENTIFIER: 97419D60

Date/Time: Mon 5 Sep 02:45:19 2005
Sequence Number: 18211
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: topsvcs

Description
Topology Services daemon started

Probable Causes
Daemon started during system start-up
Daemon re-started automatically by SRC
Daemon started during installation
Daemon started manually by user

User Causes
Daemon started manually by user

Recommended Actions
Confirm that this is desirable

Detail Data
DETECTING MODULE
rsct,bootstrp.C,1.184,4129
ERROR ID
6UpNEL0jAu41/Tih//2E.5/...................
REFERENCE CODE

Topology Services daemon started by:
SRC
Topology Services daemon log file location
/var/ha/log/topsvcs.05.024518.cert.en_GB
Topology Services daemon run directory
/var/ha/run/topsvcs.cert/
---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Mon 5 Sep 02:43:12 2005
Sequence Number: 18210
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.46,209
ERROR ID
6eKora0k8u41/CRl./2E.5/...................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: REBOOT_ID
IDENTIFIER: 2BFA76F6

Date/Time: Mon 5 Sep 02:28:01 2005
Sequence Number: 18208
Machine Id: 004214CA4C00
Node Id: liv
Class: S
Type: TEMP
Resource Name: SYSPROC

Description
SYSTEM SHUTDOWN BY USER

Probable Causes
SYSTEM SHUTDOWN

Detail Data
USER ID
0
0=SOFT IPL 1=HALT 2=TIME REBOOT
0
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
0
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE

Date/Time: Mon 5 Sep 02:41:53 2005
Sequence Number: 18207
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

Recommended Actions
NONE

---------------------------------------------------------------------------
LABEL: ERRLOG_OFF
IDENTIFIER: 192AC071

Date/Time: Mon 5 Sep 02:26:31 2005
Sequence Number: 18206
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED OFF

Probable Causes
ERRSTOP COMMAND

User Causes
ERRSTOP COMMAND

Recommended Actions
RUN ERRDEAD COMMAND
TURN ERROR LOGGING ON

---------------------------------------------------------------------------
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Sat 3 Sep 22:37:50 2005
Sequence Number: 18205
Machine Id: 004214CA4C00
Node Id: liv
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.46,209
ERROR ID
6eKora0iSV41/6ts//2E.5/...................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: DUMP_STATS
IDENTIFIER: C0A
 
Upvote 0 Downvote
Status
Not open for further replies.

Similar threads

nmorph
  • Locked
  • Question Question
HA Problems
Replies
2
Views
303
nmorph
nmorph
aixchild
  • Locked
  • Question Question
fsck at bootup
Replies
1
Views
137
khalidaaa
khalidaaa
holdahl
  • Locked
  • Question Question
problems using DSH
Replies
5
Views
489
holdahl
holdahl
holdahl
  • Locked
  • Question Question
problems with NFS mount
Replies
3
Views
403
holdahl
holdahl
holdahl
  • Locked
  • Question Question
problems with LAN adapter
Replies
4
Views
278
khalidaaa
khalidaaa

Part and Inventory Search

Sponsor

Back
Top