Alarm Interpretations

[techom]

Previously, the redundant controller won't takeover the task of handling current transactions when the Primary was experiencing errors. The peripheral cards (TICs & DTICs) of the upper shelf would be inoperational when this happens. Only the lower half was capable of handling calls and transactions. The TSB was replaced but after several days the problem would re-occur. We've replaced almost all major cards in both the Primary and Backup Controllers, all four Switching Shelf Controller cards, TimeSlot Bridge and the corresponding paddleboards of our Aspect ACD 200R System but we still are at a loss why the System went down. Perhaps, the activity logs we've collected would be of help in interpreting the possible causes of the problem or problems we've encountered. Here is a cut from the collected error logs.

03262004 07:50:22 SYSINFO Switching shelf communication failure Count 201 1 126 SWITCH ss_txclean 809
03262004 07:50:22 SYSINFO Enabling digital master Card 41 SSC Mstr 1 State 7 Priority 95 135 SWITCH ssfind_t1mas 6425
03262004 07:50:22 SYSINFO Switching shelf communication failure Count 201 2 126 SWITCH ss_txclean 809
03262004 07:50:23 CRITICAL Hardware error Error SHF FAULT Devtype SHELF Devnum 1 40 CP shelf_down 220
03262004 07:50:23 CUSTINFO Hardware error Devtype CARD Devnum 3 40 CP ad_offcard 324
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 1 CardPort 3001 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 2 CardPort 3002 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 CRITICAL BACKUP SYSTEM CONTROLLER ACTIVATED Reason request 0 (null) 141 WCHDOG wdutil.c 480
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 3 CardPort 3003 ErrCode 1 51 VOICE vss_music 272
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 4 CardPort 3004 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 5 CardPort 3005 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 6 CardPort 3006 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 7 CardPort 3007 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 8 CardPort 3008 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 CUSTINFO Port error report PortType VSI PortNum 9 CardPort 3009 ErrCode 1 51 VOICE vss_idle_onln 177
03262004 07:50:23 COMMENT Error Suppression: error 51 from vss_idle_onln is being suppressed logd
03262004 07:50:23 CUSTINFO Hardware error Devtype CARD Devnum 7 40 CP ad_offcard 324
03262004 07:50:23 CUSTINFO Hardware error Devtype INSTR Devnum 1 CardPort 7001 40 INSTR ad_offinst 229
03262004 07:50:23 CUSTINFO Hardware error Devtype INSTR Devnum 2 CardPort 7002 40 INSTR ad_offinst 229
03262004 07:50:23 CUSTINFO Hardware error Devtype INSTR Devnum 3 CardPort 7003 40 INSTR ad_offinst 229
03262004 07:50:23 CUSTINFO Hardware error Devtype INSTR Devnum 4 CardPort 7004 40 INSTR ad_offinst 229
03262004 07:50:23 CUSTINFO Hardware error Devtype INSTR Devnum 5 CardPort 7005 40 INSTR ad_offinst 229
03262004 07:50:23 WARNING Event handler error TaskName yabr EvntCode VOICE_SERV ErrCode 40000 138 ADMIN yabr 938
03262004 07:50:23 CUSTINFO Hardware error Devtype INSTR Devnum 6 CardPort 7006 40 INSTR ad_offinst 229
03262004 07:50:23 COMMENT Error Suppression: error 40 from ad_offinst is being suppressed logd
03262004 07:50:23 CUSTINFO Hardware error Devtype CARD Devnum 8 40 CP ad_offcard 324
03262004 07:50:23 CUSTINFO Hardware error Devtype CARD Devnum 9 40 CP ad_offcard 324
03262004 07:50:23 CUSTINFO Hardware error Devtype CARD Devnum 10 40 CP ad_offcard 324
03262004 07:50:23 CUSTINFO Hardware error Devtype CARD Devnum 11 40 CP ad_offcard 324
03262004 07:50:23 CUSTINFO Hardware error Devtype CARD Devnum 12 40 CP ad_offcard 324
03262004 07:50:23 COMMENT Error Suppression: error 40 from ad_offcard is being suppressed logd
03262004 07:50:23 CUSTINFO Port error report PortType TRUNK PortNum 1 CardPort 15001 ErrCode 1 51 TRUNK isport_comm_er 310
03262004 07:50:23 CUSTINFO Hardware error Devtype TRUNK Devnum 1 CardPort 15001 40 TRUNK ad_offtrk 400
03262004 07:50:23 CUSTINFO Port error report PortType TRUNK PortNum 2 CardPort 15002 ErrCode 1 51 TRUNK isport_comm_er 310
03262004 07:50:23 CUSTINFO Hardware error Devtype TRUNK Devnum 2 CardPort 15002 40 TRUNK ad_offtrk 400
03262004 07:50:23 CUSTINFO Port error report PortType TRUNK PortNum 3 CardPort 15003 ErrCode 1 51 TRUNK isport_comm_er 310
03262004 07:50:23 CUSTINFO Hardware error Devtype TRUNK Devnum 3 CardPort 15003 40 TRUNK ad_offtrk 400
03262004 07:50:23 CUSTINFO Port error report PortType TRUNK PortNum 4 CardPort 15004 ErrCode 1 51 TRUNK isport_comm_er 310
03262004 07:50:23 CUSTINFO Hardware error Devtype TRUNK Devnum 4 CardPort 15004 40 TRUNK ad_offtrk 400
03262004 07:50:23 CUSTINFO Port error report PortType TRUNK PortNum 5 CardPort 15005 ErrCode 1 51 TRUNK isport_comm_er 310
03262004 07:50:23 CUSTINFO Hardware error Devtype TRUNK Devnum 5 CardPort 15005 40 TRUNK ad_offtrk 400
03262004 07:50:24 COMMENT Error Suppression: error 51 from isport_comm_er is being suppressed logd
03262004 07:50:24 COMMENT Error Suppression: error 40 from ad_offtrk is being suppressed logd
03262004 07:50:24 CRITICAL Hardware error Error SHF FAULT Devtype SHELF Devnum 2 40 CP shelf_down 220
03262004 07:50:26 SYSINFO Message queue exceeded previous maxdepth QName DABR OldDepth 24 NewDepth 462 64 SYSTEM acc_system.cpp 915
03262004 07:50:26 SYSINFO Message queue exceeded previous maxdepth QName all_queues OldDepth 63 NewDepth 513 64 SYSTEM acc_system.cpp 962
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 30 CardPort 8014 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 SYSINFO Hardware initialization failure Error 2 DevType INSTRUMENT DevNum 2941 CardPort 27010 43 INSTR uss_vcntopen 158
03262004 07:50:27 COMMENT Error Suppression: error 43 from uss_vcntopen is being suppressed logd
03262004 07:51:24 CRITICAL Voice system not up! Will retry in time VsSys 1 cRetry 0 Time 0 seconds Reason No VS KA 21 VOICE dovs_alive 758
03262004 07:51:24 CUSTINFO Voice system reset successful VsSys 1 cPort 0 cDisk 0 cMsg 0 23 VOICE dovs_reset 0
03262004 07:51:24 CUSTINFO Voice system disk usage VsSys 1 BlkTotal 0 BlksUsed 0 Reason 0 16 VOICE dovs_reset 156
03262004 07:51:24 CUSTINFO Voice system message inventory VsSys 1 TotalRcd 0 20 VOICE dovs_reset 164
03262004 07:51:30 CRITICAL Voice system not up! Will retry in time VsSys 1 cRetry 0 Time 0 seconds Reason No VS DCD 21 VOICE dovs_alive 802
03262004 07:51:30 CRITICAL Voice system not up! Will retry in time VsSys 1 cRetry 1 Time 1 minute Reason No VS DCD 21 VOICE dovs_reset 215
03262004 07:52:00 WARNING Event handler error TaskName du EvntCode 2 ErrCode 0 138 ADMIN du 641
03262004 07:52:22 WARNING Network is considered down Function aspt_socke Errno 0 SpecParm 10054 76 SYSTEM rmss.c 191
03262004 07:52:29 WARNING Network is considered down Function aspt_socke Errno 0 SpecParm 10054 76 SYSTEM rmsc.c 417
03262004 07:52:29 WARNING Network is considered down Function sockmanage Errno 35 SpecParm 56 76 SYSTEM rmsc.c 657
03262004 07:52:47 WARNING Error returned from OS system call Syscall aspt_socke Errno 112 ErrParm1 0 ErrParm2 328 29 SWITCH ss_rx 196
03262004 07:52:47 SYSINFO Switching shelf hardware error Shelf 1 Status 128 129 SWITCH ss_sscinit 646
03262004 07:52:48 WARNING Error returned from OS system call Syscall aspt_socke Errno 112 ErrParm1 1 ErrParm2 432 29 SWITCH ss_rx 196
03262004 07:52:48 SYSINFO Switching shelf hardware error Shelf 2 Status 128 129 SWITCH ss_sscinit 646
03262004 07:53:25 CRITICAL Voice system not up! Will retry in time VsSys 1 cRetry 2 Time 1 minute Reason No VS DCD 21 VOICE dovs_reset 215
03262004 07:55:08 SYSINFO Network connection established Remote client Port 6000 86 SYSTEM rmss.c 167
03262004 07:55:08 SYSINFO Network connection established Remote server Port 6000 86 SYSTEM rmsc.c 316
03262004 07:55:21 CRITICAL Voice system not up! Will retry in time VsSys 1 cRetry 3 Time 1 minute Reason No VS DCD 21 VOICE dovs_reset 215
03262004 07:55:31 CUSTINFO Task initialization Task Cnctd to V Code 4200 1 306 44 VOICE vs_rx 158
03262004 07:57:16 CUSTINFO Voice system reset successful VsSys 1 cPort 0 cDisk 0 cMsg 0 23 VOICE dovs_reset 0
03262004 07:57:16 CUSTINFO Voice system disk usage VsSys 1 BlkTotal 0 BlksUsed 0 Reason 0 16 VOICE dovs_reset 156
03262004 07:57:16 CUSTINFO Voice system message inventory VsSys 1 TotalRcd 0 20 VOICE dovs_reset 164
03262004 08:00:01 SYSINFO Background procedure activity(Begin/Comp Action BEGIN ProcNum 2 PID 2 ADMIN ExecuteProcedu 1432
03262004 08:00:01 SYSINFO Background procedure activity(Begin/Comp Action BEGIN ProcNum 3 PID 2 ADMIN ExecuteProcedu 1432
03262004 08:00:01 SYSINFO Background procedure activity(Begin/Comp Action BEGIN ProcNum 5 PID 2 ADMIN ExecuteProcedu 1432
03262004 08:00:01 SYSINFO Background procedure activity(Begin/Comp Action FINISHED ProcNum 5 PID 2 ADMIN ExecuteProcedu 1432
03262004 08:00:02 SYSINFO Background procedure activity(Begin/Comp Action FINISHED ProcNum 2 PID 2 ADMIN ExecuteProcedu 1432
03262004 08:00:02 SYSINFO Background procedure activity(Begin/Comp Action FINISHED ProcNum 3 PID 2 ADMIN ExecuteProcedu 1432

We'd appreciate it very much if we can have a helping hand in interpreting the error trail.

 

[databit]

Looks like an SSC problem and if you have already replaced that it could possibly be a shelf problem. If you are still having the problem post a hardware config of you system. With card and slot numbers for both shelves. Are you having any other symptoms like a all telesets from a specific TIC not getting dialtone when pressing one of the line buttons? This could signify a bad DTIC on the same shelf and I have seen it cause the SSCs to not switch properly.

 

[databit]

Just talked to a friend of mine and he said that there is a cable that goes to a daughter board which tells the system when it is cutting to backup. Might want to check that too

 

[techom]

Thank you Databit for your valuable contribution. Are you talking of the paddleboard connected to every switching shelf controller module (there are four SSC modules in the system)?

 

[databit]

There is a 5 volt cable that connects to all shelves. This cable goes to a daughter board that controls cut to backups for the whole ACD. The utilitiy card tells the SIM that its cutting and the sim sends a 5 volt burts to each shelf when the SSC on the backup takes control of ongoing calls. If that burst isn't happening or is having problems then it could drop calls that are currently in process. Is this only dropping all current calls then the system immediately resumes taking new incoming calls?