Receiving error on CS1000E Call Server

[johntel]

Hello Sir,
Receiving error on CS1000E Call Server




SRPT308 [CemuxIOLinkMgmt]: Connection closed. Return Status[-1]

MSDL300 4 0 3 FROM: ENBL TO: SYS DSBL - NOT RESPONDING
TIME: 20:07:35

SRPT016 OMM: IP link is DOWN between Call Server and IPMG[4 0]

SRPT257 shutDownServerSession: shutting down ssd server session [2] for IPMG [4 0]

ELAN009 ELAN 0 host IP=172.16.3.5 disabled, write to socket
fail due to far end disconnect or Ethernet problems

SRPT226 Registration has been granted for IP[172.16.3.5] IPMG[4 0]


SRPT017 OMM: IP link is UP between Call Server and IPMG[4 0]

XMI002 4 0 10 XMLC

XMI002 4 0 7 XMLC

XMI002 4 0 8 CMOD

XMI002 4 0 9 XMLC

XMI002 4 0 4 EXUT

XMI002 4 0 3 PRI2

DTA102 3

DTA105 3 0

MSDL300 4 0 3 FROM: SYS DSBL - NOT RESPONDING TO: SYS DSBL - SELF TESTING
TIME: 20:09:13

SRPT091 HB: Local side IPL health change from 10 to 8.

SRPT091 HB: Local side IPL health change from 8 to 10.

ELAN014 ELAN 0 host IP=172.16.3.5 is enabled

MSDL300 4 0 3 FROM: SYS DSBL - SELF

 

[bignose21]

Your Media gateway 4 0 rebooted and recovered is it still doing it or a one off?

 

[johntel]

Hello Sir,
Daily at latest one time, that time PRI goes down.

 

[42SKPK]

Hello,
I have some problem after UPG OPT11MINI R4.5 to CS1000E to Rel7. IPMG 0-1 randomly lose ELAN connectivity to the CPU ??? Last Time problem repeated 4 time over about 50 minutes, see file below.
But ELAN is taking a direct cable connection between IPMG and CP-MCS.I have repalced (before one week) IPMG,CP-MGS also ELAN patch cable to new one, but problem after a week reiterated, I do not understand??

%
% TIM1111 22:00 14/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 23:00 14/4/2011 CPU 0
%
% DROL000 DAILY ROUTINE BEGIN
% DUMP LD43 BEGIN 23:01 14/4/2011
%
% DB SEQ NUM = 3923
% CONFIG
% PHYSICAL MAP
% BCS TEMPLATE
% PBX TEMPLATE
% SUPER
% IPMG
% CUST
% ACUST
% CLID
% ROUTE
% DAPC
% LTN TN
% LTN LNK
% PGRP
% TN
% SCL
% ESN 00
% NCTL
% PREXL
% ACD
% DISA
% FRL
% NFCR TREES
% IDC TREES
% DTAD
% DIGITAL
% DTI
% ASNCH
% BG-TIME
% BG-CAT
% DCH
% PRI2
% ARIES
% FDL
% SYSP
% XTDT
% MCAD
% FCAD
% FDCT
% DTI2
% FFC
% LAPW
% MSDL/MISP BLK
% SOCKET ID BLK
% MISP
% TBAR RCDT
% TBAR ART
% TIME
% SDAC
% DTI PDCA
% PLUGIN
% STS BLK
% NODE
% CPND NM
% SPECIFIC DATA
% HI
% ALARM_MGT
% ZONE_MGT
% SYSCFG_MGT
% ACCOUNT_MGT
% ERL DB
% SUBNET DB
% ELIN DB
% BANNER_MGT
% NTP_MGT
% ZBD ZONES DB
% ZBD DB
% MSC ZSRN DB
% IPMG_MGT
% SYSTEM_PARAMS
% PORT_STATE_MGT
% DVLA DB
% HSEM DB
% CHECKING
%
% SIZE (KBYTES) = 184
% DATADUMP COMPLETE
%
%
% Backing up reten.bkp
%
% Starting database backup
% to local Removable Media Device
%
% USB mass storage device found available
% Backing up reten.bkp to "/var/opt/nortel/cs/fs/usb/backup/single"
%
% Database backup Complete!
%
% TEMU207 Backup process to local Removable Media Device ended successfully.
%
% DROL001 DAILY ROUTINE END
% DUMP LD43 END 23:02 14/4/2011
%
% DROL000 DAILY ROUTINE BEGIN
% AUDIT LD44 BEGIN 23:02 14/4/2011
%
% DROL001 DAILY ROUTINE END
% AUDIT LD44 END 23:04 14/4/2011
%
% AUD000
%
% DROL000 DAILY ROUTINE BEGIN
% CCED LD135 BEGIN 23:04 14/4/2011
%
% DROL001 DAILY ROUTINE END
% CCED LD135 END 23:04 14/4/2011
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 00:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 01:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 02:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% TIM1111 03:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 04:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 05:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 06:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% TIM1111 07:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 08:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 09:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 10:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 11:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% TIM1111 12:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 13:00 15/4/2011 CPU 0
%
% AUD000
%
% AUD000
%
% AUD000
%
% TIM1111 14:00 15/4/2011 CPU 0
%
% AUD000
%
% MSDL300 0 1 2 FROM: ENBL TO: SYS DSBL - NOT RESPONDING
% TIME: 14:12:10 <=======
% SRPT016 OMM: IP link is DOWN between Call Server and IPMG[0 1]
%
% SRPT257 shutDownServerSession: shutting down ssd server session [1] for IPMG [0 1]
%
% ELAN009 ELAN 0 host IP=10.0.0.30 disabled, write to socket
% fail due to far end disconnect or Ethernet problems
%
% SRPT226 Registration has been granted for IP[10.0.0.30] IPMG[0 1]
%
%
% SRPT017 OMM: IP link is UP between Call Server and IPMG[0 1]
%
% XMI002 0 1 1 DMC
%
% XMI002 0 1 3 XFALC
%
% XMI002 0 1 4 XDLC
%
% XMI002 0 1 5 XDLC
%
% XMI002 0 1 6 XDLC
%
% ELAN014 ELAN 0 host IP=10.0.0.30 is enabled
%
% XMI002 0 1 2 PRI2
%
% DTA102 200
%
% DTA105 200 0
%
% MSDL300 0 1 2 FROM: SYS DSBL - NOT RESPONDING TO: SYS DSBL - SELF TESTING
% TIME: 14:14:46
%
% MSDL105 MSDL 0 1 2 00000000
%
% MSDL112 MSDL 0 1 2
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS FAILED
% TIME: 14:15:28 REASON: CPU TEST
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS FAILED TO: SYS DSBL - SELF TESTING
% TIME: 14:15:28
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS PASSED
% TIME: 14:16:32
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS PASSED TO: ENBL
% TIME: 14:16:32
%
% DCH: 2 EST CONFIRM TIME: 14:17:04 15/04/2011
%
% MSDL300 0 1 2 FROM: ENBL TO: SYS DSBL - NOT RESPONDING
% TIME: 14:25:46 <========
%
% SRPT016 OMM: IP link is DOWN between Call Server and IPMG[0 1]
%
% SRPT257 shutDownServerSession: shutting down ssd server session [1] for IPMG [0 1]
%
% ELAN009 ELAN 0 host IP=10.0.0.30 disabled, write to socket
% fail due to far end disconnect or Ethernet problems
%
% SRPT226 Registration has been granted for IP[10.0.0.30] IPMG[0 1]
%
%
% SRPT017 OMM: IP link is UP between Call Server and IPMG[0 1]
%
% XMI002 0 1 1 DMC
%
% XMI002 0 1 3 XFALC
%
% XMI002 0 1 4 XDLC
%
% XMI002 0 1 5 XDLC
%
% XMI002 0 1 6 XDLC
%
% ELAN014 ELAN 0 host IP=10.0.0.30 is enabled
%
% XMI002 0 1 2 PRI2
%
% DTA102 200
%
% MSDL300 0 1 2 FROM: SYS DSBL - NOT RESPONDING TO: SYS DSBL - SELF TESTING
% TIME: 14:28:22
%
% DTA105 200 0
%
% MSDL105 MSDL 0 1 2 00000000
%
% MSDL112 MSDL 0 1 2
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS FAILED
% TIME: 14:29:04 REASON: CPU TEST
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS FAILED TO: SYS DSBL - SELF TESTING
% TIME: 14:29:04
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS PASSED
% TIME: 14:30:08
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS PASSED TO: ENBL
% TIME: 14:30:08
%
% DCH: 2 EST CONFIRM TIME: 14:30:40 15/04/2011
%
% AUD000
%
% DTA203 200 4 2
%
% DTA111 200 LS
%
% DTA203 200 4 1
%
% DCH: 2 RLS NO RESPONSE TIME: 14:35:08 15/04/2011
%
% DCH: 2 RLS NO RESPONSE TIME: 14:35:14 15/04/2011
%
% DTA106 200
%
% DTA203 200 4 0
%
% DTA203 200 0 2
%
% DTA111 200 RA
%
% DTA203 200 0 1
%
% DCH: 2 EST REMOTE TIME: 14:35:30 15/04/2011
%
% DTA106 200
%
% DTA203 200 0 0
%
% MSDL300 0 1 2 FROM: ENBL TO: SYS DSBL - NOT RESPONDING
% TIME: 14:42:00 <===========
%
% SRPT016 OMM: IP link is DOWN between Call Server and IPMG[0 1]
%
% SRPT257 shutDownServerSession: shutting down ssd server session [1] for IPMG [0 1]
%
% ELAN009 ELAN 0 host IP=10.0.0.30 disabled, write to socket
% fail due to far end disconnect or Ethernet problems
%
% SEC054 A device has connected to, or disconnected from, a pseudo tty without authentica
% ting
%
% SRPT226 Registration has been granted for IP[10.0.0.30] IPMG[0 1]
%
%
% SRPT017 OMM: IP link is UP between Call Server and IPMG[0 1]
%
% XMI002 0 1 1 DMC
%
% XMI002 0 1 3 XFALC
%
% XMI002 0 1 4 XDLC
%
% XMI002 0 1 5 XDLC
%
% XMI002 0 1 6 XDLC
%
% TTY #14 LOGGED IN ADMIN 14:43 15/4/2011
%
% SEC116 User ADMIN has been allowed to log in using an emergency account.
%
% TTY #14 LD 022 ADMIN 14:43:52 15/4/2011
%
% XMI002 0 1 2 PRI2
%
% ELAN014 ELAN 0 host IP=10.0.0.30 is enabled
%
% DTA102 200
%
% MSDL300 0 1 2 FROM: SYS DSBL - NOT RESPONDING TO: SYS DSBL - SELF TESTING
% TIME: 14:44:36
%
% DTA105 200 0
%
% MSDL105 MSDL 0 1 2 00000000
%
% MSDL112 MSDL 0 1 2
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS FAILED
% TIME: 14:45:20 REASON: CPU TEST
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS FAILED TO: SYS DSBL - SELF TESTING
% TIME: 14:45:20
%
% TTY #14 LD 096 ADMIN 14:45:44 15/4/2011
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS PASSED
% TIME: 14:46:24
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS PASSED TO: ENBL
% TIME: 14:46:24
%
% DCH: 2 EST CONFIRM TIME: 14:46:56 15/04/2011
%
% TTY #14 LD 060 ADMIN 14:47:06 15/4/2011
%
% TTY #14 LD 096 ADMIN 14:48:08 15/4/2011
%
% TTY #14 LD 022 ADMIN 14:48:18 15/4/2011
%
% TTY #14 LD 032 ADMIN 14:52:34 15/4/2011
%
% TIM1111 14:52 15/4/2011 CPU 0
%
% DTA203 200 4 2
%
% DTA111 200 LS
%
% DTA203 200 4 1
%
% DTA106 200
%
% DTA203 200 4 0
%
% TTY #14 LOGGED OUT ADMIN 14:54 15/4/2011
% SESSION DURATION: 00:11
%
% MSDL300 0 1 2 FROM: ENBL TO: SYS DSBL - NOT RESPONDING
% TIME: 14:55:32 <===========
%
% AUD000
%
% SRPT016 OMM: IP link is DOWN between Call Server and IPMG[0 1]
%
% SRPT257 shutDownServerSession: shutting down ssd server session [1] for IPMG [0 1]
%
% ELAN009 ELAN 0 host IP=10.0.0.30 disabled, write to socket
% fail due to far end disconnect or Ethernet problems
%
% SRPT226 Registration has been granted for IP[10.0.0.30] IPMG[0 1]
%
%
% SRPT017 OMM: IP link is UP between Call Server and IPMG[0 1]
%
% XMI002 0 1 1 DMC
%
% XMI002 0 1 3 XFALC
%
% XMI002 0 1 4 XDLC
%
% XMI002 0 1 5 XDLC
%
% XMI002 0 1 6 XDLC
%
% ELAN014 ELAN 0 host IP=10.0.0.30 is enabled
%
% XMI002 0 1 2 PRI2
%
% DTA102 200
%
% DTA105 200 0
%
% MSDL300 0 1 2 FROM: SYS DSBL - NOT RESPONDING TO: SYS DSBL - SELF TESTING
% TIME: 14:58:10
%
% MSDL105 MSDL 0 1 2 00000000
%
% MSDL112 MSDL 0 1 2
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS FAILED
% TIME: 14:58:50 REASON: CPU TEST
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS FAILED TO: SYS DSBL - SELF TESTING
% TIME: 14:58:50
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELF TESTING TO: SYS DSBL - SELFTESTS PASSED
% TIME: 14:59:54
%
% MSDL300 0 1 2 FROM: SYS DSBL - SELFTESTS PASSED TO: ENBL
% TIME: 14:59:54
%
% TIM1111 15:00 15/4/2011 CPU 0
%
% DCH: 2 EST CONFIRM TIME: 15:00:26 15/04/2011
%
% AUD000
%
% AUD000
%
% SEC054 A device has connected to, or disconnected from, a pseudo tty without authentica
% ting
%
% TTY #14 LOGGED IN ADMIN 15:56 15/4/2011
%
% TTY #14 LD 022 ADMIN 15:56:52 15/4/2011


Have You any ideas?

I'm thinking about replace the chassis in which situated IPMG 0-1 card ...I fighting with this problem almost a two month, I´m already desperate...It seems as if someone turned off the power for IPMG 0-1 chassis,but it is not possible because access to the server room is controlled.

 

[Tman45]

Make sure you are on the latest patches - some patches also may cause a reboot so you need to be careful.

We found that some port scan was running that was messing with our CS1000E so also ensure that no scanning is reaching your ELAN. ELAN is very sensitive and does not do well with any non-Nortel data traffic violating it. I would triple check this - or a really good way to MAKE SURE is to put a sniffer on the ELAN and review packets. That is how I found our scanning issue. It would cause a reboot just about every time our company was running it (without informing the department of a scan).

 

[42SKPK]

Hi Tman45, thaks for update, my CS1000E PBX consist only from two chassis. IPMG[0-0] has CP-MGS Card and IPMG[0-1] has MGC Card. ELAN is completely abandoned, it consists only from one patch cord pulled from CP-MGS card to port E1 on MGC. Today I uploaded the new LW for DSP, we'll see what it will bring the coming days ... whether the system more stable ...

 

[bignose21]

The first time you posted the failure it was MGC 4 0 the second time it was MGC 0 1. I would double check your config.

Print the Superloops in LD 97 and write down all the ELAN addresses of the MGC's.

In elephant Manager open each MGC config and write down the ELANS and TLANs of MGC's and DSPs double check the Masks, Gateways and the Callserver address.

Edit the Node and write down the ELANs and TLANs for Sig Servers and any Media Cards (If still used).

LD 48 stat ELAN for any applications eg Callpilot.

I would hazard a guess that you will find a clash