Urgent help needed - EPN removal / PNC problem 1

[mnealtx]

Hello all,

I'm trying to completely remove 2 EPNs from a location that is being taken down.

We're running into issues with attempting to remove the SNI board(s). Switch is a G3RV12, S8700, CM 2.2, single pnc, unduplicated.

All stations and station boards have been removed from both EPNs, the only cards left are the expansion interface, test/maint and call classifier boards.

Fiber link was busied out and removed, SNI card was busied out and removed. List config still shows SNI card as inserted. When attempting to remove card via 'change circuit-packs', the message returned is "cannot delete card that is physically inserted". Two other unused (and unseated) SNI cards also show as present in list config and return the same error message when attempting to remove via change circuit-packs.

I've done a lot of digging in the error logs over the last several days, and have found the following:

All SNI cards originally showed SNI-BD error 257: Cannot communicate with active SNC. I was able to clear the error by doing a reset on individual SNI cards. Two cards have not been reset and still show the 257 error - the SNI cards for the PPN cabinets. Is it safe to reset those SNI cards? Do I risk a hard crash by doing so? I realize that it will drop any calls working through the PPN.

All sys-links (stat sys location) show major SNI-BD alarm for one or the other of the PPN SNI boards.

All SNI cards show SN-CONF error 6005: Link does not exist to SNC (Aux data is "0")
Several SNI cards show SN-CONF error 6001: No neighbor link exists between SNI and neighbor, but link is administered
All SNI boards show SN-CONF error 6004 (no peer link to indicated SNI) pointing to one of the unused/unseated SNI cards (was used for a PPN link prior to conversion to S8700)

When testing the SNI cards, all fail test 779, code 112: Cannot communicate with active SNC
Testing the SNC cards, all tests abort, code 2306: Card did not respond to test requests from software.
SNC card LEDs do *NOT* light when led test is initiated.

I do *NOT* have any spare SNC cards (2 installed in slots 10 and 12, 12 active) and have only 2 unused SNI cards.

I've made these points to the servicing company and have not received any response from them - they continue to talk about clearing errors on the SNI cards for the EPN being removed. I'm convinced there is a larger issue, given the inability to remove SNI cards from software and the connection issues with the SNC.

Any help or advice is GREATLY appreciated!!

- Mike

 

[AvayaTier3]

This is not a difficult issue to resolve. If your servicing company cannot help, maybe you should escalate this
to someone that can.



A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Unfortunately, my only recourse is the servicing company (Gov't contract).

Thank you for your helpful advice and teaching me "how and where to find the answers". /s

 

[AvayaTier3]

Just so you know, if you had the xln file and load it in a lab server, you can do any translation changes that are
necessary without any hardware, save translations, move the xln file back to your production machine and reload it and it will be fixed.

media-module and circuit-pack hardware is not required to remove programming.

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Thank you, that's slightly more helpful.

Being a field tech, I don't have access to the xln file or a lab server - only the commands available available via ASA.

Is there anything else you can recommend to check? Procedures to follow in the field?

 

[AvayaTier3]

ASA and IE web browser on the network will allow backup of xln1 and xln2 and you could get the files.
I could load xln1 on s8700 server and change programming that you need to see if there may be some type problem the servicing company will not be able to do.

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Thank you for the advice and offer - I will have to run that past the supervisor, however, to see if the files could be released (security issues).

I will investigate the web browser avenue you mention - I recall seeing it before.

Is there any advice you can provide regarding how to check if the switch-node shelf is locked up, or if the switch node clock cards are actually bad?

 

[AvayaTier3]

duplicated PNC?
or
simplex with duplicated SNC

Try:

clear firmware-counters a-pnc
test switch-node-clock long
test board 1D10
test board 1D12
status switch-node
status switch-node x (1, 2, 3)
status pnc
set switch-node-clock (xD10, xD12, xE10, xE12)


A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[AvayaTier3]

start with commands above, and alarm log, and error log, and especially with a maintenance manual to decipher the error codes as well as steps for resolution.

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Simplex with duplicated SNC, if I understand you correctly. Two PPN cabinets, two S8700 servers. Two IPSI boards in each PPN cabinet. One switch-node shelf, two switch node clock cards installed.

Clear firmware a-pnc: All SNI cards physically installed pass 982, SNC cards pass 983. The two cards being 'held' in software (01E14 and 01E20) do not show up in the list but pass 982 if directly cleared ("clear firm 01e14").

Test switch-node-clock is an unavailable command (running CM 2.2, if that makes a difference)

Both SNC cards abort all tests with code 2306 - "The SNC circuit pack is not responding to test requests sent by software". Neither card lights LEDs (other than the 'active' LED on 01E12) during a 'test led switch-node-1 a-pnc' command.

status switch-node 1

SWITCH NODE STATUS

Major Minor Warning Active SNC Standby SNC
Switch Node Location Mode Alarms Alarms Alarms Location Location

1 A-PNC 01E active 10 0 0 01E12 01E10

status pnc
PORT NETWORK CONNECTIVITY

Duplicated? no
Software Locked?
Standby Busied?
Direct Connect? no
Standby Refreshed?
Interchange Disabled?
A-PNC B-PNC

Mode: active Mode:
State of Health: State of Health:
Inter PN Index: Inter PN Index:
Inter SN Index: Inter SN Index:
Major Alarms: 34 Major Alarms:
Minor Alarms: 1 Minor Alarms:
Warning Alarms: 5 Warning Alarms:
SN Locations: 01E SN Locations:

I recall mention in the maintenance manual to look for a 'state of health' of "functional" regarding a duplicated PNC. Does the same hold true for an un-duplicated one?

Will setting switch-node clock to the standby card interrupt service? Any danger of hard-crashing the switch?

I've been going through the alarm and error logs and working through the maintenance manual to determine issues/causes. Several resolution paths point back to the SNC, but I have not done any resolution concerning the SNC as I have no replacement cards - only what is in the switch.

Additional info: Originally, all SNI cards showed an error 257: Cannot communicate with active SNC. I was able to clear that fault through resetting the SNI cards for the various EPNs, but was unsure about resetting the SNI cards servicing the PPN cabinets so they are still in status 257. Also, issuing 'stat sys location' for the various SNI cards shows a major alarm with one or the other PPN SNI card.

Can the SNI cards serving the PPN be reset? What (if any) affect would resetting those SNI cards have on the system other than interrupting trunk calls and any calls terminating to stations served from cards in the PPN cabinet? Is there a risk of a hard crash of the switch?

I was unable to log into the IE access - not sure why. I recall logging into it to try and set the server time, but don't recall exactly *how* I did it, now.

 

[AvayaTier3]

duplicated PNC
busy pnc-standby
clear firmware-counters x-pnc (where x = standby pnc)
releash pnc-standby
reset pnc interchange (change active to standby and standby to active) not service affecting
test board xD10 xD12 xE10 xE12 long (busy board, reset board)
unplug active tn572 switches active to the other tn572 if software will not do it. This may be what
you need to do. I recommend doing this when its the standby PNC.

set switch-node-clock is not service affecting

You really should unplug all sni-boards and ds1c boards for fibers that go to EPN cabinets no longer used, then do
clear firmware-counters for both pnc. This updates knowledge of all sni-peers for all boards to know which other boards
they should be able to communicate with.

After doing all of this, you should be able to remove programming for SNI, DS1c, EI, epn hardware that has all programming removed, stations, trunks, fibers, etc.


A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

I have two SNC cards, not two PNCs.

If I busy the active SNC (01E12) to do a long test, will that affect service?

I will attempt the busy, long test, reset on the standby board (01E10).

One active SNI is connected via amphenol cable (goes to PPN Cab 06). All others are fiber connected. Unused SNI boards are physically removed, but list-config still shows them in the carrier and change circuit packs will not let me delete them, it returns "cannot delete a card that is physically inserted". It also does this for the SNI cards serving the cabinets I am trying to remove, even after the fiber link is busied/deleted and the SNI card is busied/removed.

 

[AvayaTier3]

PPN = Processor Port Network for g3r

linux platforms only have EPN cabinets

list cabinet and post here
list fiber and post here

status switch-node and post here

status health and post here

list config carrier 1E and post here

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Busied out SNC card 01e10 and tested.
test board 01e10 long clear

TEST RESULTS

Port Maintenance Name Alt. Name Test No. Result Error Code

01E10 SNC-BD 778 ABORT 2306
01E10 SNC-BD 779 ABORT 2306
01E10 SNC-BD 759 ABORT 2306
01E10 SNC-BD 777 ABORT 2306
01E10 SNC-BD 760 PASS
COUNT 1
01E10 SNC-BD ACTIVE

Attempted reset of SNC card 01e10. Reset aborted with code 2306.


list cabinet

CABINET REPORT

No Layout Rack Room Floor Building Loc A B C D E
1 5-car DRAGON 2050C 1340C 1 PN 01 PN 01 PN 01 PN 01 SN 01
2 5-car SOUTHTOWN 1 PN 02 PN 02 PN 02 PN 02 PN 02
3 stack Commo B 1 PN 03
4 stack HOSPITAL 1 PN 04
5 stack NORTHTOWN 1 PN 05
6 5-car DRAGON 2050C 1340C 1 PN 06 PN 06 PN 06 PN 06 PN 06
7 stack HOSPITAL 1 PN 07
8 5-car SSA 1 PN 08 PN 08 PN 08 PN 08 PN 08
9 stack MIDTOWN 1 PN 09

list fiber-link Page 1

FIBER LINK ADMINISTRATION

FIBER -------- ENDPOINT 1 -------- -------- ENDPOINT 2 --------
LINK# TYPE A-PNC DS1 B-PNC DS1 TYPE A-PNC DS1 B-PNC DS1 DS1
LOC CONV LOC CONV LOC CONV LOC CONV CONV
LOC LOC LOC LOC TYPE

1 sni 01E02 ei 01A01
2 sni 01E03 ei 02A01
3 sni 01E19 ei 03A01
4 sni 01E15 ei 04A01
5 sni 01E18 ei 05A01
6 sni 01E17 ei 06A01
7 sni 01E04 ei 07A01
8 sni 01E05 ei 08A01
9 sni 01E06 ei 09A01

status switch-node 1

SWITCH NODE STATUS

Major Minor Warning Active SNC Standby SNC
Switch Node Location Mode Alarms Alarms Alarms Location Location

1 A-PNC 01E active 22 0 0 01E12 01E10


status health
OCC: St: 1% Sm: 1% Cp: 6% Idl: 92%
PNC: Simple? ALARMS:Maj: 49 Min: 4 Wrn: 431
Pwr: comm Syncrimary Logins:1 BUSYOUT:Trk: 0 Stn: 54 Oth: 0

Cab EmTr Maj Min Wrn PNC Cab EmTr Maj Min Wrn PNC
1 auto- 22| 0|109 up
2 auto- 0| 2|150 up
3 auto- 0| 0| 0 up
4 auto- 0| 0| 8 up
5 auto- 0| 0| 0 up
6 auto- 0| 0| 9 up
7 auto- 0| 0| 15 up
8 auto- 0| 1| 66 up
9 auto- 0| 0| 72 up

list configuration carrier 01e

SYSTEM CONFIGURATION

Board Assigned Ports
Number Board Type Code Vintage u=unassigned t=tti p=psa

01E02 SWITCH NODE INTF TN573B 000003
01E03 SWITCH NODE INTF TN573B 000003
01E04 SWITCH NODE INTF TN573B 000003
01E05 SWITCH NODE INTF TN573B 000004
01E06 SWITCH NODE INTF TN573B 000006
01E10 SWITCH NODE CLOCK TN572 000010
01E12 SWITCH NODE CLOCK TN572 000010
01E14 SWITCH NODE INTF TN573B 000003
01E15 SWITCH NODE INTF TN573B 000006
01E17 SWITCH NODE INTF TN573B 000004
01E18 SWITCH NODE INTF TN573B 000003
01E19 SWITCH NODE INTF TN573B 000003
01E20 SWITCH NODE INTF TN573B 000006

Note: 01E14 and 01E20 are the unused cards. 01E02 is the direct amphenol connection. 01E18 and 01E19 are the two I am attempting to remove. I restored the fiber link for 01E19 in an attempt to clear errors.

Here is the config for the cabinets 03 and 05
list configuration port-network 3

SYSTEM CONFIGURATION

Board Assigned Ports
Number Board Type Code Vintage u=unassigned t=tti p=psa

03A01 EXPANSION INTRFC TN570C 000002
03A03 CALL CLASSIFIER TN744D 000001 01 02 03 04 05 06 07 08
03A04 MAINTENANCE/TEST TN771D 000007 u 02 03 04

list configuration port-network 5

SYSTEM CONFIGURATION

Board Assigned Ports
Number Board Type Code Vintage u=unassigned t=tti p=psa

05A01 EXPANSION INTRFC TN570C 000002
05A03 CALL CLASSIFIER TN744D 000001 01 02 03 04 05 06 07 08
05A04 MAINTENANCE/TEST TN771D 000007 u 02 03 04

 

[mnealtx]

Tested SNC card 01E12 without busyout:

test board 01e12 long clear

TEST RESULTS

Port Maintenance Name Alt. Name Test No. Result Error Code

01E12 SNC-BD 778 ABORT 2306
01E12 SNC-BD 779 ABORT 2306
01E12 SNC-BD 759 ABORT 2306
01E12 SNC-BD 777 ABORT 2306
01E12 SNC-BD 760 PASS
COUNT 1
01E12 SNC-BD RESOLVED

Display errors category PNC still shows error 6005 for the SNI cards - cannot communicate to SNC.

Set switch-node clock 01E10 did *NOT* work. If I pull 01E12 to force a switchover, will it interrupt service?


I have not switched clock cards yet.

 

[mnealtx]

Disregard the last line, above - attempted set switch-node clock 01E10 and it did not work.

 

[AvayaTier3]

remove fiber 3
remove fiber 5
change circuit-pack 3 (blank out all boards)
change circuit-pack 5 (blank out all boards)

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[AvayaTier3]

remove cabinet 3 and 5
change circuit-pack 1 (remove all SNI boards that have been unplugged and fibers are gone)
clear firmware-counter a-pnc

This could drop calls:
reseat 1e10
unplug 1e12 and 1e10 should become active
plug 1e10 back in and if it swaps back to 1e12 active, then 1e10 may be bad


A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

remove fiber 3
remove fiber 5
change circuit-pack 3 (blank out all boards)
change circuit-pack 5 (blank out all boards)

No busy out of the fiber link?

Stand by, will attempt.

 

[mnealtx]

Command: Remove fib 3
Response: Must busyout fiber before removal.

Command: busyout fib 3
Command: rem fib 3
Command: cha cir 3
Response: no cards seen after removal of fiber.

Do I physically remove the SNI card and the fiber xcvr at this point, or busyout first?