Urgent help needed - EPN removal / PNC problem 1

[mnealtx]

Hello all,

I'm trying to completely remove 2 EPNs from a location that is being taken down.

We're running into issues with attempting to remove the SNI board(s). Switch is a G3RV12, S8700, CM 2.2, single pnc, unduplicated.

All stations and station boards have been removed from both EPNs, the only cards left are the expansion interface, test/maint and call classifier boards.

Fiber link was busied out and removed, SNI card was busied out and removed. List config still shows SNI card as inserted. When attempting to remove card via 'change circuit-packs', the message returned is "cannot delete card that is physically inserted". Two other unused (and unseated) SNI cards also show as present in list config and return the same error message when attempting to remove via change circuit-packs.

I've done a lot of digging in the error logs over the last several days, and have found the following:

All SNI cards originally showed SNI-BD error 257: Cannot communicate with active SNC. I was able to clear the error by doing a reset on individual SNI cards. Two cards have not been reset and still show the 257 error - the SNI cards for the PPN cabinets. Is it safe to reset those SNI cards? Do I risk a hard crash by doing so? I realize that it will drop any calls working through the PPN.

All sys-links (stat sys location) show major SNI-BD alarm for one or the other of the PPN SNI boards.

All SNI cards show SN-CONF error 6005: Link does not exist to SNC (Aux data is "0")
Several SNI cards show SN-CONF error 6001: No neighbor link exists between SNI and neighbor, but link is administered
All SNI boards show SN-CONF error 6004 (no peer link to indicated SNI) pointing to one of the unused/unseated SNI cards (was used for a PPN link prior to conversion to S8700)

When testing the SNI cards, all fail test 779, code 112: Cannot communicate with active SNC
Testing the SNC cards, all tests abort, code 2306: Card did not respond to test requests from software.
SNC card LEDs do *NOT* light when led test is initiated.

I do *NOT* have any spare SNC cards (2 installed in slots 10 and 12, 12 active) and have only 2 unused SNI cards.

I've made these points to the servicing company and have not received any response from them - they continue to talk about clearing errors on the SNI cards for the EPN being removed. I'm convinced there is a larger issue, given the inability to remove SNI cards from software and the connection issues with the SNC.

Any help or advice is GREATLY appreciated!!

- Mike

 

[mnealtx]

Pulled SNC 01e12. Active LED now lit on 01E10.

Re-inserted 01E12, 01E10 remains active.

 

[AvayaTier3]

remove cabinet 3 and 5
remove sni boards and transceivers not used
change circuit-pack 1 and blank the tn573 that have no hardware
clear firmware-counter a-pnc
save trans

you should be able to set switch-node-clock 1D10 or 1D12 now without errors and light should be on the tn752 that is active.


A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

IT WORKED!!!

Attempted removal of cards after reset of SNC.

Unseated 01E19, was able to remove in cha cir 1

Busyout/remove fib 5
Unseated 01E18, removed in cha cir 1
Seated 01E14 and 01E20, then removed. Was able to remove 01E14 and 01E20 in cha cir 1

Was able to switch clock back to 01E12.
SNC cards now answer led test command.

Error status:
All SNI cards show SN-CONF error 6005, aux data 0 - Error Type 6005: a link does not exist to one of the SNCs. If the Aux Data is not 1 or 2, the link from this SNI to the active SNC does not exist. If the Aux Data is 1, the link from this SNC to the active SNC does not exist. If the Aux Data is 2, the link from this SNC to the standby SNC does not exist.

To resolve this error, refer to FAIL code 112 for the Configuration Audit test (#759). <--- is this for the SNC board or the SNI board?

SNI 04, 06 and 15 show SN-CONF error 6001, aux data 0: Error Type 6001: no neighbor link exists between the SNI and its neighbor, but a neighbor link (that is, fiber link) is administered. To resolve this error, refer to FAIL code 135 for test 759."

All SNI boards show SN-CONF error 6004, aux data 20: Error Type 6004: a peer link does not exist to another equipped SNI. The Aux Data equals the slot number of the SNI to which this SNI or SNC does not have a peer link. To resolve this error, refer to the FAIL code for test 759 corresponding to the Aux Data on the following table:

Will attempt to re-test of all SNI boards to attempt to clear errors.

 

[AvayaTier3]

clear firmware-counter a-pnc
this will fix the sn-conf errors


A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Tier 3, thank you SO much for your help!!


Test alarm long clear, category pnc cleared most errors.

Remaining:

01E02: SN-CONF error 6005, aux data 0
SN-CONF error 6004, aux data 20
SNI-PEER error 3585 (points to removed SNI 19)

Problem: SNI 01E02 serves cabinet 1, which is an IPSI connected EPN. What effect will a reset or reseating of the SNI board have on the cabinet or the switch as a whole?

 

[AvayaTier3]

the problem is not 1E02, it's 1E19
If you removed 1E19 board and blanked it out in change circuit-pack 1, then do clear firmware a-pnc, all errors should clear.

you have removed fiber 3 and 5, cabinets 3 and 5, correct
you have blanked out all sni boards that do not have hardware using change circuit-pack 1



A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[AvayaTier3]

unplugging 1E02 will drop all voice connectivity over all fibers to all cabinets.

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Correct.

I ran test long against 01E02. It aborted test 989 with code 1415 due to no fiber xcvr (direct amphenol cable connection). It also failed 777, I'm presuming also due to the lack of fiber xcvr.

Cleared firmware 01E02 and ran display errors again for category PNC and the only faults remaining are for EPN 04.
04A01 EXP-INTF 2049 117
04A01 EXP-INTF 3076
04A01 EXP-INTF 257 84
04A01 EXP-INTF 2305 18
04A01 EXP-INTF 1 0


Fiber 3 and 5 are removed.
Cabinets 3 and 5 are removed.
Cards 01e14, 01e18, 01e19 and 01e20 are removed from the carrier and from cha circuit-packs 1.
List config car 01E no longer shows those cards present.

 

[AvayaTier3]

good work Mike

Maintenance book will tell you what to do to resolve the few remaining errors.

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

40 years Bell, AT&T, Lucent, Avaya
Tier 3 for 30 years and counting
[URL unfurl="true"]http://bshtele.com[/url]

 

[mnealtx]

Thanks again for your help.

I'm going to try to knock out that EPN next week, and then work through the UDS1 errors in the full log.