Finding the BEST engineer.....G3r 9

[TheCardMan]

I need to hire an Engineer who has real deep experience with G3r issues. I am battling Sync issues and it is getting worse as time goes on. I can not get support form local partners and AVAYA has not been returning phone calls. Long story.... Anyone know of where I can find this expertise that will be able to come on site for a day and look into my issues? Willing to pay T&M while onsite. site is in the Orange County, NY area.

 

[mikeydidit]

Unfortunately this is one of those that you need the MSS-NET, disk and Optical disk to work to save translations.

The G3R the save translations command takes the translations from the memory boards and saves then to the hard disk. The backup disk takes them off the hard drive saves them to the optical disk. If you do not have them stored to your optical disk you could loose god knows how many weeks of translations in you switch.

So this is something we are going to have to be sure about and take our time to work through. I lost 1 weeks worth of translations once and it took 3 weeks to fix them.

Hell, there are no rules here - we're trying to accomplish something.
Thomas A. Edison

For the best response to a question, read faq690-6594

 

[TheCardMan]

Understood - I believe the last sucessful backup we have is from April of 2006 and we have done many many changes since.

 

[rgtrgt]

i wasnt here until 01

 

[TheCardMan]

OK - I am going to list a typical day of issues starting with what has happened so far today.....

At 10:25am - We had sporadic internal phone issues, can't call VM, can't call extensions, trouble dialing out and callers can not call in on our main number (ring no answer).

At 10:53am - Same issues as above, + calls stuck in queue

At 11:21am - Same as above + calls stuck in queue

At 11:42am - Same as above + calls stuck in queue

At 1:40pm - Same as above but much more noticable + calls stuck in queue

At 1:50pm - Same as above

Lost about 50 to 75 calls due to abandons and who know how many could not call into our center.

This seems to be happening more and more on a daily bases.

Snap shot of recent li mease ds1 log:

list measurements ds1 log 4a12                                         Page   1

Switch Name: Banker Warranty Grp              Date: 2:08 pm  THU OCT 11, 2007

             DS-1 LINK PERFORMANCE MEASUREMENTS DETAILED LOG REPORT

                   Counted Since: 5:39 pm  THU APR 6, 2006

        Date   Time        ES      BES     SES     UAS/FS  CSS     LOFC

        10/11  14:04       0       0       0       0       0       0
        10/11  13:49       92      0       0       0       92      0
        10/11  13:34       62      0       0       0       61      0
        10/11  13:19       0       0       0       0       0       0
        10/11  13:04       0       0       0       0       0       0
        10/11  12:49       0       0       0       0       0       0
        10/11  12:34       0       0       0       0       0       0
        10/11  12:19       0       0       0       0       0       0
        10/11  12:04       0       0       0       0       0       0
        10/11  11:49       0       0       0       0       0       0
        10/11  11:34       6       0       0       0       6       0
        10/11  11:19       33      0       0       0       33      0
        10/11  11:04       14      0       0       0       14      0
        10/11  10:49       0       0       0       0       0       0
        10/11  10:34       0       0       0       0       0       0
        10/11  10:19       4       0       0       0       4       0
        10/11  10:04       0       0       0       0       0       0
        10/11  09:49       0       0       0       0       0       0
        10/11  09:34       0       0       0       0       0       0
        10/11  09:19       0       0       0       0       0       0
        10/11  09:04       0       0       0       0       0       0
        10/11  08:49       0       0       0       0       0       0
        10/11  08:34       0       0       0       0       0       0
        10/11  08:20       0       0       0       0       0       0
        10/11  08:05       0       0       0       0       0       0
        10/11  07:50       0       0       0       0       0       0
        10/11  07:35       0       0       0       0       0       0
        10/11  07:20       0       0       0       0       0       0
        10/11  07:05       0       0       0       0       0       0
        10/11  06:50       0       0       0       0       0       0
        10/11  06:35       0       0       0       0       0       0
        10/11  06:20       0       0       0       0       0       0
        10/11  06:05       0       0       0       0       0       0

 

[rgtrgt]

your circuit is taking errors, you can call the vendor for those, as far as the SNI alarms, we still need to look at those, give us a lis con car 1e

 

[TheCardMan]

No, the circuits are fine - If you look at the measurments on ALL the ds1 boards, they have all been taken those error seconds at the same time and we have 3 different carriers, plus in different locations. So the errors are internal faults. Once you see the errors on one you know it is on all the others if you list them.

The 'li config car 1e' is pasted above. let me know if you need more info.

 

[rgtrgt]

do you have cards in 1e01, 03 & 04

 

[TheCardMan]

Not anymore - they were removed because of non use. I have them in the slots but not plugged in all the way. Do I need to take thier translations out somehow?

 

[4merAvaya]

When you removed the Switch Node Interface boards did you remove all of the associated translations (fibers, cabinets, etc.)? Your switch could be going insane looking for hardware that has been removed.

Kevin

 

[TheCardMan]

Yes - from what I see, all cabinets have been removed from translations. And Fiber Links removed. Now, since I removed those 3 cards and they still show up in "li config car 1e" but as "no board", does this tell me they have to be removed from software somewhere else where I am not looking?

 

[4merAvaya]

Try doing 'change circuit-pack' and go to the 'E' carrier and see if the switch will let you remove them.

Kevin

 

[TheCardMan]

HOLY COW - I think we are on to something here. I did that and all of a sudden the major alarms started to dwindle down. From a steady 11 Majors to only 1 now. All the SNI-PEER alarms are gone.
See below:

                                 ALARM REPORT

Port       Maintenance On   Alt     Alarm   Svc    Ack?  Date        Date
           Name        Brd? Name    Type    State  1 2   Alarmed     Resolved

           DLY-MTCE    n            MAJOR                07/25/01:05 00/00/00:00
02A07      ANN-BD      y            MINOR                12/13/14:01 00/00/00:00
02A07      ANN-BD      y            MINOR                12/13/14:01 00/00/00:00
02A07      ANN-BD      y            MINOR                12/13/14:21 00/00/00:00
02         POWER       y            MINOR                07/21/17:46 00/00/00:00

 

[TheCardMan]

Now to find the DLY-MTCE culprit.....

 

[rgtrgt]

the DLY-MTCE is because you cant save translations. also, if you dont have a card in slot 2a07, you can do change circuit pack for those as well.

 

[4merAvaya]

It you're having issues with the MO drive and hard drive - look no further. The system does a 'save translations' as part of the daily maintenance tasks. Sounds like you're not able to write to one of those drives.

Do 'clear errors' to start with a fresh slate. Let the system cook for an hour or two then do 'disp errors'. Change the Error List field to 'errors' and see what you get.

Kevin

Suggestion - Can you start a new post - There are almost 100 responses here with to much data.

 

[nohuhu]

TheCardMan,

just my 2 cent before you going to struggle with MSS-NET, use site administration to pull as much translation data as you can. i know it will be PITA but having this data as text reports is WAY better than just lose it, believe me. at least you can export (and then import if need be) all the major pieces like extensions, ars, vdns, vectors and so on.
i had an experience like that once when mss-net gone crazy and left me without translation backup... i wasn't so wise then and didn't do "report backup". well, lost a month re-creating everything and customer was angry as hell.

 

[Mitch672]

cardman,

AVAYA's "ProVision" can capture your entire translations, it can be run via the dialup modem connection. It sounds like it would take almost a day to capture your translations, because of the size of your switch.

Then these can be modified, of pushed to a new switch. ANd you would have a backup that is in "computer" usable form.

ProVision is $1350 every 3 months, for the license to use it. I have access to a license, and its installed on my work PC. contact me if you want your switch backed up this way.

This is what would have to be done to your switch if you where getting it upgraded to an S8720, for example, anyway.

Mitch

 

[TheCardMan]

If I go the ASA route - I would just Export the commands I list? Such as li extension, li ars ana, li trunk, li vdn, li vectors, li announce?

Mitch - I appreciate the offer and may consider it, but I need to understand my risks in doing such.

-Joe

 

[nohuhu]

TheCardMan,

not only exports but also you can run any command you wish and store its output in a text file. basicly it can be the whole configuration stored as text, just in case. you see, there are massive things like extensions, ars/aar analysis, coverage paths, vdns/vectors and so on, and there are not-so-massive things that should be done manually anyway, i mean if you set up the whole system from scratch. cabinets, fibers, system-parameters feature-options, locations and so on. if you have such option, better go with provision. even better, go with avaya in this case, you'll need their help anyway because if you lose yor translations completely, you'll lose any optional customer options -- call center, cti, licensed ports and so on, and you can't (well, officially modify customer-options without avaya.

 

[TheCardMan]

I have held at 1 Major Alarm for the weekend, but I did see the SNI-PEER alarms sneak back when we had another glitch towards the end of the day on Friday, then they disappeared. So although we got rid of the standing 10 Majors I had for a while, it seems to be still hicupping. Mondays are usually the worst days, I will soon see once volume starts to pick up if I still have the major issues.

My concern that this will still happen is because I still see a 'mismatch' when I do a Li Fiber, see below:

                          FIBER LINK ADMINISTRATION

FIBER -------- ENDPOINT 1 --------  -------- ENDPOINT 2 --------
LINK# TYPE A-PNC DS1   B-PNC DS1    TYPE A-PNC DS1   B-PNC DS1    DS1
           LOC   CONV  LOC   CONV        LOC   CONV  LOC   CONV   CONV
                 LOC         LOC               LOC         LOC    TYPE

 1    ei   01B02                    sni  01E02

 2    sni  01E20                    ei   02A01

 4    sni  01E19 01E21              ei   04A01 04A02              TN1654
Primary Facility:A                             B                       MISMATCH