Contact US

Question

500v2 8.0.44 reboot issue

Signo (IS/IT--Management)

(OP)

6 Aug 12 09:29

Hello everyone,

I have a system that has been in for about 3 months an has had constant issues. The issue I have is that the customer calls and says that all the phones are stuck, with either a "blank white" or I have seen them stuck on please wait or one moment please cant remember exactly. I have noticed that it seems like a power issue. The system is on a battery backup, but it does not seem to matter. Its like it rebooted and got stuck. I can pull the battery off the outlet and the system stays up and running fine.

The commercial power does go out regularly, and this only happens occasionally but the customer is naturally upset I have been out multiple times on a new system.

I think it is power related, but the system cant recover correctly from losing power. If you guys have some ideas I would love to hear them.

Thanks

7deedtz (TechnicalUser) · Answer 1 · 2012-08-06 09:40:14

What else is connected to the battery back-up source? Does this happen also when running on the commercial power outlet or only during power outages?

Mitel, Avaya, NEC, INDEX
www.vcs-sa.co.za

tlpeter (Programmer) · Answer 2 · 2012-08-06 09:46:18

When it is a power issue then try to swap the IP500 and see if it fixes the problem.

BAZINGA!

I'm not insane, my mother had me tested!

Signo (IS/IT--Management) · Answer 3 · 2012-08-06 09:55:32

7deedtz,

The IP office is the only thing on the battery all other equipment is on a separate backup. I have never had the system on commercial outlet without battery, and I think it has only happens during storms with power surging or out. But I don't live close enough to know it it is storming there for sure when it happens and the night shift workers are not there when I am to ask.

I have swapped IP office chassis and the problem persists. I was thinking about changing the battery backup, its new but that means nothing these days.

It just seems to me like the phones cant recover from power failure. If you call the auto attendant it will answer and you can call an extension with a blank screen and it will ring you just cant tell whats going on.

Telecomboy (Programmer) · Answer 4 · 2012-08-06 09:57:08

If the system is set-up to be able to send emails, set yourself up to receive alarms emailed to you. If the power goes down, the system will send you an email and you know it lost power.

7deedtz (TechnicalUser) · Answer 5 · 2012-08-06 10:24:13

I will recommend swapping out the battery power pack, seems it is the only common factor so far since you said you changed the IPO chassis. Good lucky

Mitel, Avaya, NEC, INDEX
www.vcs-sa.co.za

Signo (IS/IT--Management) · Answer 6 · 2012-08-06 12:42:22

I have a new pack in hand and am going to change it when they close for the day. I am in the mid-Atlantic area and it storms at least once a week this time of year. It wont take me long to know if it did or didn't work.

Thanks guys I will update when I know the outcome.

Signo (IS/IT--Management) · Answer 7 · 2012-08-10 09:13:43

OK I replaced the battery the other day and had a storm this morning. I turned on the email notification and it tells me a lot. The system never lost power, but I got email notifications about most if not all of the stations being unplugged from the system.

The customer called and said they were just sitting in the office and all the phones rebooted and then sat for a minute or two then came back online. The system has two combo cards and a digital station 8 card. Its almost like all the cards reset for some reason, but I did not receive and email about it.

I have surge protection modules on all the lines and any station that leaves to small building they have offices in that is not attached to the main building. The protection is edco / emerson which we have used for years on our Nortel stuff. I am wondering it they are not sufficient or compatible with the IP Office in some way. What products do you guys use to protect lines and stations?

I am really grasping here for ideas or reasons.

Telecomboy (Programmer) · Answer 8 · 2012-08-10 09:33:08

Are the phones that "unplugged" on the same cards or are they all over the board? Are all the phones that are in the small office on the same card or are they on separate cards?

tlpeter (Programmer) · Answer 9 · 2012-08-10 09:49:10

Run monitor for a while and let the customer noting the times.
You will see a power failure or a lost connection to monitor.

BAZINGA!

I'm not insane, my mother had me tested!

Signo (IS/IT--Management) · Answer 10 · 2012-08-10 09:51:00

The small external building is spread across the two combo cards. The digital station 8 only has to user on it and they are both in the same building as the IP office. All the phones that unplugged were all over the board in this case. Even phones that were not on cards with stations that leave the building.

I have been looking over every part of the install and just found that the chassis ground on the IP office is not hooked up at all. I would not think that is enough to cause these kind of problems, but I am trying to figure out how to get from the rack mount to a proper ground, which does not look easy and why I am guessing its not done. Have to discuss that with the installer later.

Telecomboy (Programmer) · Answer 11 · 2012-08-10 09:57:45

I would definitely get it grounded since they have phones in other buildings and storm a lot. You would've received a reboot email if the system lost power so to me it seems like something is zapping the cards without causing the system to lose power (which grounding the system would help).

Signo (IS/IT--Management) · Answer 12 · 2012-08-10 10:00:46

Yeah my jaw dropped to the ground when I found it not grounded. I was shocked, but every line does have grounding before the system too. Still no excuse not to ground it. Looks like I am heading into a rain filled crawl space.

amriddle01 (Programmer) · Answer 13 · 2012-08-10 10:02:46

Indeed, also grounding on the IP Office is a functional requirement as well as a protective/safety measure

NYSTECH (Vendor) · Answer 14 · 2012-08-10 10:47:40

If you have replaced the IP500 already. And have grounded the system and it still happens it is likely either one of the cards in system causing it or one of the ports on a card.. I have had 1 bad set bring down an entire system before

Signo (IS/IT--Management) · Answer 15 · 2012-08-10 18:49:34

NYSTECH

This all started when this site was hit by a major lightning strike, it took out multiple pc's, routers, most of the wiring in the building was hosed and needed replaced and a bunch of manufacturing equipment was burned up. The IP Office lost all its cards which were replaced. In the process of investigating this phone rebooting issue we have replaced the chassis, and the ground is now fixed we think. The building has had multiple additions and as we looked over all the wiring we are not sure the electrical grounds are proper, we grounded to what should be a proper ground, but it's in question as to the quality of the ground.

The bad phone makes me wonder since there was so much damage its vary possible that there is a bad phone, but its only happening when there is bad weather in the area. There is one phone that got stuck on the wait screen, might be the issue. How did you find one bad phone by the way?

Going to see how it goes tonight when the second wave of storms moves by. I have monitor setup on a laptop on site with email notification so that should give me a time frame to look at in manager when I get the emails.

I have never had so much trouble out of a single site. Their IT guy and I have got to know each other we are both there so much lol.

NYSTECH (Vendor) · Answer 16 · 2012-08-11 09:02:07

By using the Monitor program.. Just before the system would reboot I would get some wierd message about the same extension over and over then Monitor would show contact lost. I unplug the phone and system stayed up..

Do you have any IP phones ? Just curious

Signo (IS/IT--Management) · Answer 17 · 2012-08-13 08:35:59

Yes I have 3 IP phones that are remote through hardware VPN. I had a little bit of a storm roll through and the system stayed up and running and no phones dropped. I did have a weird thing on Sunday, the lan showed that it went up and down a few times. Not too worried about that though, it could have been there IT guy doing something in the off hours.

It looks like the ground really helped, I am not completely happy with the ground I am using, but it seems to be working. The customer is going to need an electrician to get the grounding system in the building up to code. With multiple additions to the building over the years some things are not right.

On a side not what do you guys use for time servers, I have been using the US nist servers but have been getting a lot of failures to reach them.

amriddle01 (Programmer) · Answer 18 · 2012-08-13 08:59:26

I have used a few time servers in the past, but I always get system errors that the time server was unreacheable. So now I always use the same one as it seems to be a system issue as oppose to the time servers themselves, I now just ignore that time server error

Signo (IS/IT--Management) · Answer 19 · 2012-08-13 13:27:09

Thanks. I have two systems that give the error, and the time is always right so it makes sense it just a bogus alarm.

Signo (IS/IT--Management) · Answer 20 · 2012-08-14 08:58:41

So My customer called me this morning with all the phones stuck on please wait again. Had a little bit of rain and a crack of thunder or two nothing I normally would worry about.

All 9508 were stuck on please wait and the system never seemed to lose power, I am going to go read the monitor trace I had running, hopefully it caught something.

I am going to put on a 1416 and see if it dies out when the 9508's get stuck or what it behavior is when this happens.

I have never seen anything in 15 years of this kind of work that is this sensitive.

Anyone think it just the 9508’s I know there have been a bunch of issue’s with them, anyone think this is just another one? All the lockup’s that I see posted are random phones not all at one time, so it’s a different scenario.

Any ideas on this one?

NYSTECH (Vendor) · Answer 21 · 2012-08-14 14:39:26

Post the trace on here.. Someone should be able to read it

Signo (IS/IT--Management) · Answer 22 · 2012-08-14 15:01:34

I have to go and grab the logs later today. I let it run a little longer today as I expect another storm. Monitor did not show anything really. One of the employees was on the phone when it quit. She told me she heard a crack of thunder then her call was gone and the phones were stuck on please wait. I can see the call end and none of the phones show disconnecting but they are all stuck on please wait. Its like the phones reset but the system never detected it, so it would not let them reconnect. I put a 1416 out there today so see how it acts. I will grab the trace on the way home today and post it.

Today we ran a complete new ground back to the main panel because we could not see the ground the whole way through the attics. Now we know it is bonded directly to the power ground.

We have lost so much on this job its become a office joke that I am going to be getting a desk there.

Telecomboy (Programmer) · Answer 23 · 2012-08-14 16:46:02

I would bet if the employee heard a crack of thunder before the "please wait" that lightening hit the wiring going between the buildings causing it to zap the system. It may not have lost power, but could cause the phones to reset.

amriddle01 (Programmer) · Answer 24 · 2012-08-14 16:56:11

Even within a few hundred yards and longish cable runs it can induce hundreds of volts into the cable

Telecomboy (Programmer) · Answer 25 · 2012-08-14 17:07:29

How many phones are in the remote building? I know it's not convenient, but is it possible to unplug those phones for a few days and see if the issue happens with that wiring taken out of the mix? Easier said than done of course, but it could narrow down your culprit pretty quickly.

Signo (IS/IT--Management) · Answer 26 · 2012-08-14 17:11:33

That's what we think. You can't stop lightning, no matter how hard you try. But it's constant, and the Nortel it replaced never had this much trouble, but they were hardy suckers. Same wiring just a direct swap and no other systems in the area suffer this problem.

I know each case if different but every storm makes me think something specific is drawing it towards the system. Going to check wiring all over tomorrow. Maybe something is laying on something electrical.

It's got me pulling my hair out, lucky the customer is being understanding

Signo (IS/IT--Management) · Answer 27 · 2012-08-14 17:17:09

I may isolate that building over the weekend if the weather is going to be bad. We were thinking the same thing, process of elimination is all we can do.

Signo (IS/IT--Management) · Answer 28 · 2012-08-14 21:40:02

Here is the log for the system when it quit. I removed some customer info and cut the two log files into one. There is a call active around 5 minutes before 8am and it shows the call as active but the customer said the phones all went please wait while on this call. In that time frame I don't see anything in the log that shows the phones quieting the only things I see is the system restarting after we instructed the customer to pull the power and plug back in. Its a lot of info and I have never really read these.

If anyone knows more or sees anything please chime in.

http://www.mediafire.com/view/?3b00be3169ihbbq

qtelcom (Vendor) · Answer 29 · 2012-08-15 00:48:27

I'm gonna go with electrical ghost in the machine as the reason on this one. There should be all kinds of protection setup on the cabling for the remote building extn's on the digital and/or analog stations on both sides of the termination with both primary and secondary protection on the wiring and grounded properly. All that protection would've needed to be replaced if the building took a direct shot of lighting that fried everything. I'm assuming this has all been done? You could look at putting a zero surge on the IP office proc. (expensive, but it's good for stormy areas with sensitive equipment) Then again, it could be a ANOTHER discovered 9508 'feature' from Avaya. What release and firmware is the IP500 and the 9508 using? Good luck! :)

tlpeter (Programmer) · Answer 30 · 2012-08-15 01:33:40

It is clear to me:

CODE

********** SysMonitor v10.0 (44) [connected to 10.0.0.204 (Sample)] **********
 487676749mS PRN: Monitor Status  IP 500 V2 8.0(44)
 487676749mS PRN: LAW=U PRI=0, BRI=0, ALOG=8, ADSL=0 VCOMP=20, MDM=0, WAN=0, MODU=0 LANM=0 CkSRC=0 VMAIL=1(VER=2 TYP=3) CALLS=0(TOT=957)
 487794641mS RES: Tue 14/8/2012 08:17:15 FreeMem=61424224(2) CMMsg=6 (8) Buff=5200 953 1000 7463 5 Links=4297
 487794642mS RES2: IP 500 V2 8.0(44) Tasks=43 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=45 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=1 SSA=0 TCP=14 TAPI=0 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
 487800641mS RES: Tue 14/8/2012 08:17:21 FreeMem=61425088(2) CMMsg=6 (8) Buff=5200 954 1000 7463 5 Links=4299
 487800641mS RES2: IP 500 V2 8.0(44) Tasks=43 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=46 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=1 SSA=0 TCP=14 TAPI=0 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
 488097141mS RES: Tue 14/8/2012 08:22:17 FreeMem=61420772(2) CMMsg=6 (8) Buff=5200 953 1000 7463 5 Links=4272
 488097141mS RES2: IP 500 V2 8.0(44) Tasks=43 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=45 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=1 SSA=0 TCP=14 TAPI=0 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
 488103141mS RES: Tue 14/8/2012 08:22:24 FreeMem=61425088(2) CMMsg=6 (8) Buff=5200 954 1000 7463 5 Links=4299
 488103141mS RES2: IP 500 V2 8.0(44) Tasks=43 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=46 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=1 SSA=0 TCP=14 TAPI=0 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
 488255499mS PRN: WARNING:
 488255499mS PRN: POWER FAILED!! TASKname Daemon f0005fd4 fff03124 f00cd230 00000000 f02d6ce4 f015685c f0168950 f02e5e9c f02e5e48
 
 488255499mS PRN: 
 488255499mS PRN: WARNING:
 488255499mS PRN:  
 488255499mS PRN: 
 488255499mS PRN: WARNING:
 488255499mS PRN: interrupt disabled, 488255499
 488255499mS PRN: 

********** contact lost with 10.0.0.204 at 08:25:08 14/8/2012 - reselect = 3 **********
******************************************************************

********** SysMonitor v10.0 (44) **********

********** contact made with 10.0.0.204 at 08:26:20 14/8/2012 **********

********** System (10.0.0.204) has been up and running for 23secs(23736mS) **********

********** Warning: TEXT File Logging selected **********


********** Warning: TEXT Logging to File STARTED on 14/8/2012 08:26:20 **********
     23736mS PRN: Monitor Started IP=10.0.0.89 IP 500 V2 8.0(44) Sample
                  (IP Office: Supports Unicode, System Locale is enu)
     23737mS PRN: LAW=U PRI=0, BRI=0, ALOG=8, ADSL=0 VCOMP=20, MDM=0, WAN=0, MODU=0 LANM=0 CkSRC=0 VMAIL=0(VER=2 TYP=3) CALLS=0(TOT=3)
     24158mS RES: Tue 14/8/2012 08:26:20 FreeMem=64730372(2) CMMsg=3 (5) Buff=5200 966 1000 7463 5 Links=8816
     24159mS RES2: IP 500 V2 8.0(44) Tasks=37 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=48 Poll=0 Ready=1 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=1 SSA=0 TCP=14 TAPI=0 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
     24680mS PRN: Config Write Wake Up
     24681mS FILESYS: FileSysTask: Started

BAZINGA!

I'm not insane, my mother had me tested!

Signo (IS/IT--Management) · Answer 31 · 2012-08-15 02:37:34

That is the power fail from the customer pulling the power to reset it, that part is very clear to me as well, unless you are thinking something else.

There is protection on both ends also, the grounding of the chassis was not proper, but has been fixed since. We had a decent storm roll through for about an hour tonight, no emails from the system, but last time I didn't get any notification, I just had stuck phones. Its 2:30 am here now but am going straight there in the morning to check it. I have a 1416 on it to see how it reacted if the system did freeze up again.

Signo (IS/IT--Management) · Answer 32 · 2012-08-15 09:15:21

Had a little joy from the site last night, I stormed a little and it made it through. Only time will tell now.

gte6996 (TechnicalUser) · Answer 33 · 2012-11-09 11:54:52

We purchased an IP office Basic system in June of 2012 and finally resolved reboot issues in October. Answer for us was to replace the chassis (shelf) and most importantly was the bad SD card. We fought all the same issues you encountered and thought it was power related. We believe there may be bad SD cards out there because the config looked good right up to 3rd level of AVAYA support.

dango510 (Programmer) · Answer 34 · 2012-11-09 15:45:44

8.0 (46) said it corrected a 8.0(44) system reboot issue... just sayin

tlpeter (Programmer) · Answer 35 · 2012-11-10 03:15:36

I had a spontaneous reboot on 8.0.44 lately while i had monitor open.
I upgraded it to 8.0.46 just to be sure

BAZINGA!

I'm not insane, my mother had me tested!

Signo (IS/IT--Management) · Answer 36 · 2012-11-13 09:04:17

I have not heard from this customer since about mid September, and I have email notifications turned on, and have only seen one reboot in the early AM when I think there electrician was making some electrical changes.

We don't even mention the name of this customer around our office. You know how it is, you say man I haven't heard from that guy in a while and they call the next day lol.

Most of the issue for me went away when it was discovered that the system was not properly bonded with the electrical ground. Its still not perfect as far as the bonding goes, but it is much better then it was.

If I have to go back for any issues I might upgrade, then again it will probably just introduce some new bug so I might not.

I came over from Nortel stuff and I wish Avaya would of just left those phones behind and stayed focused on their on products.

On a side note I would like to thank everyone on this forum for all the great suggestions and advice you dispatch, it is very much appreciated. No matter how long I work in tech I have always found my most valuable tool to be the ability to talk and rationalize things out with other techs. Places like this bring all that knowledge together and make it possible.

Thank You

Contact US

Come Join Us!

Posting Guidelines

500v2 8.0.44 reboot issue 4

500v2 8.0.44 reboot issue

500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

CODE

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

RE: 500v2 8.0.44 reboot issue

Red Flag This Post

Red Flag Submitted

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Join Tek-Tips® Today!

500v2 8.0.44 reboot issue
4

Join Tek-Tips^® Today!