OSMemPool::Allocate Out of Memory

[JoshuaThompson]

We are running IP Office 406 v3.0(69) and have experienced two random reboots recently.

We had the log monitor running at the time of the latest reboot and it captured the following:

1697451520mS PRN: OSMemPool::Allocate Out of Memory
********** contact lost with x.x.x.x at 08:25:37 7/8/2008 - reselect = 1 **********
note: ( I have removed IP Address)

I searched this forum and found the below thread:

http://www.tek-tips.com/viewthread.cfm?qid=1351280

This posting is same error I am receiving (although its for a version 2 device) It talks about the issue being because the config size has exceeded the max limit.

Another posting here(

http://www.tek-tips.com/viewthread.cfm?qid=1417364

) shows the max config file sizes for my unit is 192KB.

My config file size is showing at 78KB

Since my config file size is well under the max size, what else could be causing this issue?

Any help is greatly appreciated.

 

[Westi]

maybe it has been upgrade without the proper steps taken to upgrade the loader and reallocate the internal memory, just a guess though see if you can find any information in the GA announcement for r 3.0 it would have the steps in it and then you could try to downgrade to any interim release and then back up to the proper release.

Joe W.

FHandw., ACS

If you can't be good, be good at it!

 

[JoshuaThompson]

Thank you for you the reply.
I reviewed the GA announcement for 3.0 and it appears the additional steps needed for the loader where only for IP Office 403. We have 406.

Any other ideas?

 

[Westi]

Sorry man I just thought it might be that, no other ideas from me.
When you start the monitor do you have system resources checked on and it displays the system memory available, it might be worth looking at that to see what it is and if it is low (16MB) you might be able to switch it for the 64 MB via PCN from Avaya and that would rule out a hardware problem.
You can also look at the back and see if it is a PCS 6; 7; 8 or higher.

Joe W.

FHandw., ACS

If you can't be good, be good at it!

 

[JoshuaThompson]

I will take any help you are willing to give!

When the Monitor starts I do not see an option for System Resources. We are running SysMonitor v5.0(69)

Where should I see this?

Joshua

 

[Westi]

under the options you can check on what to monitor and that is called I(I think) system resources I am drawing a blank right now but as soon as I get to my docs I will look it up.

Joe W.

FHandw., ACS

If you can't be good, be good at it!

 

[mattKnight]

It is possible you have an issue with your programming - for example a member of a huntgroup forwarding all calls to the same huntgroup and createing a loop. Look for wierd things like that...

Take Care

Matt
I have always wished that my computer would be as easy to use as my telephone.
My wish has come true. I no longer know how to use my telephone.

 

[Westi]

just had another idea, is the IP address that you removed a public IP address or a private one?



Joe W.

FHandw., ACS

If you can't be good, be good at it!

 

[JoshuaThompson]

mattKnight - I dont see the system resources that you are referring to in the monitoring program. I will research the huntgroup tip

Westi - The IP address that was removed was a private (internal only)

 

[Westi]

the resources are in the monitor under
Filters - Trace options- under the system tab there is a check mark for resource status prints

Joe W.

FHandw., ACS

If you can't be good, be good at it!

 

[JoshuaThompson]

Thank you Westi.

The Resouce Status prints was enabled under the system tab. (I was thinking it would called something else). Sorry.

When my monitor starts up below is the first couple entries:

********** SysMonitor v5.0 (69) **********

********** contact made with x.x.x.x. at 13:17:01 18/8/2008 **********

********** System (x.x.x.x.) has been up and running for 5hrs, 26mins and 18secs(19578360mS) **********


********** Warning: TEXT File Logging selected **********



********** Warning: TEXT Logging to File STARTED on 18/8/2008 13:17:01 **********

1mS PRN: Monitor Started IP=x.x.x.x IP 406 3.0(69) Seattle
1mS PRN: LAW=U PRI=1, BRI=0, ALOG=4, ADSL=0 VCOMP=5, MDM=2, WAN=0, MODU=3 LANM=0 CkSRC=5 VMAIL=1(VER=2 TYP=1) CALLS=0(TOT=255)
273mS RES: Mon 18/8/2008 13:16:50 FreeMem=4157328(48) CMMsg=11 (12) Buff=100 610 500 1263 6 Links=5467 Elements=0
3001mS PRN: +++ START OF ALARM LOG DUMP +++
3001mS PRN: ALARM: 07/04/2008 08:37:56 IP 406 3.0(69) <TLB Data Error> CRIT RAISED addr=0000025c d=5 pc=015c39ec 015c40ec 01613cbc 0161411c 0161438c 01614518
3001mS PRN: +++ END OF ALARM LOG DUMP +++
12773mS RES: Mon 18/8/2008 13:17:03 FreeMem=4157416(49) CMMsg=11 (12) Buff=100 610 500 1263 6 Links=5467 Elements=0



Looks like FreeMem starts off about 4157416 (assuming KB).

I might be going down the wrong path but I have a script that extracts the FREEMEM value from the log files periodically and am noticing that before it rebooted it was down to 33752. Possibly memory leak with this old software?

The other note is the CRIT RAISED message in the beginning of the log file. Im not quite sure what that is.

 

[TomMills]

The CRIT RAISED is part of the TLB error if you look to the left of it,

that is part of the alarm dump.

was one of your random reboots on 07/04/2008 per chance?

 

[JoshuaThompson]

We have been having intermittant issues for a couple months now, so yes that date (which I believe is actually April 7th) could have been a reboot.

Why is it showing the log still? The other reboots arent. I dont see anything in the Alarm filter that would allow me to clear it.

I am new to IP Office. What is a TLB error. What does CRIT RAISED mean? I havent been able to find any sort of manual for the system monitor that tell you what everything means.

 

[tlpeter]

Let monitor run and log it to a file
When it happens then check the monitor file
In there you can see what happend

What it all means whe don't know, only the developers probably


ACA - Implement IP Office
ACS - Implement IP Office
ACA - Voice Services Management
______________
Women and cats can do as they please and men and dogs should relax and get used to the idea!

 

[JoshuaThompson]

Thanks tlpeter.

However I already know what happened:

1697451520mS PRN: OSMemPool::Allocate Out of Memory

What I dont know is why. We are monitoring the log file hourly and plan to reboot the system when the FreeMem values are low.

 

[tlpeter]

It has to do with something before that
But i think it is a good idea to upgrade the system



ACA - Implement IP Office
ACS - Implement IP Office
ACA - Voice Services Management
______________
Women and cats can do as they please and men and dogs should relax and get used to the idea!

 

[JoshuaThompson]

I think we are going to settle on monitoring the FreeMem and manually reboot.

Thanks everybody for the suggestions / information