Memory Hog 1

[costiles]

We are on 5.3 of AIX running a Universe database application. ~Recently, something used up all of our memory - 40gbs plus paging space such that a new process could not fork. THis is very unusual in our application. Since this is a new box on 5.3 - we are wondering if anyone has seen an errant process that has a memory leak that is so bad that over 12 days it would use up 40 Gbs of memory. We would expect the application to use 1/3 of that memory as a norm. We were running topas at the time

 

[hfaix]

Have you done any tuning?

man vmo

 

[costiles]

We have not had to tune in the past - on other versions of AIX. We are using an application that uses - at best 40 mbs per user - most are at lest than 20 - for 600 users - so we should have been using no more than 2/3 of the memory and without paging. Unfortunately, the company using this box did not notify us until the unable to fork message already was appearing.

 

[kHz]

Wow, UniVerse! I haven't used that (on AIX, too) in over 6 years at another job. There are nice things about it. Not sure if it is still around or not, I thought I heard IBM bought them and (assumed) it would be folded into DB2.

Anyway, you shouldn't have to tune it. It is certainly possible that an application could have a memory leak over two weeks that would consume that much memory. While it is running now, I would keep an eye on the processes using svmon and watch for anything that is growing (especially fast).

Have you found documentation for UniVerse on AIX 5.3 that may include bugs or PTFs or APARs to apply? Since IBM owns UniVerse now, they would be the best to ask about something like that.

 

[costiles]

We are working with IBM on this issue. However, all you have to work with is the system dump - and we find with the universe database, that systems dumps are about as useful as tits on a hog!

 

[BocaBurger]

Piglets would find them very useful. Is anything else besides UniVerse running on the box? Is the version of UniVerse you are running certified / supported on AIX 5.3?





BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[costiles]

A hog is a male pig and therefore incapable of producing milk! No there are no other programs running on this box - and it has only occurred once - this runaway memory issue. I believe that I am going to consider it an anomaly and forget it.

 

[BocaBurger]

Sorry. I am city born. I thought pigs and hogs were interchangable, and a boar was a male hog and a sow was a female hog.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[mrn]

To find a memory leak collect avm stats from vmstat

I normally collect a days worth of stats with sample taken every 5 minutes. Export the avm column into excel and create a graph. If you see a constantly upward graph you more than likely do have a memory leak. Now to find the offending process.

----postvg.sh------

#!/bin/ksh
#
#
# Correlate ps.before and ps.after data ..
#
# command output from ps vg
#
ONE_FILE=temp_ps_vg

print_help() {

print "Version 1.0"
print "Usage: post_vg.sh [single_file|before_ps after_ps]"
print " Post process ps vg output "
print " "
print " where, "
print " single_file contains a before and after snapshot"
print " "
print " No files specified - assume"
print " ==> ps_vg.before "
print " ==> ps_vg.after "
exit -1


}

main() {

if [[ $1 == "-?" ]]
then
print_help
exit -1
fi

if [[ $# == 2 ]]
then
cat $1 $2 > $ONE_FILE

elif [[ $# == 1 ]]
then

cat $1 > $ONE_FILE

else
cat ps_vg.before ps_vg.after > $ONE_FILE

fi

post_vg

rm $ONE_FILE


}

post_vg() {


cat $ONE_FILE | awk 'BEGIN {

list_label = "None"

}

/PID/ {

if( list_label == "None" )
list_label = "Before"
else
list_label = "After"

next
}

{

pid_list[$1]
pid_size[$1, list_label ] = $6
pid_name[$1] = $13
}

END {


printf("%15s\t%10s\t%11s\t%10s\t%10s\n", "pid", \
"Name", \
"Before Size", \
"After Size", \
"Delta")



for( pid in pid_list ) {

if( (pid,"Before") in pid_size && (pid,"After") in pid_size ) {

delta = pid_size[pid, "After"] - pid_size[pid, "Before"]
d_total += delta

printf("%15s\t%10s\t%11d\t%10d\t%10d\n", \
pid, \
pid_name[pid], \
pid_size[pid, "Before"], \
pid_size[pid, "After"], \
delta )

}

}

printf("*** Total Delta %d\n", d_total)


}'

}

main $@

----End-----


1) Issue the following command

# ps vg > ps.before

2) Wait some period of time -- perhaps 30 minutes and issue the command again.

# ps vg > ps.after

3) Now, post process the two files

# ./post_vg ps.before ps.after

Note: The script assumes that before is the first file.

The 'Delta' indicates a change in the process SIZE. One reason for an
increasaed SIZE value is a memory leak. For more information about
detecting memory leaks review the paper Methods for Identifying Memory
Leaks in AIX Systems.


Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

http://www.airport-parking-site.co.uk/