Sol 2.7 swap partition filling up without memory growth

[EricAndresen]

I have a server which is showing a steady reduction of free swap disk space (as shown by "swap -l"):

# swap -l (Nov 10 6:07 pm)
swapfile dev swaplo blocks free
/dev/dsk/c0t0d0s0 32,120 16 1049568 768016

# swap -l (Nov 11 10:27 am)
swapfile dev swaplo blocks free
/dev/dsk/c0t0d0s0 32,120 16 1049568 370864


However all of the other processes which show swap usage (top, vmstat, swap -s, etc...) aren't showing this growth. Even summing up the values returned by "ps -e -o vmem" is showing marginal growth, but nothing in the same magnitude as this swap is decreasing.

Is there something else which could be eating away at my swap partition, or causing a process'es memory to not show up in the other tools?

A look at the /etc/vfstab file shows that the swap partition doesn't have a mount point, so there's no way that it's getting filled up with files as happens when /tmp is used for swap:

# cat vfstab
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c0t0d0s0 - - swap - no -
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 / ufs 1 no -
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no -
/dev/dsk/c0t0d0s4 /dev/rdsk/c0t0d0s4 /opt ufs 2 yes -
/dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0s1 /tmp ufs 2 yes -
/dev/dsk/c0t0d0s3 /dev/rdsk/c0t0d0s3 /var ufs 1 no quota



I know this problem is related to processes using (and possibly leaking) memory, because if I kill the processes I get the swap space back, but since none of the other tools are showing growth I can't figure out why (or if) the total memory usage is actually growing, or which specific processes are actually causing it.

Anyone have any ideas on alternate ways to look at swap usage and process memory usage?

 

[spamly]

Okay, this may or may not help you, but it probably should be said. Solaris 7 has a known memory leak that Sun has recognized, but has failed to fix. Sun has encouraged their customers to upgrade to a latter version of Solaris if possible. Now, I'm not sure if you are experiencing the infamous memory link, but I thought I'd post this anyway.

By the way, the way that you're using swap as a separate filesystems, not linked to /tmp, is fantastic! I'm going to consider doing this on some test servers.

 

[EricAndresen]

Thanks, do you have any links to this infamous memory leak, such as how to detect if it is actually what we are seeing?

 

[spamly]

I'll see if I can dig it up. I originally heard about it through a Sun performance tuning class.