AIX swapping question

[nyck]

One of my AIX5.3 servers is staring to run low on swap:-

# lsps -s
Total Paging Space Percent Used
16384MB 75%
# lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 16384MB 75 yes yes lv

Is there an easy way to determine which process could be causing this issue?

 

[khalidaaa]

Try svmon -P | more

It will list the memory utilization for all your processes on the box sorted with hight memory utilization first.

This is not it! you can tune some of the memory parameters to avoid having this if possible! I don't know exactly what you run on the box.

Could you list the output of the vmstat 1 20 and svmon -G as a start? (that's if you didn't find the first command useful!)

Regards,
Khalid

 

[khalidaaa]

By the way, you can use topas to view a live performance data and maybe locate that process as well!

Regards,
Khalid

 

[nyck]

Output below:-

vmstat 1 20

System configuration: lcpu=16 mem=32000MB

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 5101382 8489 0 0 0 0 0 0 17 1388 236 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 11 1104 186 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 10 1118 268 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 12 1149 214 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 5 1234 500 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 7 1168 302 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 7 1110 211 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 4 1297 174 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 8 1082 257 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 3 1167 200 0 0 99 0
0 0 5101401 8470 0 0 0 0 0 0 12 1483 194 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 4 1112 289 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 3 1201 208 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 6 1144 183 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 5 1145 260 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 8 1173 210 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 4 1142 176 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 7 1103 264 0 0 99 1
0 0 5101383 8488 0 0 0 0 0 0 16 1156 210 0 0 99 0
0 0 5101383 8488 0 0 0 0 0 0 5 1659 206 0 0 99 0
# svmon -G
size inuse free pin virtual
memory 8192000 8183516 8484 753233 5101385
pg space 4194304 3115297

work pers clnt
pin 753233 0 0
in use 2092602 0 6090914

PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 8023532 3092913 647825 4868041
m 64 KB - 9999 1399 6588 14584

From the vmstat output I cannot really see anything too wrong there, its not paging in and out which is nice. But I ran a svmon -P, below is a small exert from the output:-

Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
1253618 oracle 1655286 19 24173 55179 Y N N

PageSize Inuse Pin Pgsp Virtual
s 4 KB 1654544 0 24081 52226
m 64 KB 726 3 92 2937

So i'm asuming that oracle is to blame in this case, would you agree?

 

[khalidaaa]

What's the vmstat -v output?

I usually use the same size of paging space as the RAM installed! In your case, you are using 16GB for paging space and 32GB for RAM! IHMO I think you have to increase the paging space upto 32GB.

Regards,
Khalid

 

[nyck]

The output is as follows:-

vmstat -v
8192000 memory pages
7842014 lruable pages
8933 free pages
2 memory pools
753314 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
76.0 numperm percentage
5966226 file pages
0.0 compressed percentage
0 compressed pages
77.6 numclient percentage
80.0 maxclient percentage
6090624 client pages
0 remote pageouts scheduled
6234889 pending disk I/Os blocked with no pbuf
143834 paging space I/Os blocked with no psbuf
2740 filesystem I/Os blocked with no fsbuf
35 client filesystem I/Os blocked with no fsbuf
11073251 external pager filesystem I/Os blocked with no fsbuf

 

[khalidaaa]

It seems that this system has not been tuned for memory.

So what else (other than oracle) are you running on this box and what version of oracle?

What version of AIX is this? are you using jfs or jfs2?

You need to tune the minperm, maxperm, minfree and maxfree.

Regards,
Khalid

 

[nyck]

This server has been running fine since I built it about one year ago, as its stands no one is actually complaining about it, its just that Nagios picked it up and I happend to look at the alert instead if deleting it

It's just got oracle10G and some in-house applications installed on it. Its currently running AIX5.3 and its jfs2.

Is this tuning done via vmtune?

 

[khalidaaa]

No i'm not saying that your installation has problems! I just suggest that you can better utilize your RAM if you tune it.

Look into this thread! it has very nice recommendations by our fellows here in the forum:

http://www.tek-tips.com/viewthread.cfm?qid=1302007&page=1

Regards,
Khalid

 

[nyck]

No worries, I'm still quite new to this AIX stuff and am learning new things about it each day, I'm beginning to like it more and more!

Thanks very much for your replies on my many many questions!

 

[khalidaaa]

It's ok to ask questions! that's why we are here, to help each other

Regards,
Khalid