vmstat and sar interpretation

[bi]

Does anyone know of a good web site or book that will explain in uncomplicated terms the meaning of the output of vmstat and sar as well as some guidelines for what a well running box has as output? I want more than what I have found at the docs.sun website and in the man pages.

I am running Solaris 7 on an E450.

 

[FarahRegal]

check it out itworld.com

Introducing SAR Sign up for UNIX SYSTEM ADMINISTRATION





Some companies bridge the gap between an excessive amount of available data and the bottom line system performance by creating or employing evaluation tools for the raw numbers and preparing a report that provides conclusions, not just numbers. SarCheck (a tool available from Aptitune Corporation) is one such tool. It provides some of the performance insights that might otherwise only be available to those staffs blessed by the presence of a performance specialist.

The sar command can be thought of as running in two modes: interactive or "real-time". "Real-time" mode reports on the system's current activity and "historical", which uses data previously collected and stored in log files. In both cases, the reports reflect data that is routinely collected in the kernel but, in the latter case, this data is sampled and stored so that past performance can be analyzed.

sar is not strictly a Solaris tool, either. It's available in other flavors of Unix as well, though configuration and default behavior may vary between implementations. RedHat Linux systems collect system activity data routinely and save it in files in the /var/log/sa directory. Solaris systems come prepared for running sar in either mode, but collection of data in performance logs must be specifically invoked by un-commenting lines in the start script (/etc/rc2.d/S21perf) and crontab file (/var/spool/cron/crontabs/sys) associated with the tool.

The Solaris package containing the sar commands is called SUNWaccu. The interactive and historical versions of the sar command differ only in where the data is coming from -- from the kernel moment by moment or from one of the log files containing previously collected performance data. The simplest version of the sar command (sar with no options) on a RedHat Linux system provides a report that looks like this:

[root@dragonfly /]# sar
Linux 2.4.7-10 (dragonfly) 03/31/2002

03:50:00 PM CPU %user %nice %system %idle 04:00:00 PM all 13.96 0.29 9.75 76.00 04:10:00 PM all 0.94 0.00 0.51 98.55 04:20:00 PM all 3.32 0.01 0.13 96.54 04:30:00 PM all 1.28 0.00 0.34 98.38 04:40:00 PM all 0.76 0.00 0.14 99.11 04:50:00 PM all 88.00 0.10 0.41 11.48 05:00:00 PM all 88.97 0.10 0.52 10.41 05:10:00 PM all 54.77 0.08 0.81 44.35 05:20:00 PM all 1.42 0.00 1.28 97.30 05:30:00 PM all 1.17 0.00 0.68 98.15 Average: all 25.46 0.06 1.46 73.03

This is almost the same as the Solaris output (though each of the commands Includes one unique column):

$ sar

SunOS froggie 5.7 Generic_106541-16 sun4u 04/04/02

00:00:00 %usr %sys %wio %idle
01:00:00 0 1 0 99 02:00:00 0 1 0 99 03:00:00 0 1 0 99 04:00:00 0 1 0 99 05:00:00 0 1 0 99 06:00:00 0 1 0 99 07:00:00 0 1 0 99 08:00:00 0 1 0 99 08:20:01 2 1 0 97 08:40:00 0 1 0 99 09:00:00 6 1 0 93 09:20:00 0 1 0 99 09:40:00 3 1 1 94

Average 1 1 0 99

This simplest form of the sar command extracts CPU activity from the current systems activity log. If you try this on a Solaris system on which the collection of data into the log files has not been enabled, the system will respond with a complaint that it cannot open the log file.

The easiest way to see all the data available to you is to use the sar - A command. This option is the same for Linux and Solaris. The -A denotes "all". In other words, this command reports all available activity counters. If you're not collecting data in log files, try "sar -A 10 1" for a single 10-second report.

sar's data files contain binary data, so you need to use the sar command to make sense of the log file contents.

Sar provides more than a dozen options for looking at more than a dozen kernel counters. The sar -B command, for example, reports on system paging activity. The data below shows a flurry of page-in activity at 4 P.M. (the time this system was starting processes following boot-up) and little subsequent activity.

03:50:00 PM pgpgin/s pgpgout/s activepg inadtypg inaclnpg inatarpg 04:00:00 PM 124.46 56.32 4038 29778 0 1994 04:10:00 PM 1.25 2.34 3976 30117 0 75 04:20:00 PM 0.54 1.75 3987 30264 0 82 04:30:00 PM 0.16 6.08 4249 30186 0 129 04:40:00 PM 0.08 2.37 3923 30617 0 34 04:50:00 PM 0.00 1.57 4054 30563 0 149 05:00:00 PM 0.00 1.57 4054 30636 0 148 05:10:00 PM 0.02 2.04 3939 30832 0 535 05:20:00 PM 0.00 1.33 3939 30898 0 541 05:30:00 PM 0.01 2.33 4051 30882 0 134 Average: 12.65 7.77 4021 30477 0 382

This complements the memory report created by the sar -R command:

03:50:00 PM frmpg/s shmpg/s bufpg/s campg/s 04:00:00 PM -52.81 0.04 12.11 20.00 04:10:00 PM -0.42 0.00 0.15 0.31 04:20:00 PM -2.21 0.00 0.13 0.13 04:30:00 PM -1.34 0.00 0.27 0.04 04:40:00 PM 0.24 0.01 0.15 0.02 04:50:00 PM -1.09 0.00 0.13 0.00 05:00:00 PM -0.16 0.00 0.12 0.00 05:10:00 PM 0.81 0.00 0.13 0.01 05:20:00 PM -0.52 0.00 0.11 0.00 05:30:00 PM 0.16 0.00 0.15 0.01 Average: -5.73 0.01 1.34 2.05

Here, we see the frmpg column (indicating the number of pages freed by the system per second) is reporting mostly negative numbers -- indicating that memory pages are being allocated rather than freed.

For interactive commands, you can specify that you want to see particular counters over a particular sampling interval. The command "sar 5 5", for example, will prepare a report showing CPU activity over 5 intervals of 5 seconds each.

Linux 2.4.7-10 (dragonfly) 03/31/2002

06:08:14 PM CPU %user %nice %system %idle 06:08:19 PM all 3.40 0.00 0.20 96.40 06:08:24 PM all 0.40 0.00 0.20 99.40 06:08:29 PM all 0.00 0.00 0.00 100.00 06:08:34 PM all 0.00 0.00 0.00 100.00 06:08:39 PM all 0.20 0.00 0.00 99.80 Average: all 0.80 0.00 0.08 99.12

Looking at intervals shorter than 5 seconds is a bad idea because the data collection at this frequency can worsen performance.

Sar log files are named "sadd" where "dd" represents the day of the month. The data file for April 8th, for example, is named sa08. This makes it easy to figure out what file to use to look at some other day's activity. Let's say we want to look at CPU activity from March 27th. This command would pull that data from the appropriate log file:

[root@dragonfly sa]# sar -f /var/log/sa/sa27
Linux 2.4.7-10 (dragonfly) 01/27/2002

05:20:00 AM CPU %user %nice %system %idle 05:30:00 AM all 0.36 0.00 0.53 99.11 05:40:00 AM all 0.46 0.00 0.56 98.98 05:50:00 AM all 0.61 0.00 0.59 98.80 06:00:00 AM all 0.48 0.00 0.73 98.79 06:10:00 AM all 0.96 0.00 0.98 98.06 06:20:00 AM all 3.03 0.00 1.93 95.04 06:30:00 AM all 68.54 0.06 1.13 30.28 06:40:00 AM all 88.55 0.07 1.10 10.29 06:50:00 AM all 88.74 0.10 1.06 10.10 07:00:01 AM all 88.35 0.11 1.06 10.48 07:10:00 AM all 88.00 0.11 1.13 10.76 07:20:00 AM all 88.54 0.10 1.06 10.30 07:30:00 AM all 88.60 0.11 1.04 10.25 07:40:00 AM all 88.62 0.09 1.04 10.24 (truncated)

We can conclude that this particular system began to be busy at roughly 6:30 in the morning, maybe when the first users showed up at their desks.

If you want to look at a particular period of time (let's say Friday evening between 6 and 6:30 PM, you can specify a beginning and ending time and use it to select data from the stored data file. For example:

[shs@dragonfly shs]$ sar -f /var/log/sa/sa05 -s 18:00:00 -e 18:30:00 Linux 2.4.7-10 (dragonfly) 02/05/2002

06:00:00 PM CPU %user %nice %system %idle 06:10:00 PM all 81.33 0.00 1.61 7.07 06:20:00 PM all 91.57 0.00 1.82 6.61 06:30:00 PM all 81.45 0.00 1.12 17.43 Average: all 81.45 0.00 1.71 16.84

If you collect log data routinely with sar, you will have a lot of data with which to evaluate performance. The issue that still remains is how to best analyze all these numbers and determine what needs to be done to improve performance on a particular system.

.

Farah regal
good luck
"think twice and hit enter once"