Collect and report Linux System Activity Information with sar

Installation

As noted, sar is a component of the sysstat package, which is already contained in numerous distribution files and can be easily installed, depending on the distribution file, either by yum install sysstat or apt-get install sysstat. Generally at the same time, the cron jobs will be created in /etc/cron.d/sysstat.

Functional Approach of Logging

sa1 and sa2 will store the recorded data in the directories, /var/log/sa/ (RHEL/CentOS) or /var/log/sysstat/ (Debian). The data will be retained for a week retroactively in the default configuration. This duration can be increased to 28 days (in /etc/sysconfig/sysstat under RHEL/CentOS or in /etc/sysstat/config under Debian 4.0 and /etc/sysstat/sysstat under Debian 5.0).

Activating sar Logging in Debian: To activate automatic logging under Debian, the "ENABLED" parameters must be set to "true" in the /etc/default/sysstat file. This parameter has been set to "false" by default.

Here is an example of this:

[root@testserver ~]# cat /etc/sysconfig/sysstat
# How long to keep log files (days), maximum is a month
HISTORY=7
[root@testserver ~]# cat /etc/cron.d/sysstat 
# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

[root@testserver ~]# ls -l /var/log/sa/
total 8952
-rw-r--r--  1 root root 410352 May 18 23:50 sa18
-rw-r--r--  1 root root 410352 May 19 23:50 sa19
-rw-r--r--  1 root root 410352 May 20 23:50 sa20
-rw-r--r--  1 root root 410352 May 21 23:50 sa21
-rw-r--r--  1 root root 410352 May 22 23:50 sa22
-rw-r--r--  1 root root 410352 May 23 23:50 sa23
-rw-r--r--  1 root root 410352 May 24 23:50 sa24
-rw-r--r--  1 root root 410352 May 25 23:50 sa25
-rw-r--r--  1 root root 253712 May 26 14:40 sa26
-rw-r--r--  1 root root 611085 May 17 23:53 sar17
-rw-r--r--  1 root root 611129 May 18 23:53 sar18
-rw-r--r--  1 root root 611127 May 19 23:53 sar19
-rw-r--r--  1 root root 611127 May 20 23:53 sar20
-rw-r--r--  1 root root 611085 May 21 23:53 sar21
-rw-r--r--  1 root root 611100 May 22 23:53 sar22
-rw-r--r--  1 root root 611085 May 23 23:53 sar23
-rw-r--r--  1 root root 611085 May 24 23:53 sar24
-rw-r--r--  1 root root 611125 May 25 23:53 sar25
[root@testserver ~]#

Displaying the recorded data using sar

Limiting the Display Period

sar can display either the data for the current day (which does not require any special parameters) or the data for a specific day (such as with the -f /var/log/sa/sa21parameter).

The query period can be further limited by the following parameters:

• -s [ hh:mm:ss ] - sets the starting time (such as -s 07:30:00); if a specific time is not specified, 08:00:00 will be used as the default starting time.
• -e [ hh:mm:ss ] - sets the ending time (such as -e 19:30:00); if a specific time is not specified, 18:00:00 will be used as the default ending time.

Parameters for Various Sub-systems

Here are several examples for displaying various values (the values originated on a system using CentOS 4.7 with sysstat version 5.0.5).

Displaying the CPU Load: sar -P ALL

The values for all CPUs (-P ALL) will be displayed starting from the 25th of the current month (-f /var/log/sa/sa25).

[root@testserver ~]# sar -P ALL -f /var/log/sa/sa25
Linux 2.6.9-023stab048.6-smp (testserver.thomas-krenn.com)  05/25/09

00:00:01          CPU     %user     %nice   %system   %iowait     %idle
00:10:01          all      9.63      0.00      3.81      6.20     80.36
00:10:01            0      8.69      0.00      3.63      7.18     80.50
00:10:01            1      9.28      0.00      3.96      4.47     82.29
00:10:01            2     10.22      0.00      3.86      6.31     79.62
00:10:01            3      9.84      0.00      3.66      6.91     79.59
00:10:01            4      9.86      0.00      3.94      5.28     80.92
00:10:01            5      9.97      0.00      3.84      6.37     79.81
00:10:01            6      9.90      0.00      4.04      7.21     78.85
00:10:01            7      9.32      0.00      3.54      5.83     81.32
[...]
Average:          all      9.21      0.00      1.68      0.27     88.84
Average:            0      9.03      0.00      1.70      0.31     88.97
Average:            1      9.20      0.00      1.67      0.25     88.88
Average:            2      9.45      0.00      1.70      0.28     88.56
Average:            3      9.84      0.00      1.68      0.26     88.22
Average:            4      9.23      0.00      1.68      0.26     88.84
Average:            5      9.05      0.00      1.68      0.27     89.00
Average:            6      8.90      0.00      1.68      0.29     89.14
Average:            7      8.97      0.00      1.66      0.28     89.10
[root@testserver ~]#

Displaying the RAM Load: sar -r

The RAM values (-r) for the current day will be displayed here. A value of approximately 100 for the percentage of memory used is thoroughly positive (if swapping is not required). See also the Operating System Caches section of the RAID Controller and Hard Disk Cache Settings article.

[root@testserver ~]# sar -r
Linux 2.6.9-023stab048.6-smp (testserver.thomas-krenn.com)   05/26/09

00:00:01    kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
00:10:01        67248  16292680     99.59     42320  12502420   2047648       628      0.03         0
00:20:01       129776  16230152     99.21     47020  12527100   2047648       628      0.03         0
[...]
14:50:01       638680  15721248     96.10    158392   6953128   2047648       628      0.03         0
15:00:01       646272  15713656     96.05    159880   6966980   2047648       628      0.03         0
15:10:01        59784  16300144     99.63    132044   6565036   2047648       628      0.03         0
Average:       554290  15805638     96.61    124889  10992802   2047648       628      0.03         0
[root@testserver ~]# 

Displaying the Load Average: sar -q

The average load values (-q) for the current day will be displayed here.

[root@testserver ~]# sar -q 
Linux 2.6.9-023stab048.6-smp (testserver.thomas-krenn.com)  05/26/09

00:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
00:10:01            0       590      1.62      1.89      1.21
00:20:01           11       574      0.24      0.63      0.86
[...]
15:10:01            7       671      2.36      2.26      2.03
15:20:01            9       651      1.85      1.96      1.99
Average:            7       600      1.02      1.04      1.00
[root@testserver ~]# 

Displaying the I/O Transfer Rate: sar -b

The I/O transfer rates (-r) for the current day will be displayed here.

[root@testserver ~]# sar -b
Linux 2.6.9-023stab048.6-smp (testserver.thomas-krenn.com)  05/26/09

00:00:01          tps      rtps      wtps   bread/s   bwrtn/s
00:10:01       402.21    197.20    205.00  35133.80  27616.04
00:20:01        66.61     56.36     10.24   2034.96    288.93
[...]
15:20:01        72.43      6.54     65.89    176.29   2720.35
15:30:01        73.28      6.08     67.20    399.04   3516.21
Average:        40.21      5.57     34.64    585.00   1561.78
[root@testserver ~]# 

tps - Total number of transfers per second that were issued to physical devices. A transfer is an I/O request to a physical device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size

rtps - Total number of read requests per second issued to physical devices

wtps - Total number of write requests per second issued to physical devices

bread/s - Total amount of data read from the devices in blocks per second. Blocks are equivalent to sectors with 2.4 kernels and newer and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.

bwrtn/s - Total amount of data written to devices in blocks per second.

Displaying the Network Statistics: sar -n DEV

The statistical network values (-n) for the current day will be displayed here.

[root@testserver ~]# sar -n DEV
Linux 2.6.9-023stab048.6-smp (testserver.thomas-krenn.com)  05/26/09

00:00:01        IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
00:10:01           lo      0.82      0.82    177.69    177.69      0.00      0.00      0.00
00:10:01         eth0     31.56     40.21   4537.41  35891.07      0.00      0.00      0.00
00:10:01         eth1   4876.29   9919.22 451754.61 14927283.09      0.00      0.00      0.00
00:10:01       venet0     52.88     44.19  46547.68  15279.15      0.00      0.00      0.00
[...]
15:30:01           lo      0.82      0.82    176.68    176.68      0.00      0.00      0.00
15:30:01         eth0    113.08    155.70  17825.90 188643.77      0.00      0.00      0.00
15:30:01         eth1    634.80   1268.92  58495.76 1888004.02      0.00      0.00      0.00
15:30:01       venet0    189.86    147.08 213164.21  42532.10      0.00      0.00      0.00
Average:           lo      0.82      0.82    177.10    177.10      0.00      0.00      0.00
Average:         eth0     56.42     78.90   9104.66  94428.28      0.00      0.00      0.00
Average:         eth1    281.59    559.55  25919.74 830592.96      0.00      0.00      0.00
Average:       venet0     87.26     64.75  99974.13  14933.05      0.00      0.00      0.00
[root@testserver ~]# 

Displaying All Values

The collective values for all devices (CPU, RAM, etc.) can be displayed using sar -A.

To collect the I/O wait details

 lsblk

NAME                                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0                                      11:0    1  1024M  0 rom
sda                                       8:0    0   550G  0 disk
├─sda1                                    8:1    0   500M  0 part /boot
└─sda2                                    8:2    0 549.5G  0 part
  ├─vg_prod01dbl-lv_root (dm-0)     253:0    0    10G  0 lvm  /
  ├─vg_prod01dbl-lv_swap (dm-1)     253:1    0    64G  0 lvm  [SWAP]
  ├─vg_prod01dbl-lv_tmp (dm-2)      253:2    0     5G  0 lvm  /tmp
  ├─vg_prod01dbl-lv_var (dm-3)      253:3    0    10G  0 lvm  /var
  ├─vg_prod01dbl-lv_database (dm-4) 253:4    0 450.5G  0 lvm  /database
  └─vg_prod01dbl-lv_opt (dm-5)      253:5    0    10G  0 lvm  /opt

sar -bdp 1

12:10:02 PM       tps      rtps      wtps   bread/s   bwrtn/s
12:10:03 PM   6149.49    339.39   5810.10  18828.28  64913.13

12:10:02 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
12:10:03 PM       sda   1908.08   9446.46  32331.31     21.90    289.34     75.91      0.53    101.01
12:10:03 PM vg_prod01dbl-lv_root     12.12    282.83      0.00     23.33      1.44    169.67     83.33    101.01
12:10:03 PM vg_prod01dbl-lv_swap      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:03 PM vg_prod01dbl-lv_tmp      0.00      0.00      0.00      0.00      3.03      0.00      0.00    101.01
12:10:03 PM vg_prod01dbl-lv_var      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:03 PM vg_prod01dbl-lv_database   4228.28   9098.99  32565.66      9.85    698.97     76.50      0.24    101.01
12:10:03 PM vg_prod01dbl-lv_opt      0.00      0.00      0.00      0.00     11.11      0.00      0.00    101.01

12:10:03 PM       tps      rtps      wtps   bread/s   bwrtn/s
12:10:04 PM   5772.00    318.00   5454.00  16960.00  59352.00

12:10:03 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
12:10:04 PM       sda   1925.00   8496.00  29840.00     19.91    285.40     73.32      0.52    100.10
12:10:04 PM vg_prod01dbl-lv_root      6.00     56.00      0.00      9.33      2.35     52.50    166.50     99.90
12:10:04 PM vg_prod01dbl-lv_swap      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:04 PM vg_prod01dbl-lv_tmp      0.00      0.00      0.00      0.00      3.00      0.00      0.00    100.10
12:10:04 PM vg_prod01dbl-lv_var      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:04 PM vg_prod01dbl-lv_database   3844.00   8440.00  29512.00      9.87    656.72     75.01      0.26    100.10
12:10:04 PM vg_prod01dbl-lv_opt      0.00      0.00      0.00      0.00     11.01      0.00      0.00    100.10

Tools for Visual Assessment

The following tools represent the values recorded by sar visually.

• sarface
• Linux Performance Analysis using kSar
• sar2png (A Perl Script for creating PNG files from sar data, tested under GNU/Linux and Solaris 10)
• For additional tools, see Sysstat FAQs Question 2.11. Do you know a tool which can graphically plot the data collected by sar?

References

1. Jump up↑ Sysstat Documentation
2. Jump up↑ Sysstat Download Page
3. Jump up↑ Sysstat Change log up to Version 8.0.4, Sysstat Change log from Version 8.1.1
4.  http://www.thomas-krenn.com/en/wiki/Collect_and_report_Linux_System_Activity_Information_with_sar