Linux Tricks and Techniques

For Linux Experts

Use the website for easy leaning of Linux

Saturday, 9 July 2016

Sar Command in Linux

 Sar Utility in Linux



System Activity Reporter is an important tool that helps system administrators to get an overview of the server machine with status of different important metrics at different points of time.

If suppose you are having an issue with the system currently, Like some of your customers are unable to list some data from the database. The first thing that most of the Linux system administrators do is to recall the same issue when it previously occurred, and If you remember the day of its previous occurrence then you can easily compare the internal system statistics with the current statistics.

SAR is very much helpful in doing exactly that.

The first thing that we need to do is check and confirm whether you have SAR utility installed on the machine. Which can be checked by listing all rpm's and finding for this utility.

SAR is one of the utility inside sysstat. You can easily download and install it in your machine very easily through YUM. (But yeah dont worry because most of the distribution comes prepacked with sysstat tool).

[root@myvm1 ~]# yum install sysstat

Yeah but make it sure that you have epel,rpmformge repository enabled for installing. Otherwise your distribution DVD will be a nice place to look for the package.

SAR (System Activity Reporter) will Give Information about the following things:





1. System Buffer activity

2. Information about system calls

3. Block device information

4. Overall paging information

5. Semaphore and memory allocation information

6. CPU utilization and process report

The main thing that we need to understand regarding SAR is that, everything is done using a cron. By default in many Linux distribution you will have a file named /etc/cron.d/sysstat.

Lets see how really SAR works.

If we start thinking about system monitoring, then the tool must have each and every data about the system's different aspects and must cover all time intervals. Which means a monitoring system must be able to provide the statistics of the machine for a given time.

There is no way, other than taking all the metrics and statistics of the machine at a definite time interval. Reducing the time interval for collecting the statistics will increase the amount of detailed statistics we have(because we will be having more data about the system).

SAR does exactly that. sar takes the statistics of different aspects of the machine at a definite time interval. So SAR runs through CRON.

[root@myvm ~]# cat /etc/cron.d/sysstat

# run system activity accounting tool every 10 minutes

*/10 * * * * root /usr/lib64/sa/sa1 1 1

# generate a daily summary of process accounting at 23:53

53 23 * * * root /usr/lib64/sa/sa2 -A



· So it can be seen from the above cron file for SAR that its running "sa1" script located at "/usr/lib64/sa/" at every 10 minutes

· And is also running a script /usr/lib64/sa/sa2 at the end of the day at around 23.53

So the first cron entry for SAR(/usr/lib64/sa/sa1) will run every 10 minutes which inturn will call the sadc utility to collect system stats and store it in a binary file (one file for a day)

And the second cron entry will dump all the contents of that binary file into another text file, and purges data older than a particular number of days, Normally 7 days by default(which is mentioned in the following file),

[root@archive ~]# cat /etc/sysconfig/sysstat

# How long to keep log files (days), maximum is a month

HISTORY=7
So you can modify that HISTORY entry easily by editing the file.

So although the system statistics is being collected every 10 minutes through cron(modify the cron to run every 1 minute for more accurate information) If you want to see the stats, then you need to run the command as below.

The simple sar command output is as shown below.

12:00:01 AM CPU %user %nice %system %iowait %steal %idle

12:01:01 AM all 73.28 0.00 1.25 0.00 0.00 25.47

12:02:01 AM all 7.83 0.00 0.44 0.00 0.00 91.73

12:03:01 AM all 61.65 0.00 0.70 0.00 0.00 37.66

12:04:01 AM all 57.85 0.00 0.82 0.00 0.00 41.34

12:05:01 AM all 4.25 0.00 0.41 0.00 0.00 95.34

12:06:01 AM all 4.20 0.00 0.22 0.00 0.00 95.58

12:07:01 AM all 5.05 0.00 0.33 0.00 0.00 94.63

12:08:01 AM all 4.76 0.00 0.06 0.00 0.00 95.18

12:09:01 AM all 37.57 0.00 0.37 0.00 0.00 62.05

12:10:01 AM all 70.04 0.00 0.80 0.00 0.00 29.16

12:11:01 AM all 5.03 0.00 0.12 0.00 0.00 94.84



It can be seen from the output that its reporting me the output of the collected stats for every minute(which means i have my cron at 1 minute interval), and will show the details of the whole day(or will show details collected till when you typed the command).

Understanding the output of SAR command

%user: This shows the total time that the processor is spending on different process YCX5UKN5ZKEJ

%sys: this shows the percentage of time spend by the processor for operating system tasks(because the previous user shows the time spend for user end process)

%iowait: the name iowait itself suggests that its the time spend by processor waiting for devices(input and output)

%nice: Most of you guys must be knowing that a user can change the priority of a process in linux by changing the nice value in Linux. This table shows the time spend by CPU for process whose nice value has been changed.

%steal: This column shows the amount to time spend by a CPU (which is virtualized), for resources from the physical CPU

%idle: This suggests the idle time spend by the processor.

By default sar stores all of its data under /var/log/sa/ and a days are named as shown below.

s01 - for first day of the month

s02-for second day of the month

s03,s04..and so on.

-d option in SAR command

This -d option can be used to report each and every activity related to different devices attached to the system(block devices). A typical output of the sar command with -d option is shown below.

12:00:01 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util

12:01:01 AM dev3-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:01:01 AM dev3-64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:01:01 AM dev8-0 55.62 9.98 8317.87 149.72 13.68 245.96 2.61 14.52

12:01:01 AM dev8-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:02:01 AM dev3-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:02:01 AM dev3-64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:02:01 AM dev8-0 1.55 0.00 35.29 22.70 0.01 7.26 1.86 0.29

12:02:01 AM dev8-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:03:01 AM dev3-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:03:01 AM dev3-64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
DEV: this column names devices on the machine, according to major and minor numbers of a Linux block device. You can check this by doing an ls -l in /dev directory. as shown below.

brw-r----- 1 root disk 8, 0 Nov 16 16:29 sda

brw-r----- 1 root disk 8, 1 Nov 16 16:29 sda1

brw-r----- 1 root disk 8, 2 Nov 16 16:29 sda2

in the above shown "ls -l" outut for "sda", major number is "8",and minor number is "0"...So you can easily identify the disk mentioned in the sar command output.

tps: tps stands for transfer per second, so it shows the transfer per second to that particular device

rd_sec/s: this shows you the total number of sectors on that device which is being read

wr_sec/s: if rd_sec/s is sectors being read per second then obviously wr_sec is sectors being written per second.

avgrq-sz: this column shows the average.

await: this shows the total number of time that the processor waited for requests regarding IO

%util: this column shows the usage of cpu in percentage when the request was generated

Show Memory usage in SAR command

the -r option available in sar command is very much useful. it shows the memory,swap,cached memory etc at every interval or required time interval.

02:20:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad

02:30:01 AM 609500 1487652 70.94 242420 777560 1075980 364 0.03 360

02:40:01 AM 609500 1487652 70.94 242424 777568 1075980 364 0.03 360

02:50:01 AM 609500 1487652 70.94 242424 777592 1075980 364 0.03 360

03:00:01 AM 608980 1488172 70.96 242424 777600 1075980 364 0.03 360

03:10:01 AM 608584 1488568 70.98 242424 777628 1075980 364 0.03 360

03:20:01 AM 608584 1488568 70.98 242424 777648 1075980 364 0.03 360

in the above output most of the columns are self explanatory(and most of the outputs are in KB).

kbmemfree: this shows the amount of free memory

Kbmemused: memory used

%memused: percentage of memory used

kbbuffers: buffer memory used by the kernel.

kbcached: cached memory used by the kernel

all other entries for memory are swap(free,used,percentage etc)
How to fetch metrics of a particular day using SAR in linux

As mentioned before all the metrics for a particular day are saved in sa<day of month> wise. So if i want to know my metrics for 27 th day the month i can easily find that out as shown below.

[root@archive ~]# sar -f /var/log/sa/sa27

Linux 2.6.18-194.el5xen (archive.r) 11/27/2012



02:20:01 AM CPU %user %nice %system %iowait %steal %idle

02:30:01 AM all 2.58 0.00 0.70 1.12 0.05 95.55

02:40:01 AM all 2.56 0.00 0.69 1.05 0.04 95.66

02:50:01 AM all 2.64 0.00 0.65 1.15 0.05 95.50

03:00:01 AM all 3.27 0.00 0.71 1.12 0.04 94.86

in the above command we have passed /var/log/sa/sa27 as an argument as i needed stats for that day..pass the sa<day of month>as you require in the above command

How to fetch SAR metrics for a specific time on a particular date

this can be achieved by passing another argument as shown below.

[root@archive ~]# sar -f /var/log/sa/sa27 -s 02:20:00 -e 03:20:00

Linux 2.6.18-194.el5xen (archive.r) 11/27/2012

02:20:01 AM CPU %user %nice %system %iowait %steal %idle

02:30:01 AM all 2.58 0.00 0.70 1.12 0.05 95.55

02:40:01 AM all 2.56 0.00 0.69 1.05 0.04 95.66

02:50:01 AM all 2.64 0.00 0.65 1.15 0.05 95.50

03:00:01 AM all 3.27 0.00 0.71 1.12 0.04 94.86

03:10:01 AM all 2.72 1.06 0.75 1.09 0.04 94.33

Average: all 2.76 0.21 0.70 1.11 0.04 95.18

in the above shown example i asked sar to fetch the metrics between 2:20:00 and 3:20:00 on 27th day of the month

You can even pass any other metric option along with the time interval...such as -d or -r

Have you noticed ? SAR can accurately show us the machine statistics of a particular day at a particular time...so its much easier to identify the bottlenecks.

Using -A option along with the above command will show ALL (all the metrics collected by sar).

sar -f /var/log/sa/sa27 -s 02:20:00 -e 03:20:00 -A

The output will be elaborate. and you will get almost everything in sar from that -A option on your screen!

Show network statistics using sar command

sar command even shows network statistics. This can be done by using the -n DEV option in sar command.

[root@archive ~]# sar -n DEV

Linux 2.6.18-194.el5xen (archive.r) 11/27/2012



02:20:01 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s

02:30:01 AM lo 0.01 0.01 0.77 0.77 0.00 0.00 0.00

02:30:01 AM eth0 12.30 0.12 1285.06 27.59 0.00 0.00 0.00

02:30:01 AM eth1 14.45 0.00 1399.34 0.00 0.00 0.00 0.00

02:30:01 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:40:01 AM lo 0.01 0.01 0.77 0.77 0.00 0.00 0.00

02:40:01 AM eth0 10.65 0.12 1139.38 27.00 0.00 0.00 0.00

02:40:01 AM eth1 13.96 0.00 1352.87 0.00 0.00 0.00 0.00

IFACE: stands for the nic card interface name

rxpck/s: this shows the total packets received per second

txpck/s:transmitted packets per second

rxcmp/s: compressed packets received

txcmp/s: compressed packets transmitted

rxmcst/s: packets multicasted per second.



Some other Metrics that can be determined using sar

-y option in sar: can be used to determine tty details

-X option in sar to get details of a particular process. You need to pass pid as an argument to this option.

-n SOCK option in sar: this option will report all socket details.



There are couple of more options available in SAR.You can refer for the complete list of arguments in man pages..Hope this introductory article was helpful in getting started and using SAR..

Thanks

No comments:
Write comments