Pages

Monday, August 19, 2013

SAR ( System Activity Reporter )

SAR ( System Activity Reporter ) is an excellent command available on Linux for obtaining various information regarding the System. SAR tool can be used to

  1. Monitor System Real Time Performance ( When the System is Running ).Information like CPU,Memory , I/O and many more Details.

  2. Collect Performance Data on Back Ground and allows to use that data to identify Bottlenecks.

SAR is available from the sysstat package.Here are some of the Things that can be done using the SAR command,
  • Collective CPU usage
  • Individual CPU statistics
  • Memory used and available
  • Swap space used and available
  • Overall I/O activities of the system
  • Individual device I/O activities
  • Context switch statistics
  • Run queue and load average data
  • Network statistics
  • Report sar data from a specific time

Options for the sar Command

Option
Actions
-a : Checks file access operations
-b : Checks buffer activity
-c : Checks system calls
-d : Checks activity for each block device
-g : Checks page-out and memory freeing
-k : Checks kernel memory allocation
-m : Checks interprocess communication
-v : Checks system table status
-p : Checks swap and dispatch activity
-q : Checks queue activity
-r : Checks unused memory
-u : Checks CPU utilization
-w : Checks swapping and switching volume
-y : Checks terminal activity
-A : Reports overall system performance, which is the same as entering all options.

Lets see a few examples on how we can use sar command in obtaining various information.

CPU Usage (sar -u)

rootLocal:root002-~ $ sar -u 1 3
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:05:35 PM CPU %user %nice %system %iowait %steal %idle
10:05:36 PM all 0.75 4.00 0.17 0.00 0.00 95.09
10:05:37 PM all 0.58 27.33 0.42 0.00 0.00 71.67
10:05:38 PM all 1.67 9.16 0.75 0.00 0.00 88.43

Average: all 1.00 13.49 0.44 0.00 0.00 85.06

The above output gives you the real time usage of all the CPUs available in the System.“1 3″ reports for every 1 seconds a total of 3 times. The Important thing we need to focus up here was the %IDLE which gives the idle time of the CPU.

Following are some of the other ways of using the sar command for the CPU information

sar -u : Displays the CPU Information for the Current day until at this point
sar -u 1 3 : Displays real time CPU usage every 1 second for 3 times.
sar -u ALL : Same as “sar -u” but displays additional fields.
sar -u ALL 1 3 : Same as “sar -u 1 3″ but displays additional fields.

Individual CPU Usage (sar -P)

Todays System has many CPU cores available,by using the sar utility we can find the information regarding every CPU core available like,

rootLocal:root002-~ $ sar -P ALL 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:14:33 PM CPU %user %nice %system %iowait %steal %idle
10:14:34 PM all 1.00 7.66 0.25 0.00 0.00 91.09
10:14:34 PM 0 0.00 0.00 0.00 0.00 0.00 100.00
10:14:34 PM 1 0.99 0.00 0.00 0.00 0.00 99.01
10:14:34 PM 2 8.00 0.00 0.00 0.00 0.00 92.00
10:14:34 PM 3 0.00 9.09 0.00 0.00 0.00 90.91
10:14:34 PM 4 0.99 16.83 0.99 0.00 0.00 81.19
10:14:34 PM 5 1.00 13.00 1.00 0.00 0.00 85.00
10:14:34 PM 6 0.98 37.25 0.98 0.00 0.00 60.78
10:14:34 PM 7 0.00 0.00 0.00 0.00 0.00 100.00
10:14:34 PM 8 0.00 0.00 0.00 0.00 0.00 100.00
10:14:34 PM 9 1.00 0.00 0.00 0.00 0.00 99.00
10:14:34 PM 10 0.00 0.00 0.99 0.00 0.00 99.01
10:14:34 PM 11 1.00 15.00 0.00 0.00 0.00 84.00

In the above example under “CPU” column we can see 0, 1, 2, and 3 indicates the corresponding CPU core numbers. so we have 12 core CPU available.

Here are some of the Other ways of using the sar -P

sar -P ALL : Displays CPU usage broken down by all cores for the current day.
sar -P ALL 1 3 : Displays real time CPU usage for ALL cores every 1 second for 3 times
sar -P 1 : Displays CPU usage for core number 1 for the current day.
sar -P 1 1 3 : Displays real time CPU usage for core number 1, every 1 second for 3 times.

Memory and Swap(sar -r)
Sar command can also be used in finding the memory usage like,
rootLocal:root002-~ $ sar -r 1 2
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:20:27 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
10:20:28 PM 796732 65034268 98.79 2836608 21328880 4194296 0 0.00 0
10:20:29 PM 796780 65034220 98.79 2836608 21328896 4194296 0 0.00 0

Average: 796756 65034244 98.79 2836608 21328888 4194296 0 0.00 0

Here are some of the Other ways of using the sar -r
sar -r : Display Complete memory details
sar -r 1 3 : Memory Details for every 1 second for 3 times.

Swap Usage (sar -W)

rootLocal:root002-~ $ sar -W
Linux 2.6.18-348.4.1.el5 (omhq1970) 08/18/2013

12:00:01 AM pswpin/s pswpout/s
12:10:06 AM 0.00 0.00
12:20:01 AM 0.00 0.00
12:30:01 AM 0.00 0.00
12:40:01 AM 0.00 0.00
12:50:01 AM 0.00 0.00
01:00:01 AM 0.00 0.00

Memory In pages (sar -R)
Sar command can also help in obtaining details in pages like,

rootLocal:root002-~ $ sar -R 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:40:53 PM frmpg/s bufpg/s campg/s
10:40:54 PM -54.46 0.00 10.89
Average: -54.46 0.00 10.89

This can be used to identify number of memory pages freed, used, and cached per second by the system.

Here are some of the Other ways ,
Use “sar -H” to identify the hugepages (in KB) that are used and available.
Use “sar -B” to generate paging statistics. i.e Number of KB paged in (and out) from disk per second.
Use “sar -W” to generate page swap statistics. i.e Page swap in (and out) per second.

I/O activities (sar -b)
sar -b” allows us to find the I/O statistics like,

rootLocal:root002-~ $ sar -b 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013
10:48:37 PM tps rtps wtps bread/s bwrtn/s
10:48:38 PM 71.00 0.00 71.00 0.00 664.00
Average: 71.00 0.00 71.00 0.00 664.00

The columns say
tps – Transactions per second (this includes both read and write)
rtps – Read transactions per second
wtps – Write transactions per second
bread/s – Bytes read per second
bwrtn/s – Bytes written per second

Individual Device Activities (sar -d)
Sar command can also be used in identifying various information regarding individual devices like mount point , partition etc

rootLocal:root002-~ $ sar -d 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:51:12 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
10:51:13 PM dev8-0 22.22 0.00 454.55 20.45 0.00 0.00 0.00 0.00
10:51:13 PM dev8-2 22.22 0.00 454.55 20.45 0.00 0.00 0.00 0.00
10:51:13 PM dev253-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

DEV column indicates the device.For example: “dev253-0″ means a block device with 253 as major number, and 0 as minor number.

The DEV can actually be a device name like sda , sda1 etc like

rootLocal:root002-~ $ sar -p -d 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:53:12 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
10:53:13 PM sda 27.72 0.00 639.60 23.07 0.01 0.54 0.18 0.50
10:53:13 PM sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:53:13 PM sda2 27.72 0.00 639.60 23.07 0.01 0.54 0.18 0.50

Context Switch (sar -w)
rootLocal:root002-~ $ sar -w 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:55:13 PM cswch/s
10:55:14 PM 11051.49
Average: 11051.49

Sar command can also be used to find how many context switches are happening.The above output says how many process are being created and how many context switches happened for a Second.

Run Queue and Load Average(sar -q)
Run Queue and Load average can also be obtained using the sar commnad like,

rootLocal:root002-~ $ sar -q 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

10:58:59 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
10:59:00 PM 3 7444 0.26 0.66 1.68
Average: 3 7444 0.26 0.66 1.68

Network Statistics(sar -n)
Network statistics can also be obtained using “sar -n”.This works in a different way. The Syntax is
sar -n KEYWORD
We need to pass a KEYWORD for obtaining the Details. The Available keywords are

DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
ALL – This displays all of the above information

rootLocal:root002-~ $ sar -n DEV 1 1
Linux 2.6.18-348.4.1.el5 (rootLocal) 08/18/2013

11:04:08 PM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
11:04:09 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:04:09 PM eth0 0.00 408.00 0.00 50481.00 0.00 0.00 0.00
11:04:09 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:04:09 PM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00

File System Usage (sar -v)
File System details can also be obtained using ,

05:00:01 PM dentunusd file-sz inode-sz super-sz %super-sz dquot-sz %dquot-sz rtsig-sz %rtsig-sz
05:10:01 PM 192010 1530 194516 0 0.00 0 0.00 0 0.00
05:20:01 PM 192010 1530 194516 0 0.00 0 0.00 0 0.00
05:30:01 PM 191997 1530 194500 0 0.00 0 0.00 0 0.00

sar -v” to display number of inode handlers, file handlers, and pseudo-terminals used by the system.

TTY activity (sar -y)
Using sar command we can find various information regarding the TTY like

[root@vx111a soa]# sar -y | more
Linux 2.6.32-131.0.15.el6.x86_64 (vx111a.jas.com) 08/19/2013 _x86_64_ (4 CPU)

12:00:01 AM TTY rcvin/s xmtin/s framerr/s prtyerr/s brk/s ovrun/s
12:10:01 AM 0 0.00 0.00 0.00 0.00 0.00 0.00
12:10:01 AM 1 0.00 0.00 0.00 0.00 0.00 0.00
12:20:01 AM 0 0.00 0.00 0.00 0.00 0.00 0.00
12:20:01 AM 1 0.00 0.00 0.00 0.00 0.00 0.00

Historical Data With SAR

Sar also provides a way to parse historical data, I.e data regarding the system that is stored for previous days.

Usually The system information like CPU,Memory , I/O details that are parsed by Sar command are stored in /var/log/sa/ location.if we go to this location we can see various files like

sa01
sa02
sa03
sa04
sa05
sa06
sa07
sa30

They are created as sadd  where dd is a numeric value for the day of the week (starting at 01).
The file from the current day is Used.We can restrict the time stamps by using the -s and -e parmaters

If you need to access the data of any Day , we can invoke the sar command as
sar -f /var/log/sysstat/sa04


sar -u -f /var/log/sa/sa10 : Displays CPU usage for the 10day of the month from the sa10 file.
sar -P ALL -f /var/log/sa/sa10 : Displays CPU usage broken down by all cores for the 10day day of the month from sa10 file.
Few Other Uses
sar -r -f /var/log/sa/sa10
sar -b -f /var/log/sa/sa10
sar -d -f /var/log/sa/sa10
sar -w -f /var/log/sa/sa10
sar -q -f /var/log/sa/sa10

sar -s 09:00:00 -e 10:30:00

Between Time Using SAR(sar -s and -e)
Using -s (start) and -e (end) options you can retrieve data from past days.If you want to get information a few days in the past. For example if you want data from 12:00 to 12:30 your syntax would be:

Dev:rootLocal:root002-~ $ sar -s 12:00:00 -e 12:30:00
Linux 2.6.18-348.el5xen (vx1379) 08/18/2013
12:00:01 PM CPU %user %nice %system %iowait %steal %idle
12:10:01 PM all 0.49 0.00 0.19 1.09 0.13 98.10
12:20:01 PM all 0.58 0.00 0.17 0.82 0.12 98.31
Average: all 0.54 0.00 0.18 0.95 0.13 98.21

You also can add all of the normal sar options when pulling from past logfiles, so you could run the same command and add the -r argument to get RAM statistics:
$ sar -s 17:00:00 -e 17:30:00 -f /var/log/sysstat/sa01 -r

Collecting System Activity Data using SAR at any time
Using Sar we can collect the system activity to a data file using

sar -u -o ./datafile 2 3

The -u option specifies our interest in the CPU subsystem. The -o option will create an output file that contains binary data. Finally, we will take 3 samples at two-second intervals. Upon completion of the sampling, sar will report the results to the screen. This provides us with a snapshot of current system activity.

To read the datafile that we created we can use
Sar -u -f ./datafile

Display Data In Multiple Formats
[root@vx111a ~]# sadf -d ./datafile -- -B
# hostname;interval;timestamp;pgpgin/s;pgpgout/s;fault/s;majflt/s;pgfree/s;pgscank/s;pgscand/s;pgsteal/s;%vmeff
vx111a.jas.com;2;2013-08-19 09:50:22 UTC;0.00;25.00;56.25;0.00;442.05;0.00;0.00;0.00;0.00
vx111a.jas.com;2;2013-08-19 09:50:24 UTC;0.00;20.22;28.09;0.00;900.56;0.00;0.00;0.00;0.00
vx111a.jas.com;2;2013-08-19 09:50:26 UTC;0.00;18.80;255.56;0.00;1098.72;0.00;0.00;0.00;0.00
9642534599

Keep Files For More Than 7 Days
To make SAR track data for more than 7 days, simply change the configuration file:

[root@example ~]# vim /etc/sysconfig/sysstat
HISTORY=7 # How long to keep log files (days), maximum is a month

Happy learning , More To Come :-)