sar
To know about all options of sar, best place is : $ man sarksar
I have created a page on wikipedia : http://en.wikipedia.org/wiki/Ksar_%28Unix_sar_grapher%29
Few points
Always create 1 second interval data for troubleshooting.- below command will run in background to create binary file having 1 second resolution
- binary file will be around 25M
- text file size will be 40M
sar -A -o /var/tmp/sar-bin-date-systemname.bin 1 3600 >/dev/null 2>&1 &
ls -l /var/tmp/sar-bin-date-systemname.bin
sar -A -f /var/tmp/sar-bin-date-systemname.bin
LC_ALL=C sar -A -p -f /var/tmp/sar-bin-date-systemname.bin /var/tmp/sar-txt-date-systemname.txt
Load sar-txt-date-systemname.txt in ksar to see granular details.
An example:
Application was loading a 3.5G file on a 4 GB system. It was being closed within 10 second but it was enough to generate huge page-out and ~100% utilization of disk !
$ sar -rpdB -s 02:27:29 -e 02:27:45 -f /var/tmp/sar-bin-date-systemname.bin
Linux 2.6.18-308.8.2.el5 (testsys001mysystem.com) 10/03/15 _x86_64_ (2 CPU)
02:27:28 pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
02:27:29 16.16 1826.26 239842.42 1.01 115282.83 0.00 0.00 0.00 0.00
02:27:30 60.00 56.00 331109.00 8.00 204665.00 0.00 0.00 0.00 0.00
02:27:31 0.00 60.00 517659.00 0.00 345388.00 0.00 0.00 0.00 0.00
02:27:32 0.00 327.27 610073.74 0.00 364861.62 0.00 0.00 0.00 0.00
02:27:33 0.00 60.61 330316.16 0.00 181757.58 0.00 0.00 0.00 0.00
02:27:34 0.00 526.53 590945.92 0.00 429371.43 0.00 0.00 0.00 0.00
02:27:35 16.00 1204.00 340554.00 0.00 232284.00 2272.00 4832.00 1080.00 15.20
02:27:36 252.00 704.00 60046.00 4.00 58837.00 3296.00 19806.00 17118.00 74.10
02:27:37 1706.93 334978.22 302981.19 71.29 304998.02 141251.49 44208.91 94489.11 50.95
02:27:38 26574.76 459071.84 349949.51 567.96 359980.58 183487.38 46633.01 115279.61 50.10
02:27:39 7526.53 446142.86 330251.02 262.24 328176.53 198889.80 24097.96 110475.51 49.54
02:27:40 1503.85 413742.31 292217.31 59.62 292017.31 181600.00 24369.23 102852.88 49.94
02:27:41 1619.80 81607.92 56802.97 43.56 979036.63 32285.15 8807.92 20769.31 50.54
02:27:42 3155.56 113.13 8978.79 82.83 1768.69 0.00 0.00 0.00 0.00
02:27:43 13457.43 190.10 23859.41 630.69 13454.46 0.00 0.00 0.00 0.00
02:27:44 1296.97 367.68 128.28 44.44 109.09 0.00 0.00 0.00 0.00
02:27:45 1979.59 36.73 7173.47 36.73 890.82 0.00 0.00 0.00 0.00
Average: 3528.90 103975.52 258296.12 107.83 248765.63 44355.97 10310.59 27579.75 50.45
02:27:28 kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
02:27:29 3626448 286212 7.32 14448 109424 587200 9.88
02:27:30 3203408 709252 18.13 14460 109464 980300 16.49
02:27:31 2509876 1402784 35.85 14468 109472 1670800 28.11
02:27:32 1558540 2354120 60.17 14480 109516 2681920 45.12
02:27:33 1002236 2910424 74.38 14492 109480 3295636 55.44
02:27:34 402404 3510256 89.72 14508 109512 3777916 63.56
02:27:35 22300 3890360 99.43 14496 105340 4261476 71.69
02:27:36 22272 3890388 99.43 2832 52344 4266140 71.77
02:27:37 22888 3889772 99.42 520 12892 4640740 78.07
02:27:38 32216 3880444 99.18 872 25420 5006940 84.23
02:27:39 30604 3882056 99.22 1300 27428 5464516 91.93
02:27:40 35468 3877192 99.09 1316 28580 5850808 98.43
02:27:41 3759396 153264 3.92 1336 29276 478424 8.05
02:27:42 3744892 167768 4.29 1548 30996 551996 9.29
02:27:43 3743188 169472 4.33 1812 41488 446000 7.50
02:27:44 3742168 170492 4.36 1840 42508 446000 7.50
02:27:45 3733504 179156 4.58 1936 44268 511600 8.61
Average: 1834812 2077848 53.11 6863 64553 2642260 44.45
02:27:29 sda 56.57 32.32 3652.53 65.14 0.19 3.39 0.61 3.43
02:27:30 sda 11.00 120.00 112.00 21.09 0.04 4.00 4.00 4.40
02:27:31 sda 3.00 0.00 120.00 40.00 0.00 0.33 0.33 0.10
02:27:32 sda 3.03 0.00 80.81 26.67 0.01 2.00 2.00 0.61
02:27:33 sda 6.06 0.00 694.95 114.67 0.00 0.33 0.33 0.20
02:27:34 sda 22.45 0.00 1053.06 46.91 0.02 0.95 0.23 0.51
02:27:35 sda 207.00 32.00 2408.00 11.79 0.24 1.15 0.17 3.60
02:27:36 sda 45.00 504.00 1248.00 38.93 0.38 8.49 3.62 16.30
02:27:37 sda 2812.87 3413.86 666899.01 238.30 7.28 2.58 0.29 81.98
02:27:38 sda 3228.16 53141.75 919541.75 301.31 20.51 6.35 0.30 97.96
02:27:39 sda 4312.24 14995.92 888693.88 209.56 11.21 2.59 0.23 100.10
02:27:40 sda 8043.27 3069.23 832607.69 103.90 13.67 1.70 0.12 97.60
02:27:41 sda 899.01 3239.60 163215.84 185.15 1.58 1.75 0.32 28.51
02:27:42 sda 225.25 6044.44 226.26 27.84 0.41 1.80 1.11 24.95
02:27:43 sda 699.01 27176.24 380.20 39.42 1.44 2.08 1.20 83.76
02:27:44 sda 72.73 2585.86 735.35 45.67 0.12 1.69 1.54 11.21
02:27:45 sda 89.80 3967.35 73.47 45.00 0.15 1.66 1.26 11.33
Average: sda 1242.32 7057.80 207951.03 173.07 3.43 2.76 0.27 33.71
watcher-cron
A simple script provided by redhat to collect system performance data using below commands. Use this along with sar data. See redhat article https://access.redhat.com/articles/279063 for more detail.iostat
vmstat
top
mpstat
Also add iotop and ps aux --forest commands in script:
sequences=`seq -s ' ' $2`
eval "for i in $sequences; do date;ps aux --forest; sleep $1;done ${_compress_all} > $3/${_time}-info-ps-forest.log${_ext_all} &" ; pspid=$!
eval "iotop -n $1 -d $2 -b ${_compress_all} > $3/${_time}-info-iotop.log${_ext_all} &" ; iotoppid=$!
References:
https://access.redhat.com/articles/1145953 - PCP
Performance monitoring tools
No comments:
Post a Comment