What is OProfile ?
From OProfile documentation ''OProfile is capable of monitoring native hardware events occurring in all parts of a running system, from the kernel (including modules and interrupt handlers) to shared libraries to binaries. OProfile can collect event information for the whole system in the background with very little overhead. These features make it ideal for monitoring entire systems to determine bottle necks in real-world systems."Installation
OProfile require kernel-debuginfo rpm of running kernel version to install uncompressed and unstripped vmlinux binary. It is in /usr/lib/debug/lib/modules/$(uname -r)/vmlinux.# yum install kernel-debuginfo-$(uname -r) kernel-debuginfo-common-$(uname -r) kernel-headers-$(uname -r) kernel-devel-$(uname -r)
# yum install oprofile oprofile-jit
# ls -l /usr/lib/debug/lib/modules/$(uname -r)/vmlinux
# mount -t debugfs nodev /sys/kernel/debug
How to use OProfile?
- Stop OProfile completely, incase anything running# opcontrol --deinit
- Enable stats and start the oprofile daemon. It will load oprofile kernel module
# opcontrol --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux
#lsmod |grep oprofile
- Initialize and start daemon with default sample collection directory ''samples" in default session-dir /var/lib/oprofile. It will also create empty directory /var/lib/oprofile/samples/current.
# opcontrol --init
# opcontrol --start-daemon
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Error: counter 0 not available nmi_watchdog using this resource ? Try:
# opcontrol --deinit
# echo 0 > /proc/sys/kernel/nmi_watchdog
# opcontrol --status
Daemon running: pid 31084
Separate options: none
vmlinux file: /usr/lib/debug/lib/modules/2.6.18-348.3.1.el5/vmlinux
Image filter: none
Call-graph depth: 0
# opcontrol --start
- Periodically dump and save stats
# cd /var/lib/oprofile/samples
# time_now=$(date +%F_%Hh%Mm%Ss)
# /usr/bin/opcontrol --dump
# /usr/bin/opcontrol --save=$time_now
- Check if saved session is valid
#opreport -fg -l /usr/lib/debug/lib/modules/`uname -r`/vmlinux session:$time_now
- Create oparchive (using one or more sessions) for analysis on remote system. It will create oparchive-$time_now directory.
# oparchive -o oparchive-$time_now session:$time_now
# oparchive -o oparchive-$time_now session:$time_now,
- Check if created archive is also healthy and create report.
# opreport -fg -l archive:oparchive-$time_now session:$time_now >opreport-from-oparchive-${time_now}.txt 2>&1
- Saving session will stop profiling. Hence, start profiling again
# /usr/bin/opcontrol --start
Profiling using cron
Periodically, run following thru cron once oprofile daemon has been started (--start-daemon). Or use below script.
opcontrol --dump; opcontrol --save=`date +%F_%Hh%Mm%Ss`; opcontrol --start
OProfile on VM
OProfile currently does not support event-based profiling (i.e, using hardware events like cache misses, branch mispredicts) on virtual machine guests running under systems such as VMware. (Note: KVM guests are supported.). See http://blogs.epfl.ch/category/3239 to run OProfile on VMware VM and OProfile officila document.Script to collect OProfile samples
Assuming you are copying script as /var/tmp/oprofile-cron-script.sh and sample oprofile base data collection directory is /var/lib/oprofile/new. Script will create log /var/lib/oprofile/new/samples/oprofiled.log and /var/lib/oprofile/new/samples/oprofile-script-$(date +%d%b%Y).logOProfile cron job
Sample data in every 30 minutes and remove# cat /etc/cron.d/oprofile
*/30 * * * * root /var/tmp/oprofile-cron-script.sh
0 * * * * root find /var/lib/oprofile/new/samples -maxdepth 1 -mtime +3 -exec rm -rf {} \;
OProfile sample collection script
# cat /var/tmp/oprofile-cron-script.sh#!/bin/bash
# nasimuddin ansari : nasim.ansari (at) gmail.com
# 16 Feb 2015, version 2, Tested in Production environment
working_dir=/var/lib/oprofile/new
sample_dir=$working_dir/samples
script_log=$sample_dir/oprofile-script-$(date +%d%b%Y).log
time_now=$(date +%F_%Hh%Mm%Ss)
PATH=$PATH:/usr/bin
exit_status=0
#set -x
[ ! -d "$working_dir" ] && echo Failed: $working_dir does not exist && exit 1
[ ! -d "$working_dir/samples" ] && mkdir $working_dir/samples
[ ! -f "$script_log" ] || touch $script_log
function reinitialize_oprofile {
echo -e "\nReinitializing oprofile ...."
opcontrol --deinit
pgrep -f /usr/bin/oprofiled && pkill -f /usr/bin/oprofiled
opcontrol --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux --session-dir=$working_dir
opcontrol --start-daemon --session-dir=$working_dir
# start capture a profile - with default events (cpu cycles)
opcontrol --start
opcontrol --status
# for each of detection, when oprofile was reinitiatlize, touch a file
touch $sample_dir/reinitialize_oprofile-$time_now
sleep 60
}
(
echo -e "\nScript $0 started on $(date)"
cd $sample_dir
# if oprofiled is not running, restart it
if ! pgrep -f /usr/bin/oprofiled >/dev/null
then
echo oprofiled is not running ...
reinitialize_oprofile
if ! pgrep -f /usr/bin/oprofiled >/dev/null
then
echo Failed: Tried to start oprofile, but could not. Please fix it !
echo Exiting ....
exit 1
fi
fi
# if oprofile is running, dump and save stats
opcontrol --dump
opcontrol --save=$time_now
# above save session will create a directory as $time_now
if [ -d "$time_now" ]; then
# run opreport on saved session to confirm it is good
if opreport -fg -l session:$time_now --session-dir=$working_dir >/dev/null 2>&1
then
echo OK: saved session $time_now is good, start capturing new profile
if opcontrol --start
then
echo OK: 'opcontrol --start' sucess. Creating oparchive now ...
oparchive -o oparchive-$time_now session:$time_now --session-dir=$working_dir >/dev/null 2>&1
# if oparchive has been created in above step, create opreport
if [ -d "oparchive-$time_now" ]; then
echo OK: oparchive-$time_now has been created, creating opreport now ....
opreport -fg -l archive:oparchive-$time_now --session-dir=$working_dir session:$time_now >opreport-from-oparchive-${time_now
}.txt 2>&1
# if opreport has been created, make tarball and do housekeeping
if [ $? = 0 ];then
echo OK: opreport-from-oparchive-${time_now}.txt has been created, creating tarball now ...
# create a tar file of session, archive and report
tar -jcf oprofile-oparchive-opreport-${time_now}.tar.bz2 \
oparchive-$time_now $time_now opreport-from-oparchive-${time_now}.txt
rm -rf oparchive-$time_now $time_now opreport-from-oparchive-${time_now}.txt
if [ -f oprofile-oparchive-opreport-${time_now}.tar.bz2 ];then
echo OK: Created oprofile-oparchive-opreport-${time_now}.tar.bz2 file having session, archive and report
else
echo Failed: to create oprofile-oparchive-opreport-${time_now}.tar.bz2 file session, archive and report
exit_status=1
fi
else
echo Failed: to create opreport using archived and session
exit_status=1
fi
else
echo Failed: oparchive did not create "$sample_dir/$oparchive-$time_now" directory. Hence opreport has failed.
exit_status=1
fi
else
echo Failed: 'opcontrol --start'. Please fix it !
exit_status=1
fi
else
echo Failed: "opreport -fg -l session:$time_now" did not work. Hence oparchive too has failed.
exit_status=1
fi
else
echo Failed: "opcontrol --save=$time_now" did not create a valid session. skipping oparchive and opreport. Fix it please !
exit_status=1
fi
if [ $exit_status = 1 ]; then
reinitialize_oprofile
echo -e "Script $0 completed at $(date) Done....\n"
exit 1
else
echo -e "Script $0 completed at $(date) Done....\n"
exit 0
fi
) >>$script_log 2>&1
#set +x
#end of script
Sample output of above script
# ls -lrt /var/lib/oprofile/new/samples
-rw-r--r-- 1 root root 0 Feb 25 06:31 reinitialize_oprofile-2015-02-25_06h30m01s
-rw-r--r-- 1 root root 0 Feb 25 06:45 reinitialize_oprofile-2015-02-25_06h45m43s
-rw-r--r-- 1 root root 75284555 Feb 26 02:00 oprofile-oparchive-opreport-2015-02-26_02h00m01s.tar.bz2
-rw-r--r-- 1 root root 78830219 Feb 26 02:30 oprofile-oparchive-opreport-2015-02-26_02h30m01s.tar.bz2
drwxr-xr-x 4 root root 4096 Feb 26 03:00 current
-rw-r--r-- 1 root root 74807618 Feb 26 03:00 oprofile-oparchive-opreport-2015-02-26_03h00m01s.tar.bz2
-rw-r--r-- 1 root root 4613 Feb 26 03:00 oprofile-script-26Feb2015.log
-rw-r--r-- 1 root root 8355537 Feb 26 03:15 oprofiled.log
References
- Documentation of oprofile tools : http://oprofile.sourceforge.net/doc/index.html
- Redhat :OProfile quick start cheat sheet for RHEL5 : https://access.redhat.com/articles/21842
- IBM : Getting started with OProfile : http://www-01.ibm.com/support/knowledgecenter/linuxonibm/liacf/oprofgetstart.htm
- Blog : Profiling Tools - ftrace, perf, and oprofile : http://kwangwoo.blogspot.sg/2011/07/profiling-tools-ftrace-perf-and.html
- How to use OProfile : https://www.nsnam.org/wiki/HOWTO_use_oprofile
- OProfile on VMware VM : http://blogs.epfl.ch/category/3239
- Redhat : Tuning with OProfile : http://people.redhat.com/wcohen/OProfileTutorial.txt
