Steps to use OProfile - opcontrol, opreport, oparchive and a script to collect samples

What is OProfile ?

From OProfile documentation ''OProfile is capable of monitoring native hardware events occurring in all parts of a running system, from the kernel (including modules and interrupt handlers) to shared libraries to binaries. OProfile can collect event information for the whole system in the background with very little overhead. These features make it ideal for monitoring entire systems to determine bottle necks in real-world systems."

Installation

OProfile require kernel-debuginfo rpm of running kernel version to install uncompressed and unstripped vmlinux binary. It is in /usr/lib/debug/lib/modules/$(uname -r)/vmlinux.

  # yum install kernel-debuginfo-$(uname -r) kernel-debuginfo-common-$(uname -r) kernel-headers-$(uname -r) kernel-devel-$(uname -r)
  # yum install oprofile oprofile-jit
  # ls -l /usr/lib/debug/lib/modules/$(uname -r)/vmlinux
  # mount -t debugfs nodev /sys/kernel/debug

How to use OProfile?

- Stop OProfile completely, incase anything running
    # opcontrol --deinit

- Enable stats and start the oprofile daemon. It will load oprofile kernel module
    # opcontrol --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux
    #lsmod |grep oprofile

- Initialize and start daemon with default sample collection directory ''samples" in default session-dir /var/lib/oprofile. It will also create empty directory /var/lib/oprofile/samples/current.
   # opcontrol --init
   # opcontrol --start-daemon
          Using default event: CPU_CLK_UNHALTED:100000:0:1:1
          Error: counter 0 not available nmi_watchdog using this resource ? Try:

   # opcontrol --deinit
   # echo 0 > /proc/sys/kernel/nmi_watchdog
   # opcontrol --status
         Daemon running: pid 31084
         Separate options: none
         vmlinux file: /usr/lib/debug/lib/modules/2.6.18-348.3.1.el5/vmlinux
         Image filter: none
         Call-graph depth: 0

- Start to capture a profile with default events (cpu cycles):
    # opcontrol --start

- Periodically dump and save stats
   # cd /var/lib/oprofile/samples
   # time_now=$(date +%F_%Hh%Mm%Ss)
   # /usr/bin/opcontrol --dump
   # /usr/bin/opcontrol --save=$time_now

- Check if saved session is valid
  #opreport -fg -l /usr/lib/debug/lib/modules/`uname -r`/vmlinux session:$time_now

- Create oparchive (using one or more sessions) for analysis on remote system. It will create oparchive-$time_now directory.
   # oparchive -o  oparchive-$time_now  session:$time_now
  # oparchive -o  oparchive-$time_now  session:$time_now,,

- Check if created archive is also healthy and create report.
    # opreport -fg -l archive:oparchive-$time_now session:$time_now >opreport-from-oparchive-${time_now}.txt 2>&1

- Saving session will stop profiling. Hence, start profiling again
  # /usr/bin/opcontrol --start

Profiling using cron

Periodically, run following thru cron once oprofile daemon has been started  (--start-daemon). Or  use below script.

   opcontrol --dump; opcontrol --save=`date +%F_%Hh%Mm%Ss`; opcontrol --start

OProfile on VM

OProfile currently does not support event-based profiling (i.e, using hardware events like cache misses, branch mispredicts) on virtual machine guests running under systems such as VMware. (Note: KVM guests are supported.). See http://blogs.epfl.ch/category/3239 to run OProfile on VMware VM and OProfile officila document.

Script to collect OProfile samples

Assuming you are copying script as /var/tmp/oprofile-cron-script.sh and sample oprofile base data collection directory is /var/lib/oprofile/new. Script will create log /var/lib/oprofile/new/samples/oprofiled.log and /var/lib/oprofile/new/samples/oprofile-script-$(date +%d%b%Y).log

OProfile cron job

Sample data in every 30 minutes and remove

# cat /etc/cron.d/oprofile
*/30 * * * * root  /var/tmp/oprofile-cron-script.sh
0 * * * * root find /var/lib/oprofile/new/samples -maxdepth 1 -mtime +3 -exec rm -rf {} \;

OProfile sample collection script

# cat /var/tmp/oprofile-cron-script.sh
#!/bin/bash
# nasimuddin ansari : nasim.ansari (at) gmail.com
# 16 Feb 2015, version 2, Tested in Production environment
working_dir=/var/lib/oprofile/new
sample_dir=$working_dir/samples
script_log=$sample_dir/oprofile-script-$(date +%d%b%Y).log
time_now=$(date +%F_%Hh%Mm%Ss)
PATH=$PATH:/usr/bin
exit_status=0
#set -x

[ ! -d "$working_dir" ] && echo Failed: $working_dir does not exist && exit 1
[ ! -d "$working_dir/samples" ] && mkdir $working_dir/samples
[ ! -f "$script_log" ] || touch $script_log

function reinitialize_oprofile {
    echo -e "\nReinitializing oprofile ...."
    opcontrol --deinit
    pgrep -f /usr/bin/oprofiled && pkill -f /usr/bin/oprofiled
    opcontrol --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux --session-dir=$working_dir
    opcontrol --start-daemon --session-dir=$working_dir
    # start capture a profile - with default events (cpu cycles)
    opcontrol --start
    opcontrol --status
    # for each of detection, when oprofile was reinitiatlize, touch a file
    touch $sample_dir/reinitialize_oprofile-$time_now
    sleep 60
}

(
echo -e "\nScript $0 started on $(date)"
cd $sample_dir

# if oprofiled is not running, restart it
if ! pgrep -f /usr/bin/oprofiled  >/dev/null
then
    echo oprofiled is not running ...
    reinitialize_oprofile
    if ! pgrep -f /usr/bin/oprofiled  >/dev/null
    then
        echo Failed: Tried to start oprofile, but could not. Please fix it !
    echo Exiting ....
        exit 1
    fi
fi

# if oprofile is running, dump and save stats
opcontrol --dump
opcontrol --save=$time_now

# above save session will create a directory as $time_now
if [ -d "$time_now" ]; then

    # run opreport on saved session to confirm it is good
    if opreport -fg -l session:$time_now --session-dir=$working_dir >/dev/null 2>&1
    then
        echo OK: saved session $time_now is good, start capturing new profile
    if opcontrol --start
    then
        echo OK: 'opcontrol --start' sucess. Creating oparchive now ...
            oparchive -o oparchive-$time_now session:$time_now --session-dir=$working_dir >/dev/null 2>&1

            # if oparchive has been created in above step, create opreport
            if [ -d "oparchive-$time_now" ]; then
                echo OK: oparchive-$time_now has been created, creating opreport now ....
                opreport -fg -l archive:oparchive-$time_now --session-dir=$working_dir session:$time_now >opreport-from-oparchive-${time_now
}.txt 2>&1

                # if opreport has been created, make tarball and do housekeeping
                if [ $? = 0 ];then
            echo OK: opreport-from-oparchive-${time_now}.txt has been created, creating tarball now ...
                    # create a tar file of session, archive and report
                    tar -jcf oprofile-oparchive-opreport-${time_now}.tar.bz2 \
                        oparchive-$time_now $time_now opreport-from-oparchive-${time_now}.txt
                    rm -rf oparchive-$time_now $time_now opreport-from-oparchive-${time_now}.txt
                    if [ -f oprofile-oparchive-opreport-${time_now}.tar.bz2 ];then
                      echo OK: Created oprofile-oparchive-opreport-${time_now}.tar.bz2 file having session, archive and report
                    else
                      echo Failed: to create oprofile-oparchive-opreport-${time_now}.tar.bz2 file session, archive and report
                      exit_status=1
                    fi
                else
                    echo Failed: to create opreport using archived and session
                    exit_status=1
                fi
            else
                echo Failed: oparchive did not create "$sample_dir/$oparchive-$time_now" directory. Hence opreport has failed.
                exit_status=1
            fi
    else
        echo Failed: 'opcontrol --start'. Please fix it !
        exit_status=1
    fi
    else
        echo Failed: "opreport -fg -l session:$time_now" did not work. Hence oparchive too has failed.
        exit_status=1
    fi
else
    echo Failed: "opcontrol --save=$time_now" did not create a valid session. skipping oparchive and opreport. Fix it please !
    exit_status=1
fi


if [ $exit_status = 1 ]; then
    reinitialize_oprofile
    echo -e "Script $0 completed at $(date) Done....\n"
    exit 1
else
    echo -e "Script $0 completed at $(date) Done....\n"
    exit 0
fi

) >>$script_log 2>&1
#set +x
#end of script

Sample output of above script

 # ls -lrt /var/lib/oprofile/new/samples

-rw-r--r-- 1 root root        0 Feb 25 06:31 reinitialize_oprofile-2015-02-25_06h30m01s

-rw-r--r-- 1 root root        0 Feb 25 06:45 reinitialize_oprofile-2015-02-25_06h45m43s

-rw-r--r-- 1 root root 75284555 Feb 26 02:00 oprofile-oparchive-opreport-2015-02-26_02h00m01s.tar.bz2

-rw-r--r-- 1 root root 78830219 Feb 26 02:30 oprofile-oparchive-opreport-2015-02-26_02h30m01s.tar.bz2

drwxr-xr-x 4 root root     4096 Feb 26 03:00 current

-rw-r--r-- 1 root root 74807618 Feb 26 03:00 oprofile-oparchive-opreport-2015-02-26_03h00m01s.tar.bz2

-rw-r--r-- 1 root root     4613 Feb 26 03:00 oprofile-script-26Feb2015.log

-rw-r--r-- 1 root root  8355537 Feb 26 03:15 oprofiled.log

References

1. Documentation of oprofile tools : http://oprofile.sourceforge.net/doc/index.html
2. Redhat :OProfile quick start cheat sheet for RHEL5 : https://access.redhat.com/articles/21842
3. IBM : Getting started with OProfile : http://www-01.ibm.com/support/knowledgecenter/linuxonibm/liacf/oprofgetstart.htm
4. Blog : Profiling Tools - ftrace, perf, and oprofile : http://kwangwoo.blogspot.sg/2011/07/profiling-tools-ftrace-perf-and.html
5. How to use OProfile : https://www.nsnam.org/wiki/HOWTO_use_oprofile
6. OProfile on VMware VM : http://blogs.epfl.ch/category/3239
7. Redhat : Tuning with OProfile : http://people.redhat.com/wcohen/OProfileTutorial.txt