43.5. Analyzing the Data
Periodically, the OProfile daemon, oprofiled collects the samples and writes them to the /var/lib/oprofile/samples/ directory. Before reading the data, make sure all data has been written to this directory by executing the following command as root:
opcontrol --dump |
Each sample file name is based on the name of the executable, with a closing curly bracket (}) replacing each forward slash (/). The file name ends with a hash mark (#), followed by the counter number used for that sample file. For example, the following file includes the sample data for the /sbin/syslogd executable collected with counter 0:
}sbin}syslogd#0 |
The following tools are available to profile the sample data once it has been collected:
op_time
oprofpp
op_to_source
op_merge
Use these tools, along with the binaries profiled, to generate reports that can be further analyzed.
 | Warning |
|---|---|
The executable being profiled must be used with these tools to analyze the data. If it must change after the data is collected, backup the executable used to create the samples as well as the sample files. |
Samples for each executable are written to a single sample file. Samples from each dynamically linked library are also written to a single sample file. While OProfile is running, if the executable being monitored changes and a sample file for the executable exists, the existing sample file is automatically deleted. Thus, if the existing sample file is needed, it must be backed up, along with the executable used to create it before replacing the executable with a new version. Refer to Section 43.4 Saving Data for details on how to backup the sample file.
43.5.1. Using op_time
The op_time tool provides an overview of all the executables being profiled.
The following is part of an example output:
581 0.2949 0.0000 /usr/bin/oprofiled 966 0.4904 0.0000 /usr/sbin/cupsd 1028 0.5218 0.0000 /usr/sbin/irqbalance 1187 0.6026 0.0000 /bin/bash 1480 0.7513 0.0000 /usr/bin/slocate 2039 1.0351 0.0000 /usr/lib/rpm/rpmq 6249 3.1722 0.0000 /usr/X11R6/bin/XFree86 8842 4.4885 0.0000 /bin/sed 31342 15.9103 0.0000 /usr/bin/gdmgreeter 58283 29.5865 0.0000 /no-vmlinux 82853 42.0591 0.0000 /usr/bin/perl |
Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is unused, and the fourth is the name of the executable.
Refer to the op_time man page for a list of available command line options such as the -r option used to sort the output from the executable with the largest number of samples to the one with the smallest number of samples. The -c option is also useful for specifying a counter number.
43.5.2. Using oprofpp
To retrieve more detailed information about a specific executable, use oprofpp:
oprofpp <mode> <executable> |
<executable> must be the full path to the executable to be analyzed. <mode> must be one of the following:
- -l
List sample data by symbols. For example, the following is part of the output from running the command oprofpp -l /usr/X11R6/bin/XFree86:
vma samples % symbol name ... 08195d10 4 3.0303 miComputeCompositeClip 080b9180 5 3.78788 Dispatch 080cdce0 5 3.78788 FreeResource 080ce4a0 5 3.78788 LegalNewID 080ce640 5 3.78788 SecurityLookupIDByClass 080dd470 9 6.81818 WaitForSomething 080e1360 12 9.09091 StandardReadRequestFromClient ...
The first column is the starting virtual memory address (vma). The second column is the number of samples for the symbol. The third column is the percentage of samples for this symbol relative to the overall samples for the executable, and the fourth column is the symbol name.
To sort the output from the largest number of samples to the smallest (reverse order), use -r in conjunction with the -l option.
- -s <symbol-name>
List sample data specific to a symbol name. For example, the following output is from the command oprofpp -s StandardReadRequestFromClient /usr/X11R6/bin/XFree86:
vma samples % symbol name 080e1360 12 100 StandardReadRequestFromClient 080e1360 1 8.33333 080e137f 1 8.33333 080e13bb 1 8.33333 080e13f4 1 8.33333 080e13fb 1 8.33333 080e144a 1 8.33333 080e15aa 1 8.33333 080e1668 1 8.33333 080e1803 1 8.33333 080e1873 1 8.33333 080e190a 2 16.6667
The first line is a summary for the symbol/executable combination.
The first column consists of the virtual memory addresses sampled. The second column is the number of samples for the memory address. The third column is the percentage of samples for the memory address relative to the total number of samples for the symbol.
- -L
List sample data by symbols with more details than -l. For example:
vma samples % symbol name 08083630 2 1.51515 xf86Wakeup 08083641 1 50 080836a1 1 50 080b8150 1 0.757576 Ones 080b8179 1 100 080b8fb0 2 1.51515 FlushClientCaches 080b8fb9 1 50 080b8fba 1 50 ...
The data is the same as the -l option except that for each symbol, each virtual memory address used is shown. For each virtual memory address, the number of samples and percentage of samples relative to the number of samples for the symbol is displayed.
- -g <file-name>
Generate output to a file in gprof format. If the generated file is named gmon.out, gprof can be used to further analyze the data. Refer to the gprof man page for details.
Other options to further restrict the data are as follows:
- -f <file-name>
Use the specified sample file <file-name>. By default, the sample file in /var/lib/oprofile/samples/ is used. Use this option to specify a sample file from a previous session.
- -i <file-name>
Use <file-name> as the name of the executable for which to retrieve data.
- -d
Demangle C++ symbol names.
- -D
Demangle C++ symbol names, and simplify STL library demangled names.
- --counter <number>
Gather information from a specific counter. The default counter is 0 if not specified.
- -o
Display the line number in the source code for each sample. When the executable was compiled, GCC's -g option should have been used. Otherwise, this option can not display the line numbers. None of the Red Hat Enterprise Linux executables are compiled with this option by default.
vma samples % symbol name linear info 0806cbb0 0 0 _start ../sysdeps/i386/elf/start.S:47
- -e <symbol-name>
Exclude the comma-separated list of symbols from the output.
- -k
Display an additional column containing the shared library. This option only produces results if the --separate=library option to opcontrol is specified when configuring OProfile and if the --dump-gprof-file option is not used in conjunction with this option.
- -t <format>
Display the output in a specific column order. This option can not be used with -g.
Use the following letters to represent the columns:
Letter Description v Virtual memory address s Number of samples S Cumulative number of samples p Percentage of samples relative to total number of samples for the executable P Cumulative percentage of samples relative to total number of samples for the executable q Percentage of samples relative to all executables sampled Q Cumulative percentage of samples relative to all executables sampled n Symbol name l File name of source file and line number, including full path L Base name of the source code file name and line number i Name of the executable, including full path I Base name of the executable d Details of the sample h Display column headers Table 43-4. Letters for Column Order
- --session <name>
Specify the full path to the session or a directory relative to the /var/lib/oprofile/samples/ directory.
- -p <path-list>
Specify a comma-separated list of paths in which the executables to be analyzed are located.
43.5.3. Using op_to_source
The op_to_source tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting files generated should have the samples for the lines at the left. It also puts in a comment at the beginning of each function listing the total samples for the function.
For this utility to work, the executable must be compiled with GCC's -g option. By default, Red Hat Enterprise Linux packages are not compiled with this option.
The general syntax for op_to_source is as follows:
op_to_source --source-dir <src-dir> <executable> |
The directory containing the source code and the executable to be analyzed must be specified. Refer to the op_to_source man page for a list of additional command line options.
43.5.4. Using op_merge
If multiple sample files exist for the exact same executable or library, the sample files can be merged for easier analysis.
For example, to merge files for the library /usr/lib/library-1.2.3.so, execute the following command as root:
op_merge /usr/lib/library-1.2.3.so |
The resulting file is /var/lib/oprofile/samples/}usr}lib}library-1.2.3.so.
To limit the samples merged to a specific counter, use the -c option followed by the counter number.