The Intel® Compilers code-coverage tool leverages the Intel Compilers' profile-guided optimization technology to present developers a complete picture of the coverage of their application code on a particular workload. To find the application's code coverage the tool combines static profile information generated by the compiler with dynamic profile information generated by running the user's instrumented binaries on the workload. The coverage tool uses this information together with the application sources to create HTML pages with color annotations that highlight the coverage of the code. Navigation is through frames that make it particularly easy to sort the application's files and functions and see the least-covered modules and functions. Developers can then use their favorite browser to display the coverage of their code.

The Intel Compilers code coverage tool can be used in a number of ways to improve development efficiency, reduce defects, and improve application performance. When applied to the profile of the application on its test space, the tool can be used to measure the overall quality of testing based on the coverage information. On the other hand, when applied to the profile of a performance workload, the code-coverage information indicates how well the workload exercises the application's critical code. High coverage of performance-critical modules is essential to taking full advantage of the profile-guided optimizations that Intel Compilers offer. The tool also provides an option, useful for both coverage and performance tuning, through which the users can display the dynamic execution count of each basic block of the application. Lastly, the coverage tool provides the ability to compare the profile of two different runs of the application. This feature can be used to find the portion of the application's code that is not covered by the application's tests but is exercised when the application is used outside the test space, such as a use by a customer.

The Intel Compilers code-coverage tool is supported on Intel Architecture 32-bit and the Itanium® Processor Family on both Windows and Linux and seamlessly supports C, C++, and Fortran.

The code-coverage tool creates two levels of coverage information:

• Top-level summary information
• Individual module source-view

Top-Level Coverage
The top-level coverage summary presents the overall code coverage of the modules. Through the -comp option, users can select the modules they would like to include in code-coverage analysis. For the selected modules, the tool generates the list of modules that were covered as well as the list of modules that were not covered in addition to the total coverage summary. The information includes the total number of functions and blocks in each module and the portions that were covered.

By clicking on the title of columns in the reported tables, the lists may be sorted in ascending or descending order based on the following keys.

• Basic-block coverage
• Function coverage
• Function name

Click to Enlarge

Figure 1 shows a sample top-level coverage summary for a project. By clicking on a module name (e.g., SAMPLE.C), the browser will display the coverage source view of that particular module. Figure 2 shows the coverage source view of SAMPLE.C.

Click to Enlarge

Coloring Scheme for the Code Coverage
The tool distinguishes between the covered code, uncovered basic blocks, uncovered functions, and partially covered code. There are default colors that the tool uses for highlighting the coverage information. These colors, however, can be customized to be any valid HTML color when the coverage tool is run to create a coverage report. The table below shows the default settings that the coverage tool uses.

Color	Meaning
Covered code	The portion of code colored in this color was exercised by the tests. The default color can be overridden through the -ccolor switch.
Uncovered basic block	Basic blocks that are colored in this color were not exercised by any of the tests. They were, however, within functions that were executed during the tests. The default color can be overridden through the -bcolor switch.
Uncovered function	Functions that are colored in this color were never called during the tests. The default color can be overridden through the -fcolor switch.
Partially-covered code	More than one basic block was generated for the code at this position. Some of the blocks were covered while some were not. The default color can be overridden through the -pcolor switch.
Unknown	No code was generated for this source position. Most probably, the source at this position is probably a comment, a header-file inclusion, or a variable declaration. The default color can be overridden through the -ucolor switch.

For source annotation with colors, the coverage tool does not parse the application sources in the same way that the compiler does. It uses the following heuristic for coloring. Source characters are scanned until reaching a position in the source that is indicated by the profile information as the beginning of a basic block. If the profile information for that basic block indicates a need for changing the color from its current color, then the coverage tool inserts the appropriate color change in the HTML files. Note that this implies that sometimes the user would need to interpret the colors in the context of the code. For instance, comment lines that follow a basic block that was never executed would be colored in the same color as the uncovered blocks. Same behavior may also be experienced with closing brackets in C/C++ applications.

Browsing helper frames
The coverage tool creates frames that facilitate browsing through the code to identify uncovered code. The top frame displays the list of uncovered functions while the bottom frame displays the list of covered functions. For uncovered functions, the total number of basic blocks of each function is also displayed. For covered functions, both the total number of blocks and the number of covered blocks as well as their ratio (i.e., the coverage rate) are displayed. For example, 66.67(4/6)indicates that four out of the six blocks of the corresponding function were covered. The block coverage rate of that function is thus 66.67%. These lists can be sorted based on the coverage rate, number of blocks, or function names. Function names are linked to the position in source view where the function body starts. So, just by one click, the user can see the least-covered function in the list and by another click the browser displays the body of the function. The user can then scroll down in the source view and browse through the function body.

Dynamic Counters
In addition to the coloring scheme showing the coverage information, the coverage tool can be configured to generate the information about the dynamic execution counts. This is achieved through the -counts option. The counts information is displayed under the code after a ^ sign precisely under the source position where the corresponding basic block begins. If more than one basic block is generated for the code at a source position (for example, for macros), then the total number of such blocks and the number of the blocks that were executed are also displayed in front of the execution count.

Click to Enlarge

In Figure 3, the information under line 11 and 12 contain the following information:

• The if statement on line 11 was executed 10 times.
• Two basic blocks were generated for the if statement on line 11.
• Only one of the two blocks was executed, hence the partial-coverage color.
• Only seven out of the ten times variable n had a value of 0 or 1.

In certain situations, it may be desirable to consider all the blocks generated for a single source position as one entity. In other words, we may want to consider all blocks generated for one source position as covered when at least one of the blocks is covered. This can be achieved through the -nopartial option. When this option is specified, decision coverage is disabled and the related statistics are adjusted appropriately. Line 11 and 12 of Figure 2 indicate that the print statement on line 12 was covered, however, only one of the conditions on line 11 was ever true. The user may direct the coverage tool, through the -nopartial option, to treat the partially covered code (like the code on line 11) as covered.

Differential Coverage
Intel Compilers code-coverage tool provides the capability of comparing the profile of an application on a reference run with that of a new run and identifying the code that is covered by the new run but not covered by the reference run. This feature can be used to find the portion of the application's code that is not covered by the application's tests but is exercised when the application is used outside the test space, such as a use by a customer. It can also be used to find the incremental coverage impact of newly added tests to an application's test space. The dynamic profile information of the reference run for differential coverage is specified through the -ref switch such as in the following command.

codecov -prj Project_Name -dpi customer.dpi -ref appTests.dpi

The coverage statistics of a differential-coverage run show the percentage of the code that was exercised on a new run but was missed in the reference run. In such cases, the coverage tool shows only the modules that included some code that was missed. The coloring scheme in the source views also should be interpreted accordingly. The code that has the same coverage property (covered or not covered) on both runs is considered as covered code. Otherwise, if the new run indicates that the code was executed while in the reference run the code was not executed, then the code is treated as uncovered. On the other hand, if the code is covered in the reference run but not covered in the new run, the differential-coverage source view shows the code as covered.

Requirements for running the Intel Compilers code-coverage tool
In order to run the Intel Compilers code-coverage tool on an application, the user should provide the following three items:

• The application sources
• The .SPI file generated by Intel Compilers when compiling the application for the instrumented binaries through the -Qprof_genx (on Windows) or -prof_genx (on Linux) switches.
• The .DPI file generated by Intel Compilers profmerge tool as the result of merging the dynamic profile information files *.DYN or the .DPI file generated implicitly by Intel Compilers when compiling the application with -Qprof_use (on Windows) or -prof_use (on Linux) options.

The usage model of Intel Compilers profile-guided optimizations is presented in Figure 4. Here are the steps for a simple example (myApp.c) on Windows 32-bit.

1. set PROF_DIR=c:\myApp\prof_dir
2. icl /Qprof_genx myApp.c
   This command compiles the program and generates instrumented binary myApp.exe as well as the corresponding static profile information pgopti.spi.
3. myApp.exe
   Each invocation of this command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by PROF_DIR.
4. icl /Qprof_use myApp.c
   At this step, the compiler merges all the .dyn files into one .dpi file representing the total profile information of the application and generates the optimized binary. The name of the default .dpi file is pgopti.dpi.

Note that the .dyn files can also be merged to a .dpi file by the Intel Compilers profmerge tool without recompiling the application. profmerge can also merge multiple .dpi files into one .dpi file using its -a option. The user can select the name of the output .dpi using profmerge -o option.

When these items are available, the coverage tool may be launched from the command line like:

codecov -prj Project_Name -spi pgopti.spi -dpi pgopti.dpi

Through the -spi and -dpi options, the user can specify the path to these files. The coverage tool also has the following additional options for generating a link at the bottom of each HTML page to send an electronic message to a named contact.

codecov -prj Project_Name -mname John_Smith -maddr js@company.com

Important note about collecting profile information
The profmerge tool merges all the .dyn files that exist in the given directory. It is very important that the user makes sure that unrelated .dyn files, oftentimes from previous runs, are not present in that directory. Otherwise, profile information will be skewed with invalid profile data. This can result in negatively impacting the performance of optimized code as well providing misleading coverage information.

Click to Enlarge

Coverage analysis of a subset of application modules
Intel Compilers code-coverage tool provides the capability of very efficient coverage analysis on a subset of the modules of the application. If only a subset of the application modules are compiled with the -Qprof_genx option, then the coverage information is generated only for those modules, thus avoiding the overhead incurred for profile generation of the modules that are not necessary. Alternatively, the user can generate the profile information for the whole application, or a subset of it, and then break the covered modules into different components and use the coverage tool to obtain the coverage information of each individual component.

User can specify the modules of interest through the -comp option. This option takes the name of a file as its argument. That file must be a text file that includes the name of modules or directories the user would like to be analyzed. Here is an example:

codecov -prj Project_Name -comp component1

Each line of component file should include one, and only one, module name. Any module of the application whose full path name has an occurrence of any of the names in the component file will be selected for coverage analysis. For example, if a line of file component1 contains mod1.c, then all modules in the application that have such a name will be selected. The user can specify a particular module by giving more specific path information. For instance, if the line contains /cmp1/mod1.c, then only those modules with the name mod1.c will be selected that are in a directory named cmp1. If no component file is specified, then all files that have been compiled with -Qprof_genx are selected for coverage analysis.

The following table shows all the options of the code-coverage tool.

Option	Description	Default
-help	Prints all the options of the code-coverage tool.
-spi <file>	Sets the path name of the static profile information file .spi.	pgopti.spi
-dpi <file>	Sets the path name of the dynamic profile information file .dpi.	pgopti.dpi
-prj	Sets the project name.
-counts	Generates dynamic execution counts.
-nopartial	Treats partially-covered code the same as fully-covered code.
-comp	Sets the filename that contains the list of files of interest.
-ref	Finds the differential coverage with respect to ref_dpi_file.
-demang	Demangles both C++ function names and their arguments.
-mname	Sets the email address of the web-page owner.
-maddr	Sets the email address of the web-page owner.
-bcolor	Sets the html color name or code of the uncovered blocks.	#ffff99
-fcolor	Sets the html color name or code of the uncovered functions.	#ffcccc
-pcolor	Sets the html color name or code of the partially-covered code.	#fafad2
-ccolor	Sets the html color name or code of the covered code.	#ffffff
-ucolor	Sets the html color name or code of the unknown code.	#ffffff
Table 1. Intel Compilers code-coverage tool options.

Intel Compilers test-prioritization tool

The Intel Compilers test-prioritization tool leverages the Intel Compilers profile-guided optimizations technology to select and prioritize application's tests based on prior execution profiles of the application. Using this tool, users can select and prioritize the tests that are more relevant for any subset of the application's code. When certain modules of an application are changed, the Intel Compilers test-prioritization tool suggests the tests that are most probably affected by the given change set. The tool mines the profile data from previous runs of the application, discovers the dependency between the application's components and its tests, and uses this information to guide the process of testing. The tool can be used for devising an effective hierarchical testing based on the application's code coverage. For instance, the tool may be used to find the smallest subset of the application tests that achieve exactly the same code coverage as the entire set of tests. The tool can also be used to dramatically reduce the turn-around time of testing. Instead of spending a large amount of time and finding a possibly-large number of failures, the tool may enable the users to quickly find a small number of tests that expose the defects associated with the regressions caused by a change set. The tool offers the potential of significant time saving in testing and development of large-scale applications where testing is major bottleneck. The tool can be used to minimize the number of tests that are required to achieve a given overall coverage for any subset of the application. Moreover, when the execution times of the tests are available, the tool may also be used to select and prioritize the tests to achieve certain level of code coverage in a minimum amount of time.

The Intel Compilers test-prioritization tool is supported on Intel Architecture 32-bit and the Itanium Processor Family on both Windows and Linux and seamlessly supports C, C++, and Fortran.

Requirements for running the Intel Compilers test-prioritization tool
In order to run the Intel Compilers test-prioritization tool on an application's tests, the user should provide the following items:

• The .SPI file generated by Intel Compilers when compiling the application for the instrumented binaries through the -Qprof_genx (on Windows) or -prof_genx (on Linux) switches.
• The .DPI files generated by Intel Compilers profmerge tool as the result of merging the dynamic profile information files *.DYN of each of the application tests. The user needs to apply the profmerge tool to all *.DYN files that are generated for each individual test and name the resulting .DPI in a fashion that uniquely identifies the test. The profmerge tool merges all the .dyn files that exist in the given directory. It is very important that the user makes sure that unrelated .dyn files, oftentimes from previous runs or from other tests, are not present in that directory. Otherwise, profile information will be skewed with invalid profile data. This would result in misleading coverage information.
• The user should provide the list of tests to be prioritized. The user needs to name the .DPI file that represents a test in a fashion that it uniquely identifies that test. The user should provide the name of all such .DPI files in a DPI-list file, a text file that will be given to the test prioritization tool. Each line of the DPI-list file should include one, and only one, DPI name. The name can optionally be followed by the execution time of the corresponding test in the dd:hh:mm:ss format. The execution time is optional. However, if it is not provided, then the tool would reject to prioritize the tests to minimize the execution time. It would still be able to prioritize tests so that the number of tests is minimized.

The usage model of Intel Compilers test-prioritization tool is presented in Figure 5. Here are the steps for a simple example (myApp.c) on Windows 32-bit.

1. set PROF_DIR=c:\myApp\prof_dir
2. icl /Qprof_genx myApp.c
   This command compiles the program and generates instrumented binary myApp.exe as well as the corresponding static profile information pgopti.spi.
3. rm PROF_DIR \*.dyn
   Make sure that there are no superfluous .dyn files present.
4. myApp.exe < data1
   Invocation of this command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by PROF_DIR.
5. profmerge -prof_dpi Test1.dpi
   At this step, the profmerge tool merges all the .dyn files into one file (Test1.dpi) representing the total profile information of the application on Test1.
6. rm PROF_DIR \*.dyn
   Make sure that there are no superfluous .dyn files present.
7. myApp.exe < data2
   Invocation of this command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by PROF_DIR.
8. profmerge -prof_dpi Test2.dpi
   At this step, the profmerge tool merges all the .dyn files into one file (Test2.dpi) representing the total profile information of the application on Test2.
9. rm PROF_DIR \*.dyn
   Make sure that there are no superfluous .dyn files present.
10. myApp.exe < data3
    Invocation of this command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by PROF_DIR.
11. profmerge -prof_dpi Test3.dpi
    At this step, the profmerge tool merges all the .dyn files into one file (Test3.dpi) representing the total profile information of the application on Test2.

Create a file named tests_list with three lines. The first line includes Test1.dpi, the second line includes Test2.dpi, and the third line includes Test3.dpi.

When these items are available, the test-prioritization tool may be launched from the command line in PROF_DIR directory like:

tselect -dpi_list tests_list -spi pgopti.spi

Through the -spi option, the user specifies the path to the .SPI file. Here is a sample ouuput from the test-prioritization tool.

Num	%RatCvrg	%BlkCvrg	%FncCvrg	Test Name @ Options
1	87.50	45.65	37.50	Test3.dpi
2	100.00	52.17	50.00	Test2.dpi

In this case, the test-prioritization tool has provided the following information:

• By running all three tests, we achieve 52.17% block coverage and 50.00% function coverage.
• Test3 by itself covers 45.65% of the basic blocks of the application, which is 87.50% of the total block coverage that can be achieved from all three tests.
• By adding Test2, we achieve a cumulative block coverage of 52.17% or 100% of the total block coverage of Test1, Test2, and Test3.
• Elimination of Test1 has no negative impact on the total block coverage.

Suppose that we have the execution of each test as shown following in tests_list file.

Test1.dpi	00:00:60:35
Test2.dpi	00:00:10:15
Test3.dpi	00:00:30:45

We can run the test-prioritization tool to minimize the execution time as follows.

tselect -dpi_list tests_list -spi pgopti.spi -mintime

Here is a sample output.

Num	elapsedTime	%RatCvrg	%BlkCvrg	%FncCvrg	Test Name @ Options
1	10:15	75.00	39.13	25.00	Test2.dpi
2	41:00	100.00	52.17	50.00	Test3.dpi

The results indicate that running all tests sequentially would require one hour, 45 minutes, and 35 seconds, while the selected tests would achieve the same total block coverage in only 41 minutes. Note that the order of tests when prioritization is based on minimizing time (first Test2, then Test3) could be different than when prioritization is done based on minimizing just number of tests (first Test3, then Test2). In the above example, Test2 is the test that gives the highest coverage per execution time. So, it is picked as the first test to run.

The prioritization tool may also be given an option to exit when it reaches certain level of basic block coverage. This is done through the -cutoff option such as following.

tselect -dpi_list tests_list -spi pgopti.spi -cutoff 85.00

If the tool is run with this cutoff value on the above example, only Test3 will be selected, as it achieves 45.65% block coverage, which is 87.50% of the total block coverage that can be achieved from all three tests.

The test-prioritization tool does an initial merging of all the profile information to figure out the total coverage that is achieved by running all the tests. The user can request to skip this step by the -nototal option. In such a case, only the absolute coverage information will be reported as the overall coverage is unknown upfront.

The coverage tool has some additional options that are listed in Table 2.

Click to Enlarge

Option	Description	Default
-help	Prints all the options of the test-prioritization tool.
-spi <file>	Sets the path name of the static profile information file .spi.	pgopti.spi
-dpi_list <file>	Sets the path name of the file that contains the name of the dynamic profile information (.dpi) files. Each line of the file should contain one .dpi name optionally followed by its execution time. The name must uniquely identify the test.
-o <file>	Sets the path name of the output report file.
-comp	Sets the filename that contains the list of files of interest.
-cutoff <value>	Terminates when the cumulative block coverage reaches <value>% of pre-computed total coverage. <value> must be greater than 0.0 (e.g., 99.00). It may be set to 100.
-nototal	Does not pre-compute the total coverage.
-mintime	Minimizes testing execution time. The execution time of each test must be provided on the same line of dpi_list file after the test name in dd:hh:mm:ss format.
-verbose	Generates more logging information about the program progress.
Table 2. Intel Compilers test-prioritization tool options.