Testing for Regressions: Recommended
Approach
The recommended approach for regression testing with
the
Intel Inspector
involves these basic steps:
- Create a gold standard:
- Use the GUI (or thecommand) to create a result from which you have successfully eliminated all real problems.inspxe-cl
- Use thecommand to create a suppression file to match any remaining problems you have determined you can ignore.inspxe-cl
- Use thecommand (or the GUI) to check for new problems:inspxe-cl
- Apply the suppression file when you run a new analysis.
- Determine if the changed source code introduced new problems that do not match those in the suppression file.
Create a Result
In the GUI (or the
command tool if you prefer), create a result from which you have successfully
eliminated all real problems. Any problem remaining in the result is a positive
you have determined you can ignore. For example:
inspxe-cl
- You may haveData raceproblem caused by more than one thread accessing a variable without synchronization; however, you know there are other timing factors that make it impossible for theData raceto occur.
- You may have an issue in a third-party library that is of no interest to you.
Create one result for each data set you plan to test, or one result
for your entire application. You can use a result created with the same
analysis type you plan to use for the new analysis or with the widest analysis
type available.
Create a Suppression File
In the
command tool, use the
inspxe-cl
create-suppression-file
action to generate a
suppress-all file for all remaining problems.
For example: The following
command creates a
inspxe-cl
mySup
suppression file with rules that suppress
every error in the
r002ti2
result.
$ inspxe-cl -create-suppression-file mySup -result-dir r002ti2
Create a suppress-all file for each data set you plan to test and
store them all in one directory. If you prefer to use the GUI to perform the
remainder of this recommended approach, use the
Suppressions
tab in the
Project Properties
dialog box to identify that
directory.
Applying many suppression files or large suppressions files during
analysis may slow down finalization, so make sure you remove obsolete
suppression files from the directory and generate new suppression files when
previously known issues are eliminated.
Change Your Source Code
Use your development environment to edit your
source code.
Suppression files remain effective for changed source code with the
following caveats:
- Intel Inspectoridentifies source code lines relative to the start of a function, not as absolute values. So you can insert and delete lines of code anywhere in source files without rendering a suppression file ineffective except inside a function where a problem is marked for suppression.
- Adding or deleting lines in a function prior to the location of a suppressed problem causes a problem to no longer match the suppression rule. Therefore the problem appears as a new problem in subsequent analysis runs.
Intel Inspector
-created
suppression files contain (usually multiple) narrow rules that include line
numbers (to not accidentally suppress extra problems). Consider handwriting
your own suppression files to create fewer rules with wider reach - or at least
rules that do not include line numbers.
Run a New Analysis
In the
command tool (or the GUI if you prefer), run a new analysis with the
suppress-all file.
inspxe-cl
You can incrementally apply multiple suppress-all files if your gold
standard result changes.
For example: The following
command uses the
inspxe-cl
suppression-file
action-option to apply all suppression
files in the
C:\My Inspector Results\suppressions
directory when
collecting a new result for the
myApp
application.
$ inspxe-cl -collect ti2 -suppression-file "C:\My Inspector Results\suppressions" -- myApp
Determine if the Changed Source Code Introduced New
Problems
When analysis is complete, investigate the result to check for new
problems:
- In thecommand tool: Check the automatically generatedinspxe-clinspxe-cl.txtSummary report.
- In the GUI: Check theProblemspane.
Automate Your Regression Testing Process
Intel Inspector
must execute your code path to find errors in it, so you should consider
running the
Intel Inspector
on multiple versions of your code, on different workloads that stress different
code paths, as well as on corner cases. However, multiple
Intel Inspector
analysis runs can take time.
It may be more efficient to automate: Batch process multiple analysis
runs with different data sets passed in as arguments - possibly overnight - and
have the computer do the work for you.
The simplest automation scenario involves adding the following steps
to a script:
- Set up thecommand environment.inspxe-cl
- Invoke thecommand to run an analysis and apply the suppress-all file(s).inspxe-cl
- Determine if the changed source code introduced new problems by checking the returned exit code in the automatically generatedinspxe-cl.txtSummary report when analysis is complete. If the reported exit code equals 0, there are no new problems detected in unsuppressed code locations. If the reported exit code does not equal 0, consider sending the Summary report to a log file and email the log file to an engineer for further triage.
- Repeat steps 2 and 3 as necessary.