Intel® Trace Analyzer and Collector User and Reference Guide

Download
ID 767272
Date 8/08/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
Intel® Trace Analyzer and Collector User and Reference Guide
Intel® Trace Analyzer and Collector User and Reference Guide x
Introduction Install and Set Up Intel® Trace Analyzer and Collector Trace Your Applications Analyze Your Applications Intel® Trace Collector Reference Intel® Trace Analyzer Reference Notices and Disclaimers
Introduction x
What's New Notational Conventions Get Help
Trace Your Applications x
Tracing Conventional MPI Applications Tracing Failing MPI Applications Tracing OpenSHMEM* Applications Tracing MPI File IO Handling of Communicator Names Tracing MPI Load Imbalance Tracing User Defined Events Configuring the Collector Filtering Trace Data Filtering by Collective and P2P Operations Basic Function Filtering Advanced Function Filtering Filtering by Ranks Recording OpenMP* Regions Information Tracing System Calls (Linux* OS) Collecting Lightweight Statistics Recording Source Location Information Recording Hardware Performance Information (Linux* OS) Recording Operating System Counters Tracing Library Calls Correctness Checking Tracing Distributed Non-MPI Applications
Correctness Checking x
Correctness Checking of MPI Applications Running with Valgrind* (Linux* OS) Configuring Error Checks Analyzing the Results Debugger Integration
Debugger Integration x
TotalView* Debugger GNU* Symbolic Debugger Allinea* Distributed Debugging Tool* (DDT*)
Analyze Your Applications x
Starting Intel® Trace Analyzer Intel Trace Analyzer Graphical User Interface Navigating Timelines Concepts Viewing Correctness Checking Reports Comparing Two Trace Files Interoperability with Intel® VTune™ Profiler and Intel® Advisor OpenMP* Regions Display Support OTF2 Format Support
Navigating Timelines x
Zoom Stack
Concepts x
Level of Detail Aggregation Advanced Aggregation Tagging and Filtering
Viewing Correctness Checking Reports x
Event Timeline Correctness Checking Reports Qualitative Timeline Correctness Checking Reports Detailed Dialog
Comparing Two Trace Files x
Mappings in Comparison Views Comparison Charts
Mappings in Comparison Views x
Mapping of Processes Mapping of Functions
Comparison Charts x
Comparison Function Profile Comparison Message Profile Comparison Collective Operations Profile
Intel® Trace Collector Reference x
API Reference Configuration Reference Correctness Checking Errors Structured Tracefile Format stftool Utility Time Stamping Secure Loading of Dynamic Link Libraries* on Windows* OS
API Reference x
Initialization, Termination and Control Defining and Recording Source Locations Defining and Recording Functions or Regions Defining and Recording Scopes Defining Groups of Processes Defining and Recording Counters Recording Communication Events Additional API Calls in libVTcs C++ API
Initialization, Termination and Control x
VT_initialize VT_finalize VT_getrank VT_registerthread VT_registernamed VT_registerprefixed VT_getthrank VT_traceon VT_traceoff VT_tracestate VT_symstate VT_flush VT_timestamp VT_timestart VT_setfinalizecallback VT_getdescription VT_countsetcallback
Defining and Recording Functions or Regions x
New Interface Old Interface State Changes
C++ API x
VT_FuncDef Class Reference VT_SclDef Class Reference VT_Function Class Reference VT_Region Class Reference
Configuration Reference x
Configuration File Format Protocol File Configuration Options
Configuration Options x
ACTIVITY ALTSTACK AUTOFLUSH CHECK CHECK-LEAK-REPORT-SIZE CHECK-MAX-DATATYPES CHECK-MAX-ERRORS CHECK-MAX-PENDING CHECK-MAX-REPORTS CHECK-MAX-REQUESTS CHECK-SUPPRESSION-LIMIT CHECK-TIMEOUT CHECK-TRACING CLUSTER COMPRESS-RAW-DATA COUNTER CURRENT-DIR DEADLOCK-TIMEOUT DEADLOCK-WARNING DEMANGLE DETAILED-STATES ENTER-USERCODE ENVIRONMENT EXTENDED-VTF FLUSH-PID FLUSH-PREFIX GROUP HANDLE-SIGNALS INTERNAL-MPI KEEP-RAW-EVENTS LOGFILE-FORMAT LOGFILE-NAME LOGFILE-PREFIX LOGFILE-RANK MEM-BLOCKSIZE MEM-FLUSHBLOCKS MEM-INFO MEM-MAXBLOCKS MEM-MINBLOCKS MEM-OVERWRITE NMCMD OS-COUNTER-DELAY PCTRACE PCTRACE-CACHE PCTRACE-FAST PLUGIN PROCESS PROGNAME PROTOFILE-NAME STATISTICS STATE STF-PROCS-PER-FILE STF-USE-HW-STRUCTURE STOPFILE-NAME SYMBOL SYNC-MAX-DURATION SYNC-MAX-MESSAGES SYNC-PERIOD SYNCED-CLUSTER SYNCED-HOST TIME-WINDOWS (Experimental) TIMER TIMER-SKIP UNIFY-COUNTERS UNIFY-GROUPS UNIFY-SCLS UNIFY-SYMBOLS VERBOSE VT_START_PAUSED VT_COMPRESS_TRACE
Correctness Checking Errors x
Supported Errors How the Collection Works
How the Collection Works x
Parameter Checking Premature Exit Overlapping Memory Detecting Illegal Buffer Modifications Buffer Given to MPI Cannot Be Read or Written Distributed Memory Checking Illegal Memory Access Request Handling Datatype Handling Buffered Sends Deadlocks Checking Message Transmission Datatype Mismatches Data Modified during Transmission Checking Collective Operations Freeing Communicators
Structured Tracefile Format x
STF Components Single-File STF Configuring STF
stftool Utility x
stftool Utility Options Expanded ASCII output of STF Files
Time Stamping x
Clock Synchronization Choosing a Timer
Choosing a Timer x
gettimeofday/_ftime QueryPerformanceCounter CPU Cycle Counter Normalized CPU Cycle Counter MPI_Wtime() High Precision Event Timers POSIX* clock_gettime
Intel® Trace Analyzer Reference x
Graphical User Interface Reference Intel® Trace Analyzer Command Line Interface Reference Filter Expression Grammar otf2-to-stf Utility
Graphical User Interface Reference x
Welcome Page Summary Page Main Menu Bar View Menu Bar View Bars Charts Dialogs Settings
Main Menu Bar x
File Menu Options Menu Project Menu  Windows Menu Help Menu
View Menu Bar x
View Charts Navigate Advanced Layout Comparison Menu
Advanced x
Tagging Specific Events Filtering Events Simulating Ideal Communication Checking Application Imbalance Aggregating Results Aggregating Functions Creating Command Line for Intel® VTune™ Profiler and Intel® Advisor
View Bars x
Toolbar Trace Map Status Bar
Charts x
Event Timeline Qualitative Timeline Quantitative Timeline Counter Timeline Function Profile Message Profile Collective Operations Profile Performance Assistant Common Chart Features
Event Timeline x
Context Menu Filtering and Tagging
Qualitative Timeline x
Context Menu Filtering and Tagging
Quantitative Timeline x
Context Menu Filtering and Tagging
Counter Timeline x
Context Menu Filtering and Tagging
Function Profile x
Flat Profile Load Balance Call Tree Call Graph Context Menu Filtering and Tagging Function Profile Settings
Message Profile x
Context Menu Filtering and Tagging Aggregation Message Profile Settings
Collective Operations Profile x
Context Menu Filtering and Tagging Collective Operations Profile Settings
Dialogs x
Process Aggregation Function Aggregation Function Group Color Editor Filtering Dialog Box Tagging Dialog Box Idealization Dialog Box Imbalance Diagram Dialog Box Trace Merge Dialog Box  Details Dialog Box Source View Dialog Time Interval Selection Configuration Dialogs Find Dialog Box Command line for Intel® VTune™ Profiler and Intel® Advisor Dialog Box OTF2 to STF Conversion Dialog Box Configuration Assistant
Process Aggregation x
Comparison Mode
Function Aggregation x
Comparison Mode
Filtering Dialog Box x
Building Filter Expressions Using Graphical Interface Building Filter Expressions Manually Filter Expressions in Comparison Mode
Details Dialog Box x
Detailed Attributes of Function Events Detailed Attributes of Message Events Detailed Attributes of Collective Operation Events
Configuration Dialogs x
Load Configuration File Dialog Edit Configuration File Dialog
Settings x
Preferences Font Settings Number Formatting Settings
Preferences x
General Preferences Tracefile Preferences Event Timeline Settings Qualitative Timeline Settings Quantitative Timeline Settings Counter Timeline Settings
Notices and Disclaimers x
Appendix A Copyright and Licenses
Intel® Trace Analyzer and Collector User and Reference Guide

Filtering Trace Data

Filtering in Intel® Trace Collector applies specified filters to the trace collection process. This directly reduces the amount of data collected. The filter rules can be defined in a configuration file, in the environment variables or as command line arguments (see Configuring Intel® Trace Collector for details). Filters are evaluated in the order they are listed, and all matching is case-insensitive. For filtering by collective and point-to-point MPI operations the corresponding mpirun/mpiexec options are also available.

In the following discussion, items within angle brackets (< and >) are placeholders for actual values, optional items are put within square brackets ([ and ]), and alternatives are separated by a vertical bar |.

Filtering by Collective and P2P Operations

Use the following mpirun/mpiexec options at runtime:

  • -trace-collectives – to collect information only about collective operations
  • -trace-pt2pt – to collect information only about point-to-point operations
NOTE:
These options are mutually exclusive, use only one option at a time.

An example command line for tracing collective operations may look as follows: 

$ mpirun -trace -n 4 -trace-collectives ./myApp

These options are special cases of filtering by specific functions discussed below. They use predefined configuration files to collect the appropriate operation types. The options override the VT_CONFIG environment variable, so you cannot use your own configuration files when using them. If you need to set additional filters, you can set them through individual environment variables.

Basic Function Filtering

Function filtering is accomplished by defining the STATE, SYMBOL and ACTIVITY options. Each option accepts the pattern matching the filtered function, and the filtering rule. The formal definition is as follows: 

STATE|SYMBOL|ACTIVITY <PATTERN> <RULE>

The general option is STATE, while SYMBOL and ACTIVITY are its replacers:

  • SYMBOL filters functions by their names, regardless of the class. The following definitions are considered equal: 

    SYMBOL <PATTERN> <RULE>
STATE **:<PATTERN> <RULE>

  • ACTIVITY filters functions by their class names. The following definitions are considered equal: 

    ACTIVITY <PATTERN> <RULE>
STATE <PATTERN>:* <RULE>

  • STATE can filter functions both by names and their class names.

Pattern should match the name of the function for filtering. You can use the following wildcards:

Wildcard Description
* Any number of characters, excluding ":"
** Any number of characters, including ":"
? A single character
[ ] A list of characters

For example the following definition will filter out all functions containing the word send, regardless of the class: 

STATE **send* OFF

The basic filter rule should contain one of the following entries: 

<RULE> = ON | OFF | <trace level> | <skip level>:<trace level>

The <trace level> value defines how far up the call stack is traced. The <skip level> value defines how many levels to skip while going up the call stack. This is useful if a function is called within a library, and the library stack can be ignored. Specifying ON turns on tracing with a <trace level> of 1 and a <skip level> of 0, and OFF turns off tracing completely (this is not equivalent to 0:0).

Example:

In the example below the following events will be traced: all functions in the class Application, all MPI send functions except MPI_Bsend(), and all receive, test and wait functions. All other MPI functions will not be traced. 

# disable all MPI functions
ACTIVITY MPI OFF
# enable all send functions in MPI
STATE MPI:*send ON
# except MPI_Bsend
SYMBOL MPI_bsend OFF
# enable receive functions
SYMBOL MPI_recv ON
# and all test functions
SYMBOL MPI_test* ON
# and all wait functions, tracing four call levels
SYMBOL MPI_wait* 4
# enable all functions within the Application class
ACTIVITY Application 0

Advanced Function Filtering

For function filtering a finer control is also available. Here is a list of additional filter rule entries, which can be used along with the basic rule in any combination: 

<ENTRYTRIGGER> | <EXITTRIGGER> | <COUNTERSTATE> | <FOLDING> | <CALLER>

Here is the specification for each filter entry available:

Entry/Exit Trigger:

<ENTRYTRIGGER> = entry <TRIGGER>
<EXITTRIGGER> = exit <TRIGGER>

Activate a trigger on entry/exit for the matching pattern. 

<TRIGGER> = [<TRIPLET>] <ACTION> [<ACTION>]

Triggers define a set of actions over a set of processes (triplets, see Filtering by Ranks below for definition). 

<ACTION> = traceon | traceoff | restore | none | begin_scope <SCOPE_NAME> | end_scope <SCOPE_NAME>

The action defines what happens to tracing. Using traceon or traceoff turns tracing on or off, respectively. begin_scope and end_scope start or end the named scope. 

<SCOPE_NAME> = [<class name as string>:]<scope name as string>

A scope is a user-defined region in the program. See Defining and Recording Scopes.

Counter State:

<COUNTERSTATE> = counteron | counteroff

Counter state turns on or off sampling for the matching pattern. By default, all enabled counters are sampled at every state change. There is no method for controlling which counters are sampled.

Folding:

<FOLDING> = fold | unfold

Enabling folding for a function disables tracing of any functions called by that function. By default, all functions are unfolded.

Caller:

<CALLER> = caller <PATTERN>

Specifying the caller enables tracing only for functions called by the functions matching the pattern.

For details on use of the FOLDING and CALLER keywords, see Tracing Library Calls.

Filtering by Ranks

Besides filtering by functions, you can also filter the trace data by ranks in MPI_COMM_WORLD using the PROCESS configuration option. Its value is a comma separated list of Fortran 90-style triplets. The formal definition is as follows: 

PROCESS <TRIPLET>[,<TRIPLET>,...] on | off

Triplet definition is as follows: 

<TRIPLET> = <LOWER-BOUND>[:<UPPER-BOUND>[:<INCREMENT>] ]

The default value for <UPPER-BOUND> is the size of MPI_COMM_WORLD (N) and the default value for <INCREMENT> is 1.

For example, to trace only even ranks and rank 1 use the following triplets: 0:N:2,1:1:1, where N is the total number of processes. All processes are enabled by default, so you have to disable all of them first (PROCESS 0:N OFF) before enabling a certain subset again. For SMP clusters, you can also use the CLUSTER option to filter for particular SMP nodes.

Parent topic: Trace Your Applications
Level Two Title

Did you find the information on this page useful?

Characters remaining:

Feedback Message