User Guide

  • 2022.2
  • 08/08/2022
  • Public Content

Tracing Failing MPI Applications

Normally, if an MPI application fails or is aborted, all the trace data collected is lost, because
needs a working MPI to write the trace file. However, the user might want to use the data collected up to that point. To solve this problem, Intel® Trace Collector provides the
library that enables tracing of failing MPI applications.

Usage Instructions

To trace failing MPI applications, do the following:
Linux* OS
Set the
environment variable to point to the
library and run the application. For example:
$ export $ mpirun -n 4 ./myApp
Alternatively, rebuild your application with the static version of the library. For example:
$ mpiicc -profile=vtfs myApp.c -o myApp
Windows* OS
Relink your application with the
library before the Intel MPI Library and run it as usual. To do this, you should create an Intel® MPI Library configuration file that points to the
library. You can do it as follows (administrator privileges may be required):
> echo SET PROFILE_PRELIB=%VT_ROOT%\lib\VTfs.lib > %I_MPI_ROOT%\lib\VTfs.conf > mpiicc -profile=VTfs myApp.c > mpiexec -n 4 myApp.exe

How it Works

Under normal circumstances tracing works like with
, but communication during trace file writing is done through TCP sockets, so it may take more time than over MPI. In order to establish communication, it needs to know the IP addresses of all the hosts involved. It finds them by looking up the hostname locally on each machine or, if that only yields the local host IP address, falls back to broadcasting hostnames. In the latter case hostname lookup must work consistently in the cluster. In case of a failure,
freezes all MPI processes and then writes a trace file with all trace data.

Possible Failures

Includes events inside the application like segmentation faults and floating point errors, and also abort signals sent from outside, like
will abort the application without writing a trace because it cannot be caught.
Premature Exit
One or more processes exit without calling
MPI Errors
MPI detects certain errors itself, like communication problems or invalid parameters for MPI functions.
If Intel® Trace Collector observes no progress for a certain amount of time in any process, it assumes that a deadlock has occurred, stops the application and writes a trace file.
You can configure the timeout with
. "No progress" is defined as "inside the same MPI call". This is only a heuristic and may fail to lead to both false positives and false negatives.
Undetected Deadlock
If the application polls for a message that cannot arrive with
or a similar non-blocking function, Intel® Trace Collector still assumes that progress is made and does not stop the application.
To avoid this, use blocking MPI calls in the application, which is also better for performance.
Premature Abort
If all processes remain in MPI for a long time due to a long data transfer for instance, then the timeout might be reached. Because the default timeout is five minutes, this is very unlikely. After writing the trace
will try to clean up the MPI application run by sending all processes in the same process group an
signal. This is necessary because certain versions of MPICH* may have spawned child processes which keep running when an application aborts prematurely, but there is a certain risk that the invoking shell also receives this signal and also terminates. If that happens, then it helps to invoke
inside a remote shell:
rsh localhost 'sh -c "mpirun . . . "'
MPI errors cannot be ignored by installing an error handler.
overrides all requests to install one and uses its own handler instead. This handler stops the application and writes a trace without trying to proceed, otherwise it would be impossible to guarantee that any trace will be written at all.
On Windows* OS, not all features of POSIX* signal handling are available. Therefore,
on Windows* OS uses some heuristics and may not work as reliably as on Linux* OS. It is not possible to stop a Windows* application run and get a trace file by sending a signal or terminating the job in the Windows task manager.

Product and Performance Information


Performance varies by use, configuration and other factors. Learn more at