Application Performance Snapshot User Guide for Linux* OS

ID 772048
Date 3/31/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Detailed Information for Specific Ranks

Option:-D [--details]

Use this option to see details for the specified chart. This option affects charts in the following ways:

Chart name Description
Message Sizes Summary (-m -D) Shows message sizes breakdown by functions
MPI Time per Rank (-t -D)   Shows details for processes with the minimum, mean, and maximum time values
Collective Operations Time per Rank (-c -D) Shows breakdown by functions and message sizes for all ranks

To specify the ranks to show details for, use the -R option instead of -D. You can also specify multiple ranks using the -R option. For example:

$ aps-report ./aps_result_<postfix>  -t -R 1

For the Time charts, the details include the process execution time and information about each function used by the process. Information about functions is presented in the following ways:

  • Function summary
  • Details for the messages transferred by each function

These two views share the table columns but their meanings are different for each view. See the chart entries descriptions below.

Example

|#########################################################
| RANK 3
|#########################################################
| Life Time = 149.539 (sec)
| MPI Time  = 140.662 (sec) 94.0639%

Function Summary

|--------------------------------------------------------------------------------------------------------------------------
|           Function         Time(sec)         Time(%)       Volume(MB)        Volume(%)            Calls    AvrMsgSize(B)
|--------------------------------------------------------------------------------------------------------------------------
           Alltoallv          23.9206          15.9962            12330          31.2704           109624           117939
            Alltoall          23.8242          15.9317          2054.99          5.21173           109624            19656
             Barrier          19.0241          12.7218                0                0            11070                0
          Allgatherv          17.8779          11.9553          2054.99          5.21173           109624            19656
           Allgather               16          10.6996          2054.99          5.21173           109624            19656
           Allreduce          5.72988          3.83169          2054.99          5.21172           100272            21490
            Scatterv          5.53121          3.69883          2054.99          5.21173           109624            19656
             Scatter          5.40639          3.61536          2054.99          5.21173           109624            19656
             Gatherv          5.21395          3.48668          2054.99          5.21173           109624            19656
              Gather          5.20341          3.47963             2055          5.21175           110665            19472
      Reduce_scatter          3.92054          2.62174          385.307         0.977187            97624             4139
                Send          3.33028          2.22703          6164.98          15.6352           328919            19654
               Bcast          2.95886          1.97865             2055          5.21173           110288            19538
              Reduce          2.72119          1.81972          2054.98          5.21168            97624            22072
|==========================================================================================================================
| TOTAL                       140.662          94.0639          39430.2              100          1623830            25462

Details for Function Messages

|--------------------------------------------------------------------------------------------------------------------------
|           Function         Time(sec)         Time(%)       Volume(MB)        Volume(%)            Calls       MsgSize(B)
|--------------------------------------------------------------------------------------------------------------------------
           Alltoallv          23.9206          15.9962            12330              100           109624              ALL
                              3.06902          2.05232             1440          11.6789               90         16777216
                              3.04682          2.03747             1440          11.6789              180          8388608
                              2.89341          1.93488             1440          11.6789              360          4194304
                              2.83532          1.89604             1440          11.6789             2880           524288
                              2.64094          1.76605             1440          11.6789              720          2097152
                              2.53039          1.69212             1440          11.6789             1440          1048576
                              2.47286          1.65365             1230           9.9757             4920           262144
                              2.04191          1.36546              960          7.78591               30         33554432
| [skipped 17 lines]
|--------------------------------------------------------------------------------------------------------------------------
            Alltoall          23.8242          15.9317          2054.99              100           109624              ALL
                              3.14809           2.1052              240          11.6789              120          2097152
                              2.96477          1.98261              240          11.6789               60          4194304
                               2.9347           1.9625              240          11.6789              240          1048576
                              2.77462          1.85545              240          11.6789             3840            65536
                              2.67651          1.78984              240          11.6789              480           524288
                              2.52501          1.68853              240          11.6789             1920           131072
                              2.50317          1.67392              240          11.6789              960           262144
                              2.20381          1.47373            187.5          9.12411             6000            32768
| [skipped 16 lines]
...

Chart Entries

Function Summary Columns
Details for Messages Columns
Rows
Life Time Total execution time of the rank
MPI Time / Coll Op Time Time spent by the rank in MPI calls and its percentage in the Life Time
Function Function name
Time(sec) Total time of the function execution in the rank (in seconds)
Time(%) Percentage of the function execution time in the application lifetime
Volume(MB) Amount of data transferred by the function within the rank (in megabytes)
Volume(%) Percentage of the data transferred by the function
Calls Number of calls of the function in the rank
AvrMsgSize(B) Average message size transferred by the function within the rank (in bytes)
Function Function name
Time(sec) Transfer time for the message of this size (in seconds)
Time(%) Percentage of the message transfer time in the function time
Volume(MB) Amount of data transferred by the message of this size (in megabytes)
Volume(%) Percentage of the data transferred by the message in the data transferred by the function
Calls Transfers of messages of this size
MsgSize(B) Size of the message transferred by the function (in bytes)
TOTAL Total values for all columns