Large MPI Tags with the Intel® MPI Library

Different MPI implementations provide different maximum tag values. The MPI specification only guarantees a value of 32767, which is the equivalent of 15 bits reserved for it. Therefore, the maximum tag value of a given MPI implementation may even between release versions of different kind of fabrics supported.

For the InfiniBand* support via the Intel® MPI mlx provider, Intel® MPI Library (2021.3) supports MPI tags up to a value of 1048575, which is the equivalent of 20 bits.

If a code exceeds the tag value of 32767 or 15 bits, the actual upper limit must be queried in the application. Otherwise, it creates a portability risk. The wrong assumption is that MPI_TAG_UB represents the maximum tag value. In reality, MPI_TAG_UB is an attribute that is used to retrieve the upper limit by querying the MPI_Comm_get_attr function . Simple examples are shown below:

• C Example

#include "mpi.h"
#include <stdio.h>
int main(int argc, char *argv[]) {
    void *max_tag;
    int flag;
    MPI_Init(&argc, &argv);
    /* The address of a void pointer must be used! */
    MPI_Comm_get_attr( MPI_COMM_WORLD, MPI_TAG_UB, &max_tag, &flag);
    if (flag) printf("Maximum tag value queried %d\n", *(int*)max_tag);
    MPI_Finalize();
    return (0);
}

• Fortran Example

program main
  implicit none
  include 'mpif.h'
  integer  max_tag, ierr
  logical flag
  call MPI_INIT(ierr)
  call MPI_COMM_GET_ATTR( MPI_COMM_WORLD, MPI_TAG_UB, max_tag, flag, ierr)
  if (flag) write(*,*) "Maximum tag value queried", max_tag
  call MPI_FINALIZE(ierr)
end

There is a workaround in the Intel® MPI Library for shifting the trade-off between the maximum number of MPI ranks and the maximum tag value. The maximum tag value is related to the variable MPIR_CVAR_CH4_OFI_TAG_BITS:

max_tag_value = 2^{(MPIR_CVAR_CH4_OFI_TAG_BITS − 1)} -1
max_mpi_ranks = 2 ^{MPIR_CVAR_CH4_OFI_RANK_BITS}  -1

The trade-off means that both MPIR_CVAR_CH4_OFI_TAG_BITS and MPIR_CVAR_CH4_OFI_RANK_BITS must fit on a 39 bits environment:
MPIR_CVAR_CH4_OFI_TAG_BITS + MPIR_CVAR_CH4_OFI_RANK_BITS <= 39

For example, if the user desires tag values covering 30 bits, it can be divided as such

$ export MPIR_CVAR_CH4_OFI_TAG_BITS=31
$ export MPIR_CVAR_CH4_OFI_RANK_BITS=8

That would allow the use of large MPI tags, but restrict the use of Intel® MPI Library to only 8 bits – 256 MPI ranks.