Different MPI implementations provide different maximum tag values. The MPI specification only guarantees a value of 32767, which is the equivalent of 15 bits reserved for it. Therefore, the maximum tag value of a given MPI implementation may even between release versions of different kind of fabrics supported.
For the InfiniBand* support via the Intel® MPI mlx provider, Intel® MPI Library (2021.3) supports MPI tags up to a value of 1048575, which is the equivalent of 20 bits.
If a code exceeds the tag value of 32767 or 15 bits, the actual upper limit must be queried in the application. Otherwise, it creates a portability risk. The wrong assumption is that MPI_TAG_UB represents the maximum tag value. In reality, MPI_TAG_UB is an attribute that is used to retrieve the upper limit by querying the MPI_Comm_get_attr function . Simple examples are shown below:
- C Example
#include "mpi.h"
#include <stdio.h>
int main(int argc, char *argv[]) {
void *max_tag;
int flag;
MPI_Init(&argc, &argv);
/* The address of a void pointer must be used! */
MPI_Comm_get_attr( MPI_COMM_WORLD, MPI_TAG_UB, &max_tag, &flag);
if (flag) printf("Maximum tag value queried %d\n", *(int*)max_tag);
MPI_Finalize();
return (0);
}
- Fortran Example
program main
implicit none
include 'mpif.h'
integer max_tag, ierr
logical flag
call MPI_INIT(ierr)
call MPI_COMM_GET_ATTR( MPI_COMM_WORLD, MPI_TAG_UB, max_tag, flag, ierr)
if (flag) write(*,*) "Maximum tag value queried", max_tag
call MPI_FINALIZE(ierr)
end
There is a workaround in the Intel® MPI Library for shifting the trade-off between the maximum number of MPI ranks and the maximum tag value. The maximum tag value is related to the variable MPIR_CVAR_CH4_OFI_TAG_BITS:
max_tag_value = 2(MPIR_CVAR_CH4_OFI_TAG_BITS − 1) -1
max_mpi_ranks = 2 MPIR_CVAR_CH4_OFI_RANK_BITS -1
The trade-off means that both MPIR_CVAR_CH4_OFI_TAG_BITS and MPIR_CVAR_CH4_OFI_RANK_BITS must fit on a 39 bits environment:
MPIR_CVAR_CH4_OFI_TAG_BITS + MPIR_CVAR_CH4_OFI_RANK_BITS <= 39
For example, if the user desires tag values covering 30 bits, it can be divided as such
$ export MPIR_CVAR_CH4_OFI_TAG_BITS=31
$ export MPIR_CVAR_CH4_OFI_RANK_BITS=8
That would allow the use of large MPI tags, but restrict the use of Intel® MPI Library to only 8 bits – 256 MPI ranks.