Large MPI Tags with the Intel® MPI Library

ID 676380
Updated 7/1/2021
Version Latest



Different MPI implementations provide different maximum tag values. The MPI specification only guarantees a value of 32767, which is the equivalent of 15 bits reserved for it. Therefore, the maximum tag value of a given MPI implementation may even between release versions of different kind of fabrics supported.

For the InfiniBand* support via the Intel® MPI mlx provider, Intel® MPI Library (2021.3) supports MPI tags up to a value of 1048575, which is the equivalent of 20 bits.

If a code exceeds the tag value of 32767 or 15 bits, the actual upper limit must be queried in the application. Otherwise, it creates a portability risk. The wrong assumption is that MPI_TAG_UB represents the maximum tag value. In reality, MPI_TAG_UB is an attribute that is used to retrieve the upper limit by querying the MPI_Comm_get_attr function . Simple examples are shown below:

  • C Example
#include "mpi.h"
#include <stdio.h>
int main(int argc, char *argv[]) {
    void *max_tag;
    int flag;
    MPI_Init(&argc, &argv);
    /* The address of a void pointer must be used! */
    MPI_Comm_get_attr( MPI_COMM_WORLD, MPI_TAG_UB, &max_tag, &flag);
    if (flag) printf("Maximum tag value queried %d\n", *(int*)max_tag);
    return (0);
  • Fortran Example
program main
  implicit none
  include 'mpif.h'
  integer  max_tag, ierr
  logical flag
  call MPI_INIT(ierr)
  call MPI_COMM_GET_ATTR( MPI_COMM_WORLD, MPI_TAG_UB, max_tag, flag, ierr)
  if (flag) write(*,*) "Maximum tag value queried", max_tag
  call MPI_FINALIZE(ierr)

There is a workaround in the Intel® MPI Library for shifting the trade-off between the maximum number of MPI ranks and the maximum tag value. The maximum tag value is related to the variable MPIR_CVAR_CH4_OFI_TAG_BITS:

max_tag_value = 2(MPIR_CVAR_CH4_OFI_TAG_BITS − 1) -1
max_mpi_ranks = 2 MPIR_CVAR_CH4_OFI_RANK_BITS  -1

The trade-off means that both MPIR_CVAR_CH4_OFI_TAG_BITS and MPIR_CVAR_CH4_OFI_RANK_BITS must fit on a 39 bits environment:

For example, if the user desires tag values covering 30 bits, it can be divided as such


That would allow the use of large MPI tags, but restrict the use of Intel® MPI Library to only 8 bits – 256 MPI ranks.