Intel® MPI Library Developer Reference for Linux* OS

Developer Reference

Contents

Environment Variables for Memory Policy Control

Intel® MPI Library supports non-uniform memory access (NUMA) nodes with high-bandwidth (HBW) memory (MCDRAM) on Intel® Xeon Phi™ processors (codenamed Knights Landing). Intel® MPI Library can attach memory of MPI processes to the memory of specific NUMA nodes. This section describes the environment variables for such memory placement control.

I_MPI_HBW_POLICY

Set the policy for MPI process memory placement for using HBW memory.
Syntax
I_MPI_HBW_POLICY=<user memory policy>[,<mpi memory policy>][,<win_allocate policy>]
In the syntax:
  • <user memory policy>
     - memory policy used to allocate the memory for user applications (required)
  • <mpi memory policy>
     - memory policy used to allocate the internal MPI memory (optional)
  • <win_allocate policy>
     - memory policy used to allocate memory for window segments for RMA operations (optional)
Each of the listed policies may have the values below:
Arguments
<value>
The memory allocation policy used.
hbw_preferred
Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
hbw_bind
Allocate only the local HBW memory for each process.
hbw_interleave
Allocate the HBW memory and dynamic random access memory on the local node in the round-robin manner.
Description
Use this environment variable to specify the policy for MPI process memory placement on a machine with HBW memory.
By default, Intel MPI Library allocates memory for a process in local DDR. The use of HBW memory becomes available only when you specify the
I_MPI_HBW_POLICY
 variable.

Examples

The following examples demonstrate different configurations of memory placement:
  • I_MPI_HBW_POLICY=hbw_bind,hbw_preferred,hbw_bind
    Only use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local HBW memory internally allocated in Intel® MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library.
  • I_MPI_HBW_POLICY=hbw_bind,,hbw_bind
    Only use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local DDR internally allocated in Intel MPI Library.
  • I_MPI_HBW_POLICY=hbw_bind,hbw_preferred
    Only use the local HBW memory allocated in user applications. Use the local HBW memory internally allocated in Intel MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library. Use the local DDR allocated in window segments for RMA operations.

I_MPI_BIND_NUMA

Set the NUMA nodes for memory allocation.
Syntax
I_MPI_BIND_NUMA=
<value>
Arguments
<value>
Specify the NUMA nodes for memory allocation.
localalloc
Allocate memory on the local node. This is the default value.
Node_1,…,Node_k
Allocate memory according to
I_MPI_BIND_ORDER
 on the specified NUMA nodes.
Description
Set this environment variable to specify the NUMA node set that is involved in the memory allocation procedure.

I_MPI_BIND_ORDER

Set this environment variable to define the memory allocation manner.
Syntax
I_MPI_BIND_ORDER=
<value>
Arguments
<value>
Specify the allocation manner.
compact
Allocate memory for processes as close as possible (in terms of NUMA nodes), among the NUMA nodes specified in
I_MPI_BIND_NUMA
. This is the default value.
scatter
Allocate memory among the NUMA nodes specified in
I_MPI_BIND_NUMA
 using the round-robin manner.
Description
Set this environment variable to define the memory allocation manner among the NUMA nodes specified in
I_MPI_BIND_NUMA
. The variable has no effect without
I_MPI_BIND_NUMA
set.

I_MPI_BIND_WIN_ALLOCATE

Set this environment variable to control memory allocation for window segments.
Syntax
I_MPI_BIND_WIN_ALLOCATE=
<value>
Arguments
<value>
Specify the memory allocation behavior for window segments.
localalloc
Allocate memory on the local node. This is the default value.
hbw_preferred
Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
hbw_bind
Allocate only the local HBW memory for each process.
hbw_interleave
Allocate the HBW memory and dynamic random access memory on a local node in the round-robin manner.
<NUMA node id>
Allocate memory on the given NUMA node.
Description
Set this environment variable to create window segments allocated in HBW memory with the help of the
MPI_Win_allocate_shared
 or
MPI_Win_allocate
 functions.

MPI_Info

You can control memory allocation for window segments with the help of an
MPI_Info
 object, which is passed as a parameter to the
MPI_Win_allocate
 or
MPI_Win_allocate_shared
 function. In an application, if you specify such an object with the
numa_bind_policy
 key, window segments are allocated in accordance with the value for
numa_bind_policy
. Possible values are the same as for
I_MPI_BIND_WIN_ALLOCATE
.
A code fragment demonstrating the use of
MPI_Info
: 
MPI_Info info;
...
MPI_Info_create( &info );
MPI_Info_set( info, "numa_bind_policy", "hbw_preferred" );
...
MPI_Win_allocate_shared( size, disp_unit, info, comm, &baseptr, &win );
When you specify the memory placement policy for window segments, Intel MPI Library recognizes the configurations according to the following priority:
  1. Setting of
    MPI_Info.
  2. Setting of
    I_MPI_HBW_POLICY
    , if you specified
    <win_allocate policy>.
  3. Setting of
    I_MPI_BIND_WIN_ALLOCATE.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.