Environment Variables for Memory Policy Control
Intel® MPI Library supports non-uniform memory access (NUMA) nodes with high-bandwidth (HBW) memory (MCDRAM) on Intel® Xeon Phi™ processors (codenamed Knights Landing). Intel® MPI Library can attach memory of MPI processes to the memory of specific NUMA nodes. This section describes the environment variables for such memory placement control.
I_MPI_HBW_POLICY
Set the policy for MPI process memory placement for using HBW memory.
Syntax
I_MPI_HBW_POLICY=<user memory policy>[,<mpi memory policy>][,<win_allocate policy>]
In the syntax:
- <user memory policy>- memory policy used to allocate the memory for user applications (required)
- <mpi memory policy>- memory policy used to allocate the internal MPI memory (optional)
- <win_allocate policy>- memory policy used to allocate memory for window segments for RMA operations (optional)
Each of the listed policies may have the values below:
Arguments
<value> | The memory allocation policy used.
|
hbw_preferred | Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
|
hbw_bind | Allocate only the local HBW memory for each process.
|
hbw_interleave | Allocate the HBW memory and dynamic random access memory on the local node in the round-robin manner.
|
Description
Use this environment variable to specify the policy for MPI process memory placement on a machine with HBW memory.
By default, Intel MPI Library allocates memory for a process in local DDR. The use of HBW memory becomes available only when you specify the
I_MPI_HBW_POLICY
variable.
Examples
The following examples demonstrate different configurations of memory placement:
- I_MPI_HBW_POLICY=hbw_bind,hbw_preferred,hbw_bindOnly use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local HBW memory internally allocated in Intel® MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library.
- I_MPI_HBW_POLICY=hbw_bind,,hbw_bindOnly use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local DDR internally allocated in Intel MPI Library.
- I_MPI_HBW_POLICY=hbw_bind,hbw_preferredOnly use the local HBW memory allocated in user applications. Use the local HBW memory internally allocated in Intel MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library. Use the local DDR allocated in window segments for RMA operations.
I_MPI_BIND_NUMA
Set the NUMA nodes for memory allocation.
Syntax
I_MPI_BIND_NUMA=
<value>
Arguments
<value> | Specify the NUMA nodes for memory allocation.
|
localalloc | Allocate memory on the local node. This is the default value.
|
Node_1,…,Node_k | Allocate memory according to
I_MPI_BIND_ORDER on the specified NUMA nodes.
|
Description
Set this environment variable to specify the NUMA node set that is involved in the memory allocation procedure.
I_MPI_BIND_ORDER
Set this environment variable to define the memory allocation manner.
Syntax
I_MPI_BIND_ORDER=
<value>
Arguments
<value> | Specify the allocation manner.
|
compact | Allocate memory for processes as close as possible (in terms of NUMA nodes), among the NUMA nodes specified in
I_MPI_BIND_NUMA . This is the default value.
|
scatter | Allocate memory among the NUMA nodes specified in
I_MPI_BIND_NUMA using the round-robin manner.
|
Description
Set this environment variable to define the memory allocation manner among the NUMA nodes specified in
I_MPI_BIND_NUMA
. The variable has no effect without
I_MPI_BIND_NUMA
set.
I_MPI_BIND_WIN_ALLOCATE
Set this environment variable to control memory allocation for window segments.
Syntax
I_MPI_BIND_WIN_ALLOCATE=
<value>
Arguments
<value> | Specify the memory allocation behavior for window segments.
|
localalloc | Allocate memory on the local node. This is the default value.
|
hbw_preferred | Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
|
hbw_bind | Allocate only the local HBW memory for each process.
|
hbw_interleave | Allocate the HBW memory and dynamic random access memory on a local node in the round-robin manner.
|
<NUMA node id> | Allocate memory on the given NUMA node.
|
Description
Set this environment variable to create window segments allocated in HBW memory with the help of the
MPI_Win_allocate_shared
or
MPI_Win_allocate
functions.
MPI_Info
You can control memory allocation for window segments with the help of an
MPI_Info
object, which is passed as a parameter to the
MPI_Win_allocate
or
MPI_Win_allocate_shared
function. In an application, if you specify such an object with the
numa_bind_policy
key, window segments are allocated in accordance with the value for
numa_bind_policy
. Possible values are the same as for
I_MPI_BIND_WIN_ALLOCATE
.
A code fragment demonstrating the use of
MPI_Info
:
MPI_Info info; ... MPI_Info_create( &info ); MPI_Info_set( info, "numa_bind_policy", "hbw_preferred" ); ... MPI_Win_allocate_shared( size, disp_unit, info, comm, &baseptr, &win );
When you specify the memory placement policy for window segments, Intel MPI Library recognizes the configurations according to the following priority:
- Setting ofMPI_Info.
- Setting ofI_MPI_HBW_POLICY, if you specified<win_allocate policy>.
- Setting ofI_MPI_BIND_WIN_ALLOCATE.