Use Memkind to Manage Large Volatile Memory Capacity on Intel® Optane™ DC Persistent Memory

Overview

This article explores the memkind library and how to use it with Intel® Optane™ DC Persistent Memory. Learn how to configure your application for this usage, with examples included in an accompanying code sample. Go behind the scenes to learn what happens when you request memory, and see some of the API calls used for programming with memkind.

This article assumes you have basic knowledge of persistent memory programming concepts. If not, visit the Intel® Developer Zone Persistent Memory site to find the information you need to get started.

Memkind Support for Persistent Memory

For applications that can benefit from large memory capacity, Intel® Optane™ DC persistent memory modules (Intel® Optane™ DC PMM) can be a practical solution. These non-volatile dual in-line memory modules (NVDIMMs) support much larger capacities compared to other NVDIMMs on the market and can be partitioned into volatile and persistent memory pools. To learn more about non-volatile memory management, check out the Persistent Memory Development Kit (PMDK). This article focuses on a use case where memkind is used to create volatile memory on top of a persistent memory region. The usage is referred to as volatile over app-direct. To learn more about persistent memory use cases, read New Software Development Opportunities with Big, Affordable, Persistent Memory.

Managing pools of different kinds of memory can be done using the memkind library, which is a user-extensible memory allocator built atop jemalloc. Memkind enables control of memory characteristics and partitioning of the heap between kinds of memory. ‘Kind’ and ‘ size’ are keywords to describe memkind’s functionality. When persistent memory was introduced to memkind, a new kind was added called, pmem_kind.

Benefits of using pmem_kind include:

• Applications see separate pools of memory for DRAM and persistent memory
• Applications have more memory available for data sets
• Latency-sensitive data goes into DRAM for optimal quality of service
• Applications can manage data placement

How it Works

Memkind uses memory mapped files to create the perception of a volatile region.

As mentioned earlier, memkind is built on top of the heap manager jemalloc. Jemalloc is a memory allocator that emphasizes fragmentation avoidance and scalable concurrency support. Memkind provides jemalloc with a source (kind) of memory. In the case of persistent memory, that source is a memory mapped file. Memkind uses memory-mapped files when creating volatile regions on the persistent memory device.

A temporary file is created on the DAX-enabled file system and memory-mapped into the application’s virtual address space. Jemalloc uses this address space to allocate objects. The file is automatically deleted when the program terminates, giving the perception of volatility.

Two key points to understand are:

• Jemalloc is responsible for managing the heap. Memkind is just a wrapper that redirects jemalloc’s memory requests to a different place.
• The persistent memory type pmem_kind references memory that is backed by a memory-mapped file, preferably on an Intel Optane DC PMM device. When using this device with memkind, the data is not actually persistent.

Using Memkind

Provision Persistent Memory

Before you can use memkind on an Intel Optane DC PMM, the persistent memory must first be provisioned into namespaces to create the logical device /dev/pmem. Find more information about persistent namespaces and how to create them in the ndctl user guide.

When emulating persistent memory, there is no need to use ndctl for this step. When you reboot, a new /dev/pmem{N} device will be created. The first device is generally named /dev/pmem0, and the naming convention is that N will increment for each additional device.

Create and Mount the DAX Filesystem

A DAX enabled filesystem, including EXT4, XFS, or NTFS, can be mounted on this device with the –o dax option. The following commands show how to create and mount a filesystem.

$ sudo mkfs.ext4 /dev/pmem0p1
$ sudo mkdir /pmem1
$ sudo mount -o dax /dev/pmem0p1 /pmem1
$ sudo mount -v | grep /pmem1
/dev/pmem0p1 on /pmem1 type ext4 (rw,relatime,seclabel,dax,data=ordered)

Set Up Your Dev Environment

You can find downloads, sources, and binaries on GitHub.

Once downloaded, memkind can be built using the following scripts. More detailed build instructions are available on the memkind GitHub* site.

Build jemalloc first:

$ export JE_PREFIX=jemk
$ ./build_jemalloc.sh

Build memkind:

$ ./build.sh
$ make install

Memkind API

The memkind API calls relating to persistent memory programming are listed below. The complete memkind API can be found on the memkind man pages.

• memkind_create_pmem(const char *dir, size_t max_size, memkind_t *kind)
• memkind_malloc(memkind_t kind, size_t size)
• memkind_calloc(memkind_t kind, size_t num, size_t size)
• memkind_realloc(memkind_t kind, void *ptr, size_t size)
• memkind_free(memkind_t kind, void *ptr)
• C++ bindings for defining pmem kind allocator in STL containers

Example Usage

This example explores how volatile memory is allocated using memkind’s pmem_kind. First, we allocate variables to the region and see what happens when we try to allocate outside the memory capacity. After that, we demonstrate freeing and deleting the persistent memory partition. Find the complete pmem_malloc.c source code in the memkind examples directory on GitHub.

In memkind, memkind_malloc is one of the more common functions used. It allocates size bytes of uninitialized storage of the specified kind. Its function header looks like this:

void *memkind_malloc(memkind_t kind, size_t size);

Before doing this, we need to create the memory source for pmem_kind.

Creating pmem_kind

As previously mentioned, the persistent memory type in memkind is a file-backed memory type. This means that a temporary file is created and the allocations are done there.

In this example, the default directory is /tmp/, but a diffrent one can be declared by the user at runtime. Creating a directory on the DAX enabled filesystem mounted on /pmem{N} and passing the directory path at runtime allows in-place updates. This is demonstrated below:

$ cd /pmem1
$ mkdir myDir
$ ./pmem_malloc myDir
This example shows how to allocate memory and possibility to exceed pmem kind size.
PMEM kind directory: myDir

If specified, the path to the directory will be stored in the PMEM_DIR variable.

A partition of memory is created with either the memkind_create_kind() function or the memkind_create_pmem() function; we will demonstrate using the latter.

err = memkind_create_pmem(PMEM_DIR, PMEM_MAX_SIZE, &pmem_kind);
	if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition\n");
return errno ? -errno : 1;
}

memkind_create_pmem takes in the parameters PMEM_DIR, PMEM_MAX_SIZE, and pmem_kind, and creates a temporary file at the path specified by PMEM_DIR. PMEM_MAX_SIZE is defined in this program as (1024*1024*32) which is equal to 32 megabytes. pmem_kind is defined as struct memkind *pmem_kind = NULL.

Assuming all goes well, the call to memkind_create_pmem returns the MEMKIND_SUCCESS enum on success or MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE or MEMKIND_ERROR_INVALID on failure.

As an alternative to PMEM_MAX_SIZE, the size parameter could be set to 0, which would mean the allocation automatically grows when it runs out of space.

Allocating Data Structures

Next, we declare variables pmem_str1 – pmem_str4. These are used to allocate memory in the next section.

As briefly covered earlier, the memkind_malloc function allocates size bytes of uninitialized kind memory. In this case, the ­kind is always going to be pmem_kind, but the size will change based on each memory allocation.

In this first call, we allocate 512 bytes of pmem_kind.

// allocate 512 Bytes of 32 MB available
pmem_str1 = (char *)memkind_malloc(pmem_kind, 512);
if (pmem_str1 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string(pmem_str1)\n");
return errno ? -errno : 1;
}

If pmem_str1 equals NULL on return, then the call to memkind_malloc failed. We'll see an example of this on the pmem_str4 allocation.

The next few calls do the same thing. Each one allocates more of the memory partition.

// allocate 8 of 31.9 MB available
    pmem_str2 = (char *)memkind_malloc(pmem_kind, 8 * 1024 * 1024);   
    if (pmem_str2 == NULL) {
        perror("memkind_malloc()");
        fprintf(stderr, "Unable to allocate pmem string (pmem_str11)\n");
        return errno ? -errno : 1;
    }

    // allocate 16 of 23.9 MB available
    pmem_str3 = (char *)memkind_malloc(pmem_kind, 16 * 1024 * 1024); 
    if (pmem_str3 == NULL) {
        perror("memkind_malloc()");
        fprintf(stderr, "Unable to allocate pmem string (pmem_str12)\n");
        return errno ? -errno : 1;
}

    // allocate 16 of 7.9 MB available -- Out of Memory expected      
    pmem_str4 = (char *)memkind_malloc(pmem_kind, 16 * 1024 * 1024);   
    if (pmem_str4 != NULL) {
        perror("memkind_malloc()");
        fprintf(stderr,
                "Failure, this allocation should not be possible (expected result was NULL)\n");
        return errno ? -errno : 1;
    }

The image below shows a visualization of the memory allocation filling up:

The last memory allocation tries to allocate 16 MB of memory to pmem_str4 when only about 7.9 MB are left in the partition. This results in an out-of-memory error. This time, the ‘if’ statement checking for error has changed to be if pmem_str4 != NULL since we’re expecting the call to memkind_malloc to fail.

Since pmem_str4 is equal to NULL, an error would occur if we tried to assign any values to that memory. We move ahead with assigning values to only the first three variables.

sprintf(pmem_str1, "Hello world from persistent memory - pmem_str1\n"); 
sprintf(pmem_str2, "Hello world from persistent memory - pmem_str2\n");     
sprintf(pmem_str3, "Hello world from persistent memory - pmem_str3\n");   

fprintf(stdout, "%s", pmem_str1); 
fprintf(stdout, "%s", pmem_str2);
fprintf(stdout, "%s", pmem_str3);

The memory allocated in each of the variables is of type pmem_kind, which means that it is allocated as a chunk inside a memory-mapped file residing in a persistent memory aware filesystem. However, the file will be deleted when the program ends, which effectively makes these allocations volatile.

Freeing the Memory

Finally, we free the allocated memory and delete the pmem_kind that was created at the start of the program. It is necessary to free the allocated memory before deleting the kind. Otherwise, a memory leak will occur. Note: there is no need to free pmem_str4. Since that method did not complete, no memory was allocated.

memkind_free(pmem_kind, pmem_str1);
memkind_free(pmem_kind, pmem_str2);
memkind_free(pmem_kind, pmem_str3);

err = memkind_destroy_kind(pmem_kind);
    	if (err) {
    	    perror("memkind_destroy_kind()");
     	    fprintf(stderr, "Unable to destroy pmem partition\n");
    	    return errno ? -errno : 1;
}

memkind_destroy_kind is used to delete any kind that was initialized by memkind_create_kind() or memkind_create_pmem(). Pmem_kind was initialized at the start of the program with memkind_create_pmem(), so it will be effectively destroyed. Upon success, this function returns MEMKIND_SUCCESS or MEMKIND_ERROR_OPERATION_FAILED on failure.

Summary

Memkind is a powerful tool for managing different memory types. In this example, we looked at how memkind can be used with Intel Optane DC Persistent Memory modules to store large amounts of volatile data. By using temporary memory-mapped files, memkind gives the perception of volatility as shown in the memkind_malloc.c example.

Find more source code examples in memkind’s examples repo on GitHub. Familiarize yourself with memkind, and give it a try. Let us know how it works by leaving a message in the comments section below.

About the Authors

Kelly Lyon is a developer advocate at Intel Corporation with three years of experience as a software engineer. Kelly is dedicated to advocating for users and looks forward to bringing clarity to complex ideas by researching and providing simple, easy-to-understand guides and tutorials. Follow her journey on Twitter @a_lyons_tale.

Usha Upadhyayula is a software ecosystem enabling engineer for Intel memory products.

Resources:

• Intel® Developer Zone (Intel® DZ) site for persistent memory
• Jemalloc
• Memkind
• Memkind Examples Directory
• Memkind Man Pages
• Memkind Build Instructions
• Ndctl User Guide
• Persistent Memory Development Kit (PMDK)
• Persistent Memory GitHub
• Pmem_malloc.c Source Code