Abstract
Persistent memory technologies such as Intel® Optane™ persistent memory come with several challenges. Remote access seems to be one of the most challenging aspects of persistent memory applications. The most commonly used remote direct memory access (RDMA) for remote persistent memory access does not consider data durability aspects yet. Anyhow there are already available solutions that address this problem. This short paper gives an overview of how to access remote persistent memory (RPMEM) based on traditional RDMA.
Introduction
Traditional RDMA works fine with persistent memory. Intel Optane persistent memory can be handled the same way as standard DRAM is whenever persistency is not required.
However, RDMA does not understand persistence. A consumer can access remote persistent memory for reading and writing with standard RDMA Verbs. But writing to RPMEM does not ensure immediate data durability. A user must perform an additional action to guarantee remote data persistence.
Read-after-write, send-after-write, or read-after-send sequences ensure data durability on an Intel servers platform with Intel Optane persistent memory. A remote durability technique selection depends on a server configuration and an application workload characteristic.
There are several working solutions available on the market that already utilize remote persistent memory, among which the most interesting are:
- Oracle* Exadata X8M
- NetApp* MAX Data and Max Recovery
Hardware
An Intel Server platform with Intel Optane persistent memory is required to build a system with a remote persistent memory. The server must be equipped with a standard RDMA capable NIC that supports one of the existing RDMA protocols (iWarp*, InfiniBand*, RoCE). The solution is supported on Windows* and Linux* OS platforms.
Linux* Software Stack
Several software components must be available on the initiator and the target node to start working with a remote persistent memory. First of all, a vendor-specific driver must be installed to get access to RDMA hardware. The rest of the software configuration depends on the way how an application would like to utilize RDMA and remote persistent memory. Following options are available:
- the rds-rdma kernel module based on kernel level verbs API,
- rds-rdma is a zero-copy interface of Reliable Datagram Socket (RDS). It provides reliable, in-order datagram delivery between sockets over an InfiniBand transport.
- the libibverbs library - Linux kernel’s user level verbs API,
- libibverbs is a library that allows userspace processes to use RDMA services as described in the InfiniBand™ Architecture Specification and the RDMA Protocol Verbs Specification.
- the libfabric library based on libibverbs library
- libibverbs is a library that allows userspace processes to use RDMA services as described in the InfiniBand™ Architecture Specification and the RDMA Protocol Verbs Specification.
An application can build its algorithm for accessing remote persistent memory based on any of the mentioned above hardware-software solutions. The way to access remote persistent memory is similar for Memory Mode and App Direct mode. Both Memory Mode and App Direct mode can be accessed in the same way.
Add additional logic to ensure data durability in the App Direct mode must be implemented by an application.
Support
There is an experimental librpmem library provided by Persistent Memory Development Kit. The librpmem library is utilized to ensure data durability for the libpmemobj remote replication mechanism. The librpmem library supports read-after-write and send-after-write techniques. Examples are also available on GitHub* repository.
Recently the PMDK team has decided to convert the experimental librpmem library into production-quality software. The latest status of the development could be followed on the librpma repository on GitHub.
The PMDK team also provides support for RDMA/PMem integration.
References
White paper: Persistent Memory Replication Over Traditional RDMA
Persistent Memory Development Kit [Online]. Available at GitHub PMEM PMDK
Persistent Memory Programming [Online]. Available at pmem.io
The librpmem library [Online]. Available at pmem.io
C. Douglas, RDMA with PMEM: Software mechanisms for enabling access to remote persistent memory, in Storage Developer Conference, 2015.