Introduction
Distributed Asynchronous Object Storage (DAOS) is high-performance storage that pushes the limits of Intel hardware. It's based on Intel® Xeon® Scalable Processors, Intel® Optane™ Persistent Memory and NVMe SSDs.
This simple guide is meant for users that want to run DAOS on a minimal VM system running CentOS 7.9 without any Optane or NVMe storage. The installation steps are similar to a traditional DAOS install, though there are some slight differences with the configuration file and the emulation of storage using tmpfs.
RPM Installation
In this section the required RPMs will be installed on each of the nodes based upon their role. Admin and client nodes require the installation of the daos-client RPM and the server nodes require the installation of the daos-server RPM.
- Access to the DAOS Package repository at https://packages.daos.io/v2.0/
sudo wget -O /etc/yum.repos.d/daos-packages.repo https://packages.daos.io/v2.0/CentOS7/packages/x86_64/daos_packages.repo
- Perform the additional steps on all nodes:
sudo yum install -y epel-release - We install the DAOS server RPM’s on the server nodes:
sudo yum install -y daos-server - Now install the DAOS client RPMs on the client and admin nodes:
sudo yum install -y daos-client
Server Installation
On our server we will perform a Network Scan to find an available provider and note its respective interfaces that are available on our VM instance’s server.yml.
- Perform network scan with a provider, in this case we want ethernet so ofi+sockets is our provider in the server.yml
[root@localhost-server ~]# daos_server network scan DAOS Server config loaded from /etc/daos/daos_server.yml --------- localhost --------- ------------- NUMA Socket 0 ------------- Provider Interfaces -------- ---------- ofi+sockets enp0s3, virbr0 - Update our server.yml file accordingly, updating the fabric iface to match one of the available ones above with the given provider. In addition, specify a scm_mount directory and the size in GB we want to allocate for tmpfs to emulate in SCM. This step is different from a traditional DAOS install which could rely on Intel PMem or NVMe SSD’s.
[root@localhost-server ~]# vi /etc/daos/daos_server.yml#daos_server.yml name: daos_server access_points: ['localhost] port: 10001 transport_config: allow_insecure: true client_cert_dir: /etc/daos/certs/clients ca_cert: /etc/daos/certs/daosCA.crt cert: /etc/daos/certs/server.crt key: /etc/daos/certs/server.key provider: ofi+sockets socket_dir: /var/run/daos_server nr_hugepages: 4096 control_log_mask: DEBUG control_log_file: /tmp/daos_server.log helper_log_file: /tmp/daos_admin.log engines: - targets: 8 nr_xs_helpers: 0 fabric_iface: enp0s3 fabric_iface_port: 31316 log_mask: INFO log_file: /tmp/daos_engine_0.log env_vars: - CRT_TIMEOUT=30 scm_mount: /mnt/daos0 scm_class: ram scm_size: 16 #gb to allocate for tmpfs to emulate SCM - Create a new directory for our tmpfs mount. Then mount this directory and run df -h to verify the if the mounting is successful.
[root@localhost-server ~]# mkdir /mnt/daos0/ [root@localhost-server ~]# sudo mount -t tmpfs -o size=64G tmpfs /mnt/daos0 [root@localhost-server ~]# df -h /mnt/daos0 Filesystem Size Used Avail Use% Mounted on tmpfs 64G 0 64G 0% /mnt/daos0
Client Installation
Verify our server has joined with some simple dmg commands on our agent
- Update our daos_control.yml and our daos_agent.yml files to match our new daos_server.yml in step 2 of the Server Installation.
#daos_control.yml name: daos_server port: 10001 hostlist: ['localhost'] transport_config: allow_insecure: true ca_cert: /etc/daos/certs/daosCA.crt cert: /etc/daos/certs/admin.crt key: /etc/daos/certs/admin.key#daos_agent.yml name: daos_server access_points: ['localhost'] port: 10001 transport_config: allow_insecure: true ca_cert: /etc/daos/certs/daosCA.crt cert: /etc/daos/certs/agent.crt key: /etc/daos/certs/agent.key log_file: /tmp/daos_agent.log - Start our servers with the same commands you’d use in a typical DAOS installation
[root@localhost ~]# systemctl daemon-reload [root@localhost ~]# systemctl start daos_server [root@localhost ~]# systemctl status daos_server ● daos_server.service - DAOS Server Loaded: loaded (/usr/lib/systemd/system/daos_server.service; disabled; vendo> Active: active (running) since Sun 2022-01-23 12:07:28 EST; 12min ago Main PID: 3149 (daos_server) Tasks: 8 (limit: 23548) Memory: 12.2M CGroup: /system.slice/daos_server.service └─3149 /usr/bin/daos_server start - Start our agents with the same commands you’d use in a typical DAOS installation
[root@localhost daos]# sudo systemctl start daos_agent [root@localhost ~]# cat /tmp/daos_agent.log agent INFO 2022/01/23 12:20:02 DAOS Agent v2.0.0 (pid 3487) listening on /var/run/daos_agent/daos_agent.sock - Now that the server, and client are active, we can perform a storage format, in this case using our tmpfs mount we created earlier
[root@localhost ~]# df -h /mnt/daos0 Filesystem Size Used Avail Use% Mounted on tmpfs 64G 0 64G 0% /mnt/daos0 [root@localhost ~]# dmg storage format -l localhost.localdomain --force -i Format Summary: Hosts SCM Devices NVMe Devices ----- ----------- ------------ localhost.localdomain 1 0 - Using some commands from the DAOS Tour we can verify our DAOS instance and create a pool!
[root@localhost ~]# dmg system query -v Rank UUID Control Address Fault Domain State Reason ---- ---- --------------- ------------ ----- ------ 0 093ab6ea-e18e-4372-9963-ecb7e66801f8 127.0.0.1:10001 /localhost.localdomain Joined [root@localhost ~]# dmg storage query usage Hosts SCM-Total SCM-Free SCM-Used NVMe-Total NVMe-Free NVMe-Used ----- --------- -------- -------- ---------- --------- --------- localhost 17 GB 17 GB 0 % 0 B 0 B N/A [root@localhost ~]# dmg pool create --size=1G test_pool Creating DAOS pool with automatic storage allocation: 1.0 GB total, 6,94 tier ratio ^C [root@localhost ~]# dmg pool query test_pool Pool 7e1f5ffb-2f22-40e3-b8fb-c263ea694c3e, ntarget=1, disabled=0, leader=0, version=1 Pool space info: - Target(VOS) count:1 - Storage tier 0 (SCM): Total size: 1.0 GB Free: 1.0 GB, min:1.0 GB, max:1.0 GB, mean:1.0 GB - Storage tier 1 (NVMe): Total size: 0 B Free: 0 B, min:0 B, max:0 B, mean:0 B Rebuild idle, 0 objs, 0 recs