MinIO Cluster Benchmark Methods and Tools

Test environment deployment

Before benchmarking a cluster, especially a NUMA machine, we need to decide whether one socket or both sockets will be used for benchmarking, and then make the correct disk/NIC/RAM connection. Here are some commands to help get disk/NIC/RAM information and set CPU frequency scaling governor to “performance”.

1. Check system NUMA status:

numactl –hardware # show NUMA nodes and nodes distances.

numactl –show # show NUMA policy settings of the current process

2. Get PCIE bus and disk generation:

dmidecode | grep -ie pci

lspci -vv

3. Check disk and NIC connection:

lsblk    # list information about block devices

hwloc-ls   # list disk and NIC connection.

4.  Check NIC speed (100GbE network connection is recommended):

ethtool ${nic_name} 

5.  Get memory information:

dmidecode -t memory # to get memory connection and speed.

6. Set CPU frequency governor to “performance”:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 

cpupower frequency-set -g performance   # set CPU frequency scaling governor as “performance”

7. For a cluster benchmark, confirm clock synchronization across all storage servers and test clients in the cluster.  
8. Before benchmarking, run the following commands to ensure normalized performance of each drive.

write:   dd if=/dev/zero of=/dev/nvme0n1p1 bs=16M count=1024  oflag=direct

read:   dd if=/dev/nvme0n1p1 of=/dev/null bs=16M count=1024 iflag=direct

9. Run “fio” commands to get IO performance of each single disk and concurrent multiple disks:

fio -name=mytest -filename=/dev/nvme0n1  -direct=1  \

 -iodepth=20 -thread -rw=read|write|randread|randwrite -ioengine=libaio -bs=8m -size=10G  \

-runtime=300 -group_reporting



fio -name=mytest -filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1 -direct=1  \

 -iodepth=20 -thread -rw=read|write|randread|randwrite -ioengine=libaio -bs=8m -size=10G \

  -runtime=300 -group_reporting

10. Since MinIO runs above the filesystem, after setting up the filesystem, make sure JBOD (just a bunch of disks) performance is normal by running the dperf tool or the iozone tool.

Test JBOD with dperf

wget https://github.com/minio/dperf/releases/latest/download/dperf-linux-amd64

mv dperf-linux-amd64 dperf

chmod +x dperf

./dperf -b 4MiB /mnt/d{1..4}/t{1..40}

Test JBOD with iozone:

sudo apt install iozone3

iozone -s 1g -r 4m -i 0 -i 1 -i 2 -I -t 160 -b `hostname`-iozone.out -F /mnt/d1/tmpfile.{1..40}   /mnt/d2/tmpfile.{1..40}  /mnt/d3/tmpfile.{1..40}  /mnt/d4/tmpfile.{1..40}

Build and run MinIO

Download the latest MinIO binary from the MinIO web site.

wget https://dl.min.io/server/minio/release/linux-amd64/minio

Note that the binary has no debug symbols. If you need to profile MinIO performance, build MinIO with debug symbols enabled. Ensure you have the minimum Go version required to build MinIO from https://github.com/minio/minio/blob/master/go.mod   or check the Go version used to compile the MinIO published binary by running the following command:

./minio  --version   # get the MinIO version and the Go version used to build the MinIO binary.

wget https://go.dev/dl/go1.19.9.linux-amd64.tar.gz

rm -rf /usr/local/go/

tar -C /usr/local -xzf go1.19.9.linux-amd64.tar.gz

export PATH=/usr/local/go/bin:$PATH



mkdir ~/minio_src

cd  ~/minio_src

git clone https://github.com/minio/minio.git

cd minio

git tag

git checkout RELEASE.xxxxx

go build -v

Use the following commands to determine the parity disks used [3] out of the total disks available.  Use pdsh to start MinIO on the servers.

export MINIO_STORAGE_CLASS_STANDARD=EC:4

nohup ./minio server http://10.0.0.3{1...4}/mnt/d{1...4} 2>&1 > minio.log &

Benchmark MinIO

Warp[4] is an S3 benchmark tool. Run the warp client on each test client machine, then run the warp server on one test client.

mkdir ~/warp

cd ~/warp

wget https://github.com/minio/warp/releases/download/v0.6.9/warp_Linux_x86_64.tar.gz

tar zxvf warp_Linux_x86_64.tar.gz

nohup ./warp client 2>&1 >warp.log &

WARP_ACCESS_KEY=minioadmin WARP_SECRET_KEY=minioadmin warp $op  --warp-client 192.168.0.3{5…8}:7761  --host 192.168.0.3{1...4}:9000 --obj.size $size --concurrent $conn  --autoterm

Transport layer security configuration

Since object storage is transmitted via the network, transport layer security (TLS) is important for secure data transmission. An ECDSA-signed certificate is recommended because RSA decryption performance is poor in Go[5].  Use the following commands to generate an ECDSA signed certificate:

wget https://github.com/minio/certgen/releases/download/v1.2.1/certgen-linux-amd64

mv certgen-linux-amd64 certgen

chmod +x certgen

./certgen -host "10.0.0.31,10.0.0.32,10.0.0.33,10.0.0.34,192.168.0.31,192.168.0.32, 192,168.0.33,192.168.0.34,127.0.0.1" # the parameter is IP addresses of all servers.

Copy public certificate and private key files to each server:

cp private.key ~/.minio/certs/

cp public.crt ~/.minio/certs/

Copy public certificate file to each client:

cp public.crt /etc/ssl/certs/

Start MinIO with https protocol:

nohup ./minio server https://10.0.0.3{1...4}/mnt/d{1...4} 2>&1 > minio.log &

Start the WARP benchmark with TLS as true:

WARP_ACCESS_KEY=minioadmin WARP_SECRET_KEY=minioadmin warp $op  --warp-client 192.168.0.3{5…8}:7761  --host 192.168.0.3{1...4}:9000 --obj.size $size --concurrent $conn  --autoterm --tls=true --disable-http-keepalive=true  --autoterm

Encryption configuration

MinIO supports two types of server-side encryption (SSE): [7]

• SSE-C: The MinIO server encrypts and decrypts an object with a secret key provided by the S3 client as part of the HTTP request headers. Therefore, SSE-C requires TLS/HTTPS.
• SSE-S3: The MinIO server encrypts and decrypts an object with a secret key managed by a KMS. Therefore, MinIO requires a valid KMS configuration for SSE-S3.

MinIO supports multiple KMS implementations via our KES project. You can run a KES instance at https://play.min.io:7373 to experiment and quickly get started. [8]

curl -sSL --tlsv1.2   -O 'https://raw.githubusercontent.com/minio/kes/master/root.key' \
     -O 'https://raw.githubusercontent.com/minio/kes/master/root.cert'

export MINIO_KMS_KES_ENDPOINT=https://play.min.io:7373
export MINIO_KMS_KES_KEY_FILE=root.key
export MINIO_KMS_KES_CERT_FILE=root.cert
export MINIO_KMS_KES_KEY_NAME=my-minio-key
export MINIO_KMS_AUTO_ENCRYPTION=on

nohup ./minio server http://10.0.0.3{1...4}/mnt/d{1...4} 2>&1 > minio.log &

wget https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2022-02-07T09-25-34Z
mv mc.RELEASE.2022-02-07T09-25-34Z mc
chmod +x mc
./mc alias set myminio http://10.0.0.31:9000 minioadmin minioadmin
./mc encrypt set sse-s3 myminio/bucket/
./mc encrypt info myminio/bucket/

Monitor MinIO

Use Prometheus [9] and some Linux tools to monitor MinIO workloads.  Start the MinIO server with a Prometheus configuration using the following commands:

export MINIO_PROMETHEUS_AUTH_TYPE=”public”

export MINIO_PROMETHEUS_JOB_ID=”minio-job”

export MINIO_PROMETHEUS_URL=http://10.0.0.31:9090  



export MINIO_STORAGE_CLASS_STANDARD=EC:4

nohup ./minio server http://10.0.0.3{1…4}/mnt/d{1...4} 2>&1 > minio.log &

Run Prometheus on one MinIO server using the following commands:

wget https://github.com/prometheus/prometheus/releases/download/v2.44.0/prometheus-2.44.0.linux-amd64.tar.gz

tar xvfz prometheus-*.tar.gz

cd prometheus-*



vi prometheus.yml



scrape_configs:

- job_name: minio-job

  metrics_path: /minio/v2/metrics/cluster

  scheme: http

  static_configs:

  - targets: ['localhost:9000']



./prometheus –config.file=prometheus.yml

Get the MinIO status from the MinIO console by navigating to Monitoring -> Metrics -> Resources, as shown in the example below:

You can also use Linux tools to monitor MinIO runtime status, by running the following commands:

 pidstat -rp `pidof minio` 10     # Get CPU and memory usage.

iostat -dxctm 10 # Get disk status.

sar  -n DEV 10   # Get NIC status

Profile MinIO

Use the mc [10] tool, which was developed by MinIO, to profile MinIO.

1. Install mc using the following commands: 

wget https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2022-02-07T09-25-34Z
mv mc.RELEASE.2022-02-07T09-25-34Z mc
chmod +x mc
./mc alias set myminio http://10.0.0.31:9000 minioadmin minioadmin

2. Start CPU, memory, and block profiling concurrently:

./mc admin profile start --type cpu,mem,block myminio/

3. Stop profiling:

./mc admin profile stop myminio

4. View profiling result:

go tool pprof profile-10.0.0.31\:9000-cpu.pprof

go tool pprof -png profile-10.0.0.30\:9000-cpu.pprof > out.png

Summary

This document introduces the methods and tools we used to benchmark, monitor, and analyze MinIO cluster performance.  We used "fio” to check single and concurrent disk I/O performance. “dperf" and "iozone" were used to test JBOD performance. Warp [4] is a S3 benchmark tool to benchmark MinIO performance. Prometheus and Linux commands such as pidstat, iostat, and sar were used to monitor CPU/RAM/disk/NIC usage. The “mc” [10] tool, which was developed by MinIO as command line management tool, was used to profile MinIO.

Reference

1. https://blog.min.io/configurable-data-and-parity-drives-on-minio-server/
2. https://github.com/minio/minio/tree/master/docs/erasure
3. https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html
4. https://github.com/minio/warp
5. https://github.com/minio/minio/tree/master/docs/tls
6. crypto/rsa: Go 1.21 follow-up work #57752
7. https://github.com/minio/minio/tree/master/docs/security
8. https://github.com/minio/minio/blob/master/docs/kms/README.md
9. https://github.com/minio/minio/tree/master/docs/metrics/prometheus
10. https://min.io/docs/minio/linux/reference/minio-mc-admin/mc-admin-profile.html