MinIO Cluster Benchmark Methods and Tools

Benchmark, monitor, and analyze MinIO cluster performance

  • Checked single and concurrent disk I/O performance using fio

  • Tested JBOD performance using dperf and iozone

  • Benchmarked MinIO performance with Warp

  • Monitored CPU/RAM/disk/NIC usage with Prometheus and Linux commands

  • Profiled MinIO using the "mc" tool



Test environment deployment

Before benchmarking a cluster, especially a NUMA machine, we need to decide whether one socket or both sockets will be used for benchmarking, and then make the correct disk/NIC/RAM connection. Here are some commands to help get disk/NIC/RAM information and set CPU frequency scaling governor to “performance”.

  1. Check system NUMA status:
numactl –hardware # show NUMA nodes and nodes distances.

numactl –show # show NUMA policy settings of the current process


  1. Get PCIE bus and disk generation:
dmidecode | grep -ie pci

lspci -vv


  1. Check disk and NIC connection:
lsblk    # list information about block devices

hwloc-ls   # list disk and NIC connection.


  1.  Check NIC speed (100GbE network connection is recommended):
ethtool ${nic_name} 


  1.  Get memory information:
dmidecode -t memory # to get memory connection and speed.


  1. Set CPU frequency governor to “performance”:
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 

cpupower frequency-set -g performance   # set CPU frequency scaling governor as “performance”


  1. For a cluster benchmark, confirm clock synchronization across all storage servers and test clients in the cluster.  
  2. Before benchmarking, run the following commands to ensure normalized performance of each drive.
write:   dd if=/dev/zero of=/dev/nvme0n1p1 bs=16M count=1024  oflag=direct

read:   dd if=/dev/nvme0n1p1 of=/dev/null bs=16M count=1024 iflag=direct


  1. Run “fio” commands to get IO performance of each single disk and concurrent multiple disks:
fio -name=mytest -filename=/dev/nvme0n1  -direct=1  \

 -iodepth=20 -thread -rw=read|write|randread|randwrite -ioengine=libaio -bs=8m -size=10G  \

-runtime=300 -group_reporting

fio -name=mytest -filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1 -direct=1  \

 -iodepth=20 -thread -rw=read|write|randread|randwrite -ioengine=libaio -bs=8m -size=10G \

  -runtime=300 -group_reporting


  1. Since MinIO runs above the filesystem, after setting up the filesystem, make sure JBOD (just a bunch of disks) performance is normal by running the dperf tool or the iozone tool.

Test JBOD with dperf


mv dperf-linux-amd64 dperf

chmod +x dperf

./dperf -b 4MiB /mnt/d{1..4}/t{1..40}

Test JBOD with iozone:

sudo apt install iozone3

iozone -s 1g -r 4m -i 0 -i 1 -i 2 -I -t 160 -b `hostname`-iozone.out -F /mnt/d1/tmpfile.{1..40}   /mnt/d2/tmpfile.{1..40}  /mnt/d3/tmpfile.{1..40}  /mnt/d4/tmpfile.{1..40}


Build and run MinIO

Download the latest MinIO binary from the MinIO web site.



Note that the binary has no debug symbols. If you need to profile MinIO performance, build MinIO with debug symbols enabled. Ensure you have the minimum Go version required to build MinIO from   or check the Go version used to compile the MinIO published binary by running the following command:

./minio  --version   # get the MinIO version and the Go version used to build the MinIO binary.


rm -rf /usr/local/go/

tar -C /usr/local -xzf go1.19.9.linux-amd64.tar.gz

export PATH=/usr/local/go/bin:$PATH

mkdir ~/minio_src

cd  ~/minio_src

git clone

cd minio

git tag

git checkout RELEASE.xxxxx

go build -v



Use the following commands to determine the parity disks used [3] out of the total disks available.  Use pdsh to start MinIO on the servers.


nohup ./minio server{1...4}/mnt/d{1...4} 2>&1 > minio.log &


Benchmark MinIO

Warp[4] is an S3 benchmark tool. Run the warp client on each test client machine, then run the warp server on one test client.

mkdir ~/warp

cd ~/warp


tar zxvf warp_Linux_x86_64.tar.gz

nohup ./warp client 2>&1 >warp.log &

WARP_ACCESS_KEY=minioadmin WARP_SECRET_KEY=minioadmin warp $op  --warp-client{5…8}:7761  --host{1...4}:9000 --obj.size $size --concurrent $conn  --autoterm



Transport layer security configuration

Since object storage is transmitted via the network, transport layer security (TLS) is important for secure data transmission. An ECDSA-signed certificate is recommended because RSA decryption performance is poor in Go[5].  Use the following commands to generate an ECDSA signed certificate:


mv certgen-linux-amd64 certgen

chmod +x certgen

./certgen -host ",,,,,, 192,168.0.33,," # the parameter is IP addresses of all servers.


Copy public certificate and private key files to each server:

cp private.key ~/.minio/certs/

cp public.crt ~/.minio/certs/


Copy public certificate file to each client:

cp public.crt /etc/ssl/certs/


Start MinIO with https protocol:

nohup ./minio server{1...4}/mnt/d{1...4} 2>&1 > minio.log &


Start the WARP benchmark with TLS as true:

WARP_ACCESS_KEY=minioadmin WARP_SECRET_KEY=minioadmin warp $op  --warp-client{5…8}:7761  --host{1...4}:9000 --obj.size $size --concurrent $conn  --autoterm --tls=true --disable-http-keepalive=true  --autoterm


Encryption configuration

MinIO supports two types of server-side encryption (SSE): [7]

  • SSE-C: The MinIO server encrypts and decrypts an object with a secret key provided by the S3 client as part of the HTTP request headers. Therefore, SSE-C requires TLS/HTTPS.
  • SSE-S3: The MinIO server encrypts and decrypts an object with a secret key managed by a KMS. Therefore, MinIO requires a valid KMS configuration for SSE-S3.

MinIO supports multiple KMS implementations via our KES project. You can run a KES instance at to experiment and quickly get started. [8]

curl -sSL --tlsv1.2   -O '' \
     -O ''

export MINIO_KMS_KES_KEY_FILE=root.key
export MINIO_KMS_KES_CERT_FILE=root.cert
export MINIO_KMS_KES_KEY_NAME=my-minio-key

nohup ./minio server{1...4}/mnt/d{1...4} 2>&1 > minio.log &

mv mc.RELEASE.2022-02-07T09-25-34Z mc
chmod +x mc
./mc alias set myminio minioadmin minioadmin
./mc encrypt set sse-s3 myminio/bucket/
./mc encrypt info myminio/bucket/


Monitor MinIO

Use Prometheus [9] and some Linux tools to monitor MinIO workloads.  Start the MinIO server with a Prometheus configuration using the following commands:


export MINIO_PROMETHEUS_JOB_ID=”minio-job”



nohup ./minio server{1…4}/mnt/d{1...4} 2>&1 > minio.log &


Run Prometheus on one MinIO server using the following commands:



tar xvfz prometheus-*.tar.gz

cd prometheus-*

vi prometheus.yml


- job_name: minio-job

  metrics_path: /minio/v2/metrics/cluster

  scheme: http


  - targets: ['localhost:9000']

./prometheus –config.file=prometheus.yml


Get the MinIO status from the MinIO console by navigating to Monitoring -> Metrics -> Resources, as shown in the example below:

MinIO status
Figure 1: MinIO status

You can also use Linux tools to monitor MinIO runtime status, by running the following commands:

 pidstat -rp `pidof minio` 10     # Get CPU and memory usage.

iostat -dxctm 10 # Get disk status.

sar  -n DEV 10   # Get NIC status



Profile MinIO


Use the mc [10] tool, which was developed by MinIO, to profile MinIO.

  1. Install mc using the following commands: 
mv mc.RELEASE.2022-02-07T09-25-34Z mc
chmod +x mc
./mc alias set myminio minioadmin minioadmin


  1. Start CPU, memory, and block profiling concurrently:
./mc admin profile start --type cpu,mem,block myminio/


  1. Stop profiling:
./mc admin profile stop myminio


  1. View profiling result:
go tool pprof profile-\:9000-cpu.pprof

go tool pprof -png profile-\:9000-cpu.pprof > out.png


MinIO profiling result.
Figure 2: MinIO profiling result


This document introduces the methods and tools we used to benchmark, monitor, and analyze MinIO cluster performance.  We used "fio” to check single and concurrent disk I/O performance. “dperf" and "iozone" were used to test JBOD performance. Warp [4] is a S3 benchmark tool to benchmark MinIO performance. Prometheus and Linux commands such as pidstat, iostat, and sar were used to monitor CPU/RAM/disk/NIC usage. The “mc” [10] tool, which was developed by MinIO as command line management tool, was used to profile MinIO.


  6. crypto/rsa: Go 1.21 follow-up work #57752