Environment

1, EC2 and the S3 bucket are located in the same region

2, make sure fio is installed

// Redhat/CentOS/etc
$ sudo dnf install -y fio

// Ubuntu/Debian/etc
$ sudo apt install -y fio

CPU Usage is measured in percentage of cores, not overall CPU usage.

For example, on c6id.8xlarge, there are 32 vCPUs, so 1600% CPU usage corresponds to 16 cores fully utilized.

And 1600% CPU usage on c6id.8xlarge also corresponds to 1600%/32 = 50% of total CPU capacity.

Create a 100GB test file

fio --name=create_100gb_file \
    --filename=/mnt/fuse/100gb \
    --ioengine=libaio \
    --direct=1 \
    --group_reporting \
    --fallocate=none \
    --create_on_open=1 \
    --end_fsync=1 \
    --size=100000M \
    --rw=write \
    --bs=10M \
    --numjobs=1

Load the test file into local cache

$ lsblk

$ sudo mkfs.xfs /dev/nvme1n1

$ sudo mkdir /mnt/fuse /data

$ sudo chmod 0777 /mnt/fuse /data -R

$ sudo mount /dev/nvme1n1 /data

$ sudo mapfs add vol_benchmark aws <AWSAccessKey> <AWSSecretKey> <S3-BucketName> <Region> cache_dir=/data

$ sudo mapfs mount vol_benchmark /mnt/fuse

$ mapfs load /mnt/fuse/100gb

fio commands

fio: Read Performance with cache

fio: Network Write Performance

fio --name=write_benchmark \
    --directory=/mnt/fuse \
    --ioengine=libaio \
    --direct=1 \
    --group_reporting \
    --fallocate=none \
    --create_on_open=1 \
    --end_fsync=1 \
    --runtime=60 \
    --time_based \
    --size=100G \
    --rw=write \
    --bs=<1M|4M|8M> \
    --numjobs=<1|32|128>

Note:

After each fio test, delete fio generated test file(s) from the Cloud Storage:

$ rm -f /mnt/fuse/write_benchmark.*

EC2 c6id Seriess

With 1 NVME SSD instance storage.

c6id.large/xlarge/2xlarge/4xlarge: EBS and Network have a baseline performance, and burstable upper limit.

c6id Series: benchmark

c6id.large

Throughput (block size: 1MB)

IOPS (block size: 4KB)

Network Read

Network Write

c6id.xlarge

4 vCPU, 8GB RAM
On-Demand Linux pricing: 0.231 USD per Hour
Network: baseline 1.25Gbps (156MB/s), up to 12.5Gbps
NVME SSD: 1 x 237 GiB Instance Store

Throughput (block size: 1MB)

IOPS (block size: 4KB)

Network Read

Network Write

c6id.2xlarge

8 vCPU, 16GB RAM
On-Demand Linux pricing: 0.4620 USD per Hour
Network: Up to 12.5Gbps
NVME SSD: 1 x 474 GiB Instance Store

Throughput (block size: 1MB)

IOPS (block size: 4KB)

Network Read

Network Write

c6id.4xlarge

16 vCPU, 32GB RAM
On-Demand Linux pricing: 0.9240  USD per Hour
Network: Up to 12.5Gbps
NVME SSD: 1 x 950 GiB Instance Store

Throughput (block size: 1MB)

IOPS (block size: 4KB), mounted with iouring

$ sudo mapfs umount vol_benchmark

$ sudo mapfs mount vol_benchmark /mnt/fuse iouring

Network Read

Network Write

c6id.8xlarge

32 vCPU, 64GB RAM
On-Demand Linux pricing: 1.8480 USD per Hour
Network: 12.5Gbps
NVME SSD: 1 x 1900 GiB Instance Store

Throughput (block size: 1MB)

IOPS (block size: 4KB), mounted with iouring

$ sudo mapfs umount vol_benchmark

$ sudo mapfs mount vol_benchmark /mnt/fuse iouring

Network Read

Network Write

Benchmark Overview

mapfs only occupies a small amount of system memory, so doesn't illustrate in the charts.

Throughput

IOPS

Network Read

Network Write

fuse over iouring

kernel support

"mapfs mount <VolumeName> <MountPoint>" mounts in traditional mode by default;

To enable "fuse over iouring" mode, specify "iouring" during mount:

 $ sudo mapfs mount <VolumeName> <MountPoint> iouring

Note:

Specifying iouring does not guarantee "fuse over iouring" will be enabled. It also requires Linux kernel version >= 6.18, typically from these distributions:

• Amazon Linux 2023, kernel 6.18
• Ubuntu 26.04

Advantages and limitations of iouring mode

Traditional mount:
When CPU number exceeds 16 and IOPS exceeds 300K, a single lock contention point can cause CPU usage to spike while I/O performance may not improve or can even degrade greatly.
The more CPUs added, the worse the performance degradation becomes, due to scheduling pressure, frequent L3 cache invalidations, and the single kernel spinlock CPU usage.

iouring mode mount:
breaks single lock contention and scale up better when CPU number exceeds 16 and IOPS exceed 300K.
However, its limitation is that when IOPS exceed 1M and CPU number reaches 32 or more, fuse uring threads may degrade into polling mode, causing high CPU usage without a corresponding I/O performance increase.

Common benchmark bottlenecks

SSD Throughput and IOPS limitations

For example,

If you are testing the performance with a GP3 volume with default GP3 settings, which is 125 MiB/s throughput and 3000 IOPS,

you will see the Random Read performance with 1MB block size is limited to 125 MiB/s, and the Random Read performance with 4KB block size is limited to 3000 IOPS.

EC2 EBS Throughput and IOPS limitations

Just imagine you are testing the performance with a c6i.large EC2 and one GP3 SSD.

The baseline EBS performance of c6i.large is 3600 IOPS and 81.25 MiB/s throughput, and the GP3 SSD performance is 3000 IOPS and 125 MiB/s throughput.

Even if you raise the GP3 performance to 5000 IOPS and 500 MiB/s throughput, you will still be limited by the EBS performance after burst credits are exhausted (generally after 10 - 30 minutes), which is 3600 IOPS and 81.25 MiB/s throughput.

Network bandwidth

Even if you are testing the performance with a single fio job (numjobs=1), when the network bandwidth is saturated, you will see the Network Read/Write performance will not increase with more fio jobs added.

CPU Burstable, EBS Burstable, Network Burstable, NVME SSD Burstable, VPC Credits, etc.

For example,

if you use burstable EC2 instances (such as t3.small), CPU performance may fluctuate between baseline and burst limits.