1# How to test vhost-user-blk with SPDK 2 3The purpose of this document is to illustrate how to test vhost-user-blk in Cloud Hypervisor with SPDK as the backend. 4 5## Framework 6 7It's a simple test to validate the block read/write between VM and block backend. 8``` 9 +----+----------+ +-------------+-----------+ 10 | | | | | | 11 | |vhost-user|----------| vhost-user | dpdk | 12 | |blk device| | port 1 | | 13 | | | | | | 14 | +----------+ +-------------+-----------+ 15 | | | | 16 | vm | | spdk | 17 | | | | 18 +--+----------------------------------------------------+--+ 19 | | hugepages | | 20 | +----------------------------------------------------+ | 21 | | 22 | host | 23 | | 24 +----------------------------------------------------------+ 25``` 26## Prerequisites 27 28Prior to running the test, the following steps need to be performed. 29- Enable hugepages 30- Install SPDK 31 32Here are some good references for detailing them. 33- spdk 34 * https://spdk.io/doc/ 35 36## Test environment 37 38The below test environment is based on Ubuntu release (20.04.1 LTS), as for other system, please check related documents. 39This test runs with multiple queue (MQ) support enabled, using 4 queues defined for both SPDK and the virtual machine. 40Here are the detailed instructions. 41 42### The hugepages settings on the host 43Allocate enough persistent huge pages (e.g. 2GiB) on the host: 44 45```bash 46# Assume the default hugepage size is 2MiB 47echo 1024 | sudo tee /proc/sys/vm/nr_hugepages 48# Confirm the value of "Hugepagesize" and "HugePages_Total" 49cat /proc/meminfo | grep -i huge 50``` 51 52### Download the SPDK code 53```bash 54git clone https://github.com/spdk/spdk 55cd spdk 56git submodule update --init 57``` 58 59### Create the build dep 60 61```bash 62./scripts/pkgdep.sh 63``` 64 65### Build SPDK 66```bash 67./configure 68make 69``` 70 71### Set the SPDK environment 72 73Run SPDK vhost target with 2 CPU cores, i.e., core 0 and 1. 74```bash 75sudo HUGEMEM=2048 scripts/setup.sh 76sudo ./build/bin/vhost -S /var/tmp -s 1024 -m 0x3 & 77``` 78 79### Create 512M block device 80 81Create a 512M (first parameter) block device with 512 bytes (second parameter) block size. 82```bash 83sudo scripts/rpc.py bdev_malloc_create 512 512 -b Malloc0 84sudo scripts/rpc.py vhost_create_blk_controller --cpumask 0x1 vhost.1 Malloc0 85``` 86 87### Launch the VM 88 89VMs run in client mode. They connect to the socket created by the `dpdkvhostuser` in the SPDK backend. 90```bash 91# From the test terminal. We need to create one vhost-user-blk device for the --disk. 92./cloud-hypervisor \ 93 --cpus boot=4 \ 94 --memory size=1024M,hugepages=on,shared=true \ 95 --kernel linux/arch/x86/boot/compressed/vmlinux.bin \ 96 --cmdline "console=ttyS0 root=/dev/vda1 rw iommu=off" \ 97 --disk path=images/focal-server-cloudimg-amd64.raw vhost_user=true,socket=/var/tmp/vhost.1,num_queues=4,queue_size=128 \ 98 --console off \ 99 --serial tty \ 100 --rng 101``` 102 103```bash 104# How to test the vhost-user-blk device with SPDK backend 105login in guest 106 107# Use lsblk command to find out vhost-user-blk device 108lsblk 109NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT 110vda 252:0 0 2.2G 0 disk 111├─vda1 252:1 0 2.1G 0 part / 112├─vda14 252:14 0 4M 0 part 113└─vda15 252:15 0 106M 0 part /boot/efi 114vdb 253:16 0 512M 0 disk 115 116The vhost-user-blk device is /dev/vdb 117 118# How to do simple read/write test 119dd if=/dev/vdb of=/dev/null bs=2M iflag=direct 120dd of=/dev/vdb if=/dev/zero bs=2M oflag=direct count=256 121 122If you want to do fio test, please install fio binary into guest. The detailed info is not listed here. 123