xref: /cloud-hypervisor/docs/vhost-user-blk-testing.md (revision fa22cb0be515cecb5c510a69cc263d903f346129) 
1# How to test vhost-user-blk with SPDK
2
3The purpose of this document is to illustrate how to test vhost-user-blk in Cloud Hypervisor with SPDK as the backend.
4
5## Framework
6
7It's a simple test to validate the block read/write between VM and block backend.
8```
9             +----+----------+          +-------------+-----------+
10             |    |          |          |             |           |
11             |    |vhost-user|----------| vhost-user  |    dpdk   |
12             |    |blk device|          | port 1      |           |
13             |    |          |          |             |           |
14             |    +----------+          +-------------+-----------+
15             |               |          |                         |
16             |      vm       |          |           spdk          |
17             |               |          |                         |
18          +--+----------------------------------------------------+--+
19          |  |                  hugepages                         |  |
20          |  +----------------------------------------------------+  |
21          |                                                          |
22          |                       host                               |
23          |                                                          |
24          +----------------------------------------------------------+
25```
26## Prerequisites
27
28Prior to running the test, the following steps need to be performed.
29- Enable hugepages
30- Install SPDK
31
32Here are some good references for detailing them.
33- spdk
34	* https://spdk.io/doc/
35
36## Test environment
37
38The below test environment is based on Ubuntu release (20.04.1 LTS), as for other system, please check related documents.
39This test runs with multiple queue (MQ) support enabled, using 4 queues defined for both SPDK and the virtual machine.
40Here are the detailed instructions.
41
42### The hugepages settings on the host
43Allocate enough persistent huge pages (e.g. 2GiB) on the host:
44
45```bash
46# Assume the default hugepage size is 2MiB
47echo 1024 | sudo tee /proc/sys/vm/nr_hugepages
48# Confirm the value of "Hugepagesize" and "HugePages_Total"
49cat /proc/meminfo | grep -i huge
50```
51
52### Download the SPDK code
53```bash
54git clone https://github.com/spdk/spdk
55cd spdk
56git submodule update --init
57```
58
59### Create the build dep
60
61```bash
62./scripts/pkgdep.sh
63```
64
65### Build SPDK
66```bash
67./configure
68make
69```
70
71### Set the SPDK environment
72
73Run SPDK vhost target with 2 CPU cores, i.e., core 0 and 1.
74```bash
75sudo HUGEMEM=2048 scripts/setup.sh
76sudo ./build/bin/vhost -S /var/tmp -s 1024 -m 0x3 &
77```
78
79### Create 512M block device
80
81Create a 512M (first parameter) block device with 512 bytes (second parameter) block size.
82```bash
83sudo scripts/rpc.py bdev_malloc_create 512 512 -b Malloc0
84sudo scripts/rpc.py vhost_create_blk_controller --cpumask 0x1 vhost.1 Malloc0
85```
86
87### Launch the VM
88
89VMs run in client mode. They connect to the socket created by the `dpdkvhostuser` in the SPDK backend.
90```bash
91# From the test terminal. We need to create one vhost-user-blk device for the --disk.
92./cloud-hypervisor \
93        --cpus boot=4 \
94        --memory size=1024M,hugepages=on,shared=true \
95        --kernel linux/arch/x86/boot/compressed/vmlinux.bin \
96        --cmdline "console=ttyS0 root=/dev/vda1 rw iommu=off" \
97        --disk path=images/focal-server-cloudimg-amd64.raw vhost_user=true,socket=/var/tmp/vhost.1,num_queues=4,queue_size=128 \
98        --console off \
99        --serial tty \
100        --rng
101```
102
103```bash
104# How to test the vhost-user-blk device with SPDK backend
105login in guest
106
107# Use lsblk command to find out vhost-user-blk device
108lsblk
109NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
110vda     252:0    0  2.2G  0 disk
111├─vda1  252:1    0  2.1G  0 part /
112├─vda14 252:14   0    4M  0 part
113└─vda15 252:15   0  106M  0 part /boot/efi
114vdb    253:16   0  512M  0 disk
115
116The vhost-user-blk device is /dev/vdb
117
118# How to do simple read/write test
119dd if=/dev/vdb of=/dev/null bs=2M iflag=direct
120dd of=/dev/vdb if=/dev/zero bs=2M oflag=direct count=256
121
122If you want to do fio test, please install fio binary into guest. The detailed info is not listed here.
123