xref: /cloud-hypervisor/docs/vhost-user-blk-testing.md (revision 9af2968a7dc47b89bf07ea9dc5e735084efcfa3a) 
1# How to test vhost-user-blk with SPDK
2
3The purpose of this document is to illustrate how to test vhost-user-blk in Cloud Hypervisor with SPDK as the backend.
4
5## Framework
6
7It's a simple test to validate the block read/write between VM and block backend.
8```
9             +----+----------+          +-------------+-----------+
10             |    |          |          |             |           |
11             |    |vhost-user|----------| vhost-user  |    dpdk   |
12             |    |blk device|          | port 1      |           |
13             |    |          |          |             |           |
14             |    +----------+          +-------------+-----------+
15             |               |          |                         |
16             |      vm       |          |           spdk          |
17             |               |          |                         |
18          +--+----------------------------------------------------+--+
19          |  |                  hugepages                         |  |
20          |  +----------------------------------------------------+  |
21          |                                                          |
22          |                       host                               |
23          |                                                          |
24          +----------------------------------------------------------+
25```
26## Prerequisites
27
28Prior to running the test, the following steps need to be performed.
29- Enable hugepages
30- Install SPDK
31
32Here are some good references for detailing them.
33- spdk
34	* https://spdk.io/doc/
35
36## Test environment
37
38The below test environment is based on Ubuntu release (20.04.1 LTS), as for other system, please check related documents.
39This test runs with multiple queue (MQ) support enabled, using 4 queues defined for both SPDK and the virtual machine.
40Here are the detailed instructions.
41
42### The hugepages settings on the host
43Allocate enough persistent huge pages (e.g. 2GiB) on the host:
44
45```bash
46# Assume the default hugepage size is 2MiB
47echo 1024 | sudo tee /proc/sys/vm/nr_hugepages
48# Confirm the value of "Hugepagesize" and "HugePages_Total"
49cat /proc/meminfo | grep -i huge
50```
51
52### Download the SPDK code
53```bash
54git clone https://github.com/spdk/spdk
55cd spdk
56git submodule update --init
57```
58
59### Create the build dep
60
61```bash
62./scripts/pkgdep.sh
63```
64
65### Build SPDK
66```bash
67./configure
68make
69```
70
71### Set the SPDK environment
72```bash
73sudo HUGEMEM=2048 scripts/setup.sh
74sudo ./build/bin/vhost -S /var/tmp -s 1024 -m 0x3 &
75```
76
77### Create 512M block device
78```bash
79sudo scripts/rpc.py bdev_malloc_create 512 512 -b Malloc0
80sudo scripts/rpc.py vhost_create_blk_controller --cpumask 0x1 vhost.1 Malloc0
81```
82
83### Launch the VM
84
85VMs run in client mode. They connect to the socket created by the `dpdkvhostuser` in the SPDK backend.
86```bash
87# From the test terminal. We need to create one vhost-user-blk device for the --disk.
88./cloud-hypervisor \
89        --cpus boot=4 \
90        --memory size=1024M,hugepages=on,shared=true \
91        --kernel linux/arch/x86/boot/compressed/vmlinux.bin \
92        --cmdline "console=ttyS0 root=/dev/vda1 rw iommu=off" \
93        --disk path=images/focal-server-cloudimg-amd64.raw vhost_user=true,socket=/var/tmp/vhost.1,num_queues=4,queue_size=128 \
94        --console off \
95        --serial tty \
96        --rng
97```
98
99```bash
100# How to test the vhost-user-blk device with SPDK backend
101login in guest
102
103# Use lsblk command to find out vhost-user-blk device
104lsblk
105NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
106vda     252:0    0  2.2G  0 disk
107├─vda1  252:1    0  2.1G  0 part /
108├─vda14 252:14   0    4M  0 part
109└─vda15 252:15   0  106M  0 part /boot/efi
110vdb    253:16   0  512M  0 disk
111
112The vhost-user-blk device is /dev/vdb
113
114# How to do simple read/write test
115dd if=/dev/vdb of=/dev/null bs=2M iflag=direct
116dd of=/dev/vdb if=/dev/zero bs=2M oflag=direct count=256
117
118If you want to do fio test, please install fio binary into guest. The detailed info is not listed here.
119