1# How to test Vhost-user net with OpenVSwitch/DPDK 2 3The purpose of this document is to illustrate how to test vhost-user-net 4in cloud-hypervisor with OVS/DPDK as the backend. This document was 5tested with Open vSwitch v2.13.1, DPDK v19.11.3, and Cloud Hypervisor 6v15.0 on Ubuntu 20.04.1 (host kernel v5.4.0). 7 8## Framework 9 10It's a simple test to validate the communication between two virtual machine, connecting them to vhost-user ports respectively provided by `OVS/DPDK`. 11``` 12 +----+----------+ +-------------+-----------+-------------+ +----------+----+ 13 | | | | | | | | | | 14 | |vhost-user|----------| vhost-user | ovs | vhost-user |----------|vhost-user| | 15 | |net device| | port 1 | | port 2 | |net device| | 16 | | | | | | | | | | 17 | +----------+ +-------------+-----------+-------------+ +----------+ | 18 | | | | | | 19 |vm1 | | dpdk | | vm2 | 20 | | | | | | 21 +--+---------------------------------------------------------------------------------------------+--+ 22 | | hugepages | | 23 | +---------------------------------------------------------------------------------------------+ | 24 | | 25 | host | 26 | | 27 +---------------------------------------------------------------------------------------------------+ 28``` 29## Prerequisites 30 31Prior to running the test, the following steps need to be performed. 32- Enable hugepages 33- Install DPDK 34- Install OVS 35 36Here is a good reference for setting up OVS with DPDK from scratch: 37https://docs.openvswitch.org/en/latest/intro/install/dpdk/. 38 39On Ubuntu systems (18.04 or newer), the OpenVswitch-DPDK package can be 40easily installed with: 41```bash 42sudo apt-get update 43sudo apt-get install openvswitch-switch-dpdk 44sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk 45``` 46## Test 47The test runs with multiple queue (MQ) support enabled, using 2 pairs of 48TX/RX queues defined for both OVS and the virtual machine. Here are the 49detailed instructions. 50 51_Setup OVS_ 52 53Here is an example how to configure a basic OpenVswitch using DPDK: 54```bash 55# load the ovs kernel module 56modprobe openvswitch 57sudo service openvswitch-switch start 58ovs-vsctl init 59ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true 60# run on core 0-3 only 61ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0xf 62# allocate 2G huge pages (to NUMA 0 only) 63ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=1024 64# run PMD (Pull Mode Driver) threads on core 0-3 only 65ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xf 66sudo service openvswitch-switch restart 67# double check the configurations 68ovs-vsctl list Open_vSwitch 69``` 70 71Here is an example how to create a bridge and add two DPDK ports to it 72(for later use via Cloud Hypervisor): 73```bash 74# create a bridge 75ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev 76# create two DPDK ports and add them to the bridge 77ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser 78ovs-vsctl add-port ovsbr0 vhost-user2 -- set Interface vhost-user2 79type=dpdkvhostuser 80# set the number of rx queues 81ovs-vsctl set Interface vhost-user1 options:n_rxq=2 82ovs-vsctl set Interface vhost-user2 options:n_rxq=2 83``` 84 85_Launch the VMs_ 86 87VMs run in client mode. They connect to the socket created by the `dpdkvhostuser` backend. 88```bash 89# From one terminal. We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host. 90./cloud-hypervisor \ 91 --cpus boot=2 \ 92 --memory size=512M,hugepages=on,shared=true \ 93 --kernel vmlinux \ 94 --cmdline "console=ttyS0 console=hvc0 root=/dev/vda1 rw" \ 95 --disk path=focal-server-cloudimg-amd64.raw \ 96 --net mac=52:54:00:02:d9:01,vhost_user=true,socket=/var/run/openvswitch/vhost-user1,num_queues=4 97 98# From another terminal. We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host. 99./cloud-hypervisor \ 100 --cpus boot=2 \ 101 --memory size=512M,hugepages=on,shared=true \ 102 --kernel vmlinux \ 103 --cmdline "console=ttyS0 console=hvc0 root=/dev/vda1 rw" \ 104 --disk path=focal-server-cloudimg-amd64.raw \ 105 --net mac=52:54:20:11:C5:02,vhost_user=true,socket=/var/run/openvswitch/vhost-user2,num_queues=4 106``` 107 108_Setup VM1_ 109```bash 110# From inside the guest 111sudo ip addr add 172.100.0.1/24 dev ens2 112sudo ip link set up dev ens2 113``` 114 115_Setup VM2_ 116```bash 117# From inside the guest 118sudo ip addr add 172.100.0.2/24 dev ens2 119sudo ip link set up dev ens2 120``` 121 122_Ping VM1 from VM2_ 123```bash 124# From inside the guest 125sudo ping 172.100.0.1 126``` 127 128_Ping VM2 from VM1_ 129```bash 130# From inside the guest 131sudo ping 172.100.0.2 132``` 133 134__Result:__ At this point, VM1 and VM2 can ping each other successfully. We can now run `iperf3` test. 135 136_Run VM1 as server_ 137```bash 138# From inside the guest 139iperf3 -s -p 4444 140``` 141 142_Run VM2 as client_ 143```bash 144# From inside the guest 145iperf3 -c 172.100.0.1 -t 30 -p 4444 & 146``` 147