1# How to test Vhost-user net with OpenVSwitch/DPDK 2 3The purpose of this document is to illustrate how to test vhost-user-net 4in cloud-hypervisor with OVS/DPDK as the backend. This document was 5tested with Open vSwitch v2.17.8, DPDK v21.11.4, and Cloud Hypervisor 6v37.0 on Ubuntu 22.04.3 (host kernel v5.15.0). 7 8## Framework 9 10It's a simple test to validate the communication between two virtual machine, connecting them to vhost-user ports respectively provided by `OVS/DPDK`. 11``` 12 +----+----------+ +-------------+-----------+-------------+ +----------+----+ 13 | | | | | | | | | | 14 | |vhost-user|----------| vhost-user | ovs | vhost-user |----------|vhost-user| | 15 | |net device| | port 1 | | port 2 | |net device| | 16 | | | | | | | | | | 17 | +----------+ +-------------+-----------+-------------+ +----------+ | 18 | | | | | | 19 |vm1 | | dpdk | | vm2 | 20 | | | | | | 21 +--+---------------------------------------------------------------------------------------------+--+ 22 | | hugepages | | 23 | +---------------------------------------------------------------------------------------------+ | 24 | | 25 | host | 26 | | 27 +---------------------------------------------------------------------------------------------------+ 28``` 29## Prerequisites 30 31Prior to running the test, the following steps need to be performed. 32- Enable hugepages 33- Install DPDK 34- Install OVS 35 36Here is a good reference for setting up OVS with DPDK from scratch: 37https://docs.openvswitch.org/en/latest/intro/install/dpdk/. 38 39On Ubuntu systems (18.04 or newer), the OpenVswitch-DPDK package can be 40easily installed with: 41```bash 42sudo apt-get update 43sudo apt-get install openvswitch-switch-dpdk 44sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk 45``` 46## Test 47The test runs with multiple queue (MQ) support enabled, using 2 pairs of 48TX/RX queues defined for both OVS and the virtual machine. Here are the 49detailed instructions. 50 51_Setup OVS_ 52 53Here is an example how to configure a basic OpenVswitch using DPDK: 54```bash 55# load the ovs kernel module 56modprobe openvswitch 57sudo service openvswitch-switch start 58ovs-vsctl init 59ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true 60# run on core 0-3 only 61ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0xf 62# allocate 2G huge pages (to NUMA 0 only) 63ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=1024 64# run PMD (Pull Mode Driver) threads on core 0-3 only 65ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xf 66sudo service openvswitch-switch restart 67# double check the configurations 68ovs-vsctl list Open_vSwitch 69``` 70 71Here is an example how to create a bridge and add two DPDK ports to it 72(for later use via Cloud Hypervisor): 73```bash 74# create a bridge 75ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev 76# create two DPDK ports and add them to the bridge 77ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost-user1 78ovs-vsctl add-port ovsbr0 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost-user2 79# set the number of rx queues 80ovs-vsctl set Interface vhost-user1 options:n_rxq=2 81ovs-vsctl set Interface vhost-user2 options:n_rxq=2 82``` 83 84_Launch the VMs_ 85 86VMs run in client mode. They connect to the socket created by the `dpdkvhostuser` backend. 87```bash 88# From one terminal. We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host. 89./cloud-hypervisor \ 90 --cpus boot=2 \ 91 --memory size=512M,hugepages=on,shared=true \ 92 --kernel vmlinux \ 93 --cmdline "console=ttyS0 console=hvc0 root=/dev/vda1 rw" \ 94 --disk path=focal-server-cloudimg-amd64.raw \ 95 --net mac=52:54:00:02:d9:01,vhost_user=true,socket=/tmp/vhost-user1,num_queues=4,vhost_mode=server 96 97# From another terminal. We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host. 98./cloud-hypervisor \ 99 --cpus boot=2 \ 100 --memory size=512M,hugepages=on,shared=true \ 101 --kernel vmlinux \ 102 --cmdline "console=ttyS0 console=hvc0 root=/dev/vda1 rw" \ 103 --disk path=focal-server-cloudimg-amd64.raw \ 104 --net mac=52:54:20:11:C5:02,vhost_user=true,socket=/tmp/vhost-user2,num_queues=4,vhost_mode=server 105``` 106 107_Setup VM1_ 108```bash 109# From inside the guest 110sudo ip addr add 172.100.0.1/24 dev ens3 111sudo ip link set up dev ens3 112``` 113 114_Setup VM2_ 115```bash 116# From inside the guest 117sudo ip addr add 172.100.0.2/24 dev ens3 118sudo ip link set up dev ens3 119``` 120 121_Ping VM1 from VM2_ 122```bash 123# From inside the guest 124sudo ping 172.100.0.1 125``` 126 127_Ping VM2 from VM1_ 128```bash 129# From inside the guest 130sudo ping 172.100.0.2 131``` 132 133__Result:__ At this point, VM1 and VM2 can ping each other successfully. We can now run `iperf3` test. 134 135_Run VM1 as server_ 136```bash 137# From inside the guest 138iperf3 -s -p 4444 139``` 140 141_Run VM2 as client_ 142```bash 143# From inside the guest 144iperf3 -c 172.100.0.1 -t 30 -p 4444 & 145``` 146