1# Live Migration 2 3This document gives two examples of how to use the live migration 4support in Cloud Hypervisor: 5 61. local migration - migrating a VM from one Cloud Hypervisor instance to another on the same machine; 71. remote migration - migrating a VM between two machines; 8 9> :warning: These examples place sockets /tmp. This is done for 10> simplicity and should not be done in production. 11 12## Local Migration (Suitable for Live Upgrade of VMM) 13 14Launch the source VM (on the host machine): 15 16```console 17$ target/release/cloud-hypervisor 18 --kernel ~/workloads/vmlinux \ 19 --disk path=~/workloads/focal.raw \ 20 --cpus boot=1 --memory size=1G,shared=on \ 21 --cmdline "root=/dev/vda1 console=ttyS0" \ 22 --serial tty --console off --api-socket=/tmp/api1 23``` 24 25Launch the destination VM from the same directory (on the host machine): 26 27```console 28$ target/release/cloud-hypervisor --api-socket=/tmp/api2 29``` 30 31Get ready for receiving migration for the destination VM (on the host machine): 32 33```console 34$ target/release/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/sock 35``` 36 37Start to send migration for the source VM (on the host machine): 38 39```console 40$ target/release/ch-remote --api-socket=/tmp/api1 send-migration --local unix:/tmp/sock 41``` 42 43When the above commands completed, the source VM should be successfully 44migrated to the destination VM. Now the destination VM is running while 45the source VM is terminated gracefully. 46 47## Remote Migration 48 49In this example, we will migrate a VM from one machine (`src`) to 50another (`dst`) across the network. To keep it simple, we will use a 51minimal VM setup without storage. 52 53Because Cloud Hypervisor does not natively support migrating via TCP 54connections, we will tunnel traffic through `socat`. 55 56### Preparation 57 58Make sure that `src` and `dst` can reach each other via the 59network. You should be able to ping each machine. Also each machine 60should have an open TCP port. For this example we assume port 6000. 61 62You will need a kernel and initramfs for a minimal Linux system. For 63this example, we will use the Debian netboot image. 64 65Place the kernel and initramfs into the _same directory_ on both 66machines. This is important for the migration to succeed. We will use 67`/var/images`: 68 69```console 70src $ export DEBIAN=https://ftp.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64 71src $ mkdir -p /var/images 72src $ curl $DEBIAN/linux > /var/images/linux 73src $ curl $DEBIAN/initrd.gz > /var/images/initrd 74``` 75 76Repeat the above steps on the destination host. 77 78### Starting the Receiver VM 79 80On the receiver side, we prepare an empty VM: 81 82```console 83dst $ cloud-hypervisor --api-socket /tmp/api 84``` 85 86In a different terminal, configure the VM as a migration target: 87 88```console 89dst $ ch-remote --api-socket=/tmp/api receive-migration unix:/tmp/sock 90``` 91 92In yet another terminal, forward TCP connections to the Unix domain socket: 93 94```console 95dst $ socat TCP-LISTEN:6000,reuseaddr UNIX-CLIENT:/tmp/sock 96``` 97 98### Starting the Sender VM 99 100Let's start the VM on the source machine: 101 102```console 103src $ cloud-hypervisor \ 104 --serial tty --console off \ 105 --cpus boot=2 --memory size=4G \ 106 --kernel /var/images/linux \ 107 --initramfs /var/images/initrd \ 108 --cmdline "console=ttyS0" \ 109 --api-socket /tmp/api 110``` 111 112After a few seconds the VM should be up and you can interact with it. 113 114### Performing the Migration 115 116First, we start `socat`: 117 118```console 119src $ socat UNIX-LISTEN:/tmp/sock,reuseaddr TCP:dst:6000 120``` 121 122Then we kick-off the migration itself: 123 124```console 125src $ ch-remote --api-socket=/tmp/api send-migration unix:/tmp/sock 126``` 127 128When the above commands completed, the VM should be successfully 129migrated to the destination machine without interrupting the workload. 130