1# Live Migration 2 3This document gives examples of how to use the live migration support 4in Cloud Hypervisor: 5 61. local migration - migrating a VM from one Cloud Hypervisor instance to another on the same machine; 71. remote migration - migrating a VM between two machines; 8 9> :warning: These examples place sockets /tmp. This is done for 10> simplicity and should not be done in production. 11 12## Local Migration (Suitable for Live Upgrade of VMM) 13 14Launch the source VM (on the host machine): 15 16```console 17$ target/release/cloud-hypervisor 18 --kernel ~/workloads/vmlinux \ 19 --disk path=~/workloads/focal.raw \ 20 --cpus boot=1 --memory size=1G,shared=on \ 21 --cmdline "root=/dev/vda1 console=ttyS0" \ 22 --serial tty --console off --api-socket=/tmp/api1 23``` 24 25Launch the destination VM from the same directory (on the host machine): 26 27```console 28$ target/release/cloud-hypervisor --api-socket=/tmp/api2 29``` 30 31Get ready for receiving migration for the destination VM (on the host machine): 32 33```console 34$ target/release/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/sock 35``` 36 37Start to send migration for the source VM (on the host machine): 38 39```console 40$ target/release/ch-remote --api-socket=/tmp/api1 send-migration --local unix:/tmp/sock 41``` 42 43When the above commands completed, the source VM should be successfully 44migrated to the destination VM. Now the destination VM is running while 45the source VM is terminated gracefully. 46 47## Remote Migration 48 49In this example, we will migrate a VM from one machine (`src`) to 50another (`dst`) across the network. To keep it simple, we will use a 51minimal VM setup without storage. 52 53### Preparation 54 55Make sure that `src` and `dst` can reach each other via the 56network. You should be able to ping each machine. Also each machine 57should have an open TCP port. 58 59You will need a kernel and initramfs for a minimal Linux system. For 60this example, we will use the Debian netboot image. 61 62Place the kernel and initramfs into the _same directory_ on both 63machines. This is important for the migration to succeed. We will use 64`/var/images`: 65 66```console 67src $ export DEBIAN=https://ftp.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64 68src $ mkdir -p /var/images 69src $ curl $DEBIAN/linux > /var/images/linux 70src $ curl $DEBIAN/initrd.gz > /var/images/initrd 71``` 72 73Repeat the above steps on the destination host. 74 75### Unix Socket Migration 76 77If Unix socket is selected for migration, we can tunnel traffic through "socat". 78 79#### Starting the Receiver VM 80 81On the receiver side, we prepare an empty VM: 82 83```console 84dst $ cloud-hypervisor --api-socket /tmp/api 85``` 86 87In a different terminal, configure the VM as a migration target: 88 89```console 90dst $ ch-remote --api-socket=/tmp/api receive-migration unix:/tmp/sock 91``` 92 93In yet another terminal, forward TCP connections to the Unix domain socket: 94 95```console 96dst $ socat TCP-LISTEN:{port},reuseaddr UNIX-CLIENT:/tmp/sock 97``` 98 99#### Starting the Sender VM 100 101Let's start the VM on the source machine: 102 103```console 104src $ cloud-hypervisor \ 105 --serial tty --console off \ 106 --cpus boot=2 --memory size=4G \ 107 --kernel /var/images/linux \ 108 --initramfs /var/images/initrd \ 109 --cmdline "console=ttyS0" \ 110 --api-socket /tmp/api 111``` 112 113After a few seconds the VM should be up and you can interact with it. 114 115#### Performing the Migration 116 117First, we start `socat`: 118 119```console 120src $ socat UNIX-LISTEN:/tmp/sock,reuseaddr TCP:{dst}:{port} 121``` 122 123> Replace {dst}:{port} with the actual IP address and port of your destination host. 124 125Then we kick-off the migration itself: 126 127```console 128src $ ch-remote --api-socket=/tmp/api send-migration unix:/tmp/sock 129``` 130 131When the above commands completed, the VM should be successfully 132migrated to the destination machine without interrupting the workload. 133 134### TCP Socket Migration 135 136If TCP socket is selected for migration, we need to consider migrating 137in a trusted network. 138 139#### Starting the Receiver VM 140 141On the receiver side, we prepare an empty VM: 142 143```console 144dst $ cloud-hypervisor --api-socket /tmp/api 145``` 146 147In a different terminal, prepare to receive the migration: 148 149```console 150dst $ ch-remote --api-socket=/tmp/api receive-migration tcp:0.0.0.0:{port} 151``` 152 153#### Starting the Sender VM 154 155Let's start the VM on the source machine: 156 157```console 158src $ cloud-hypervisor \ 159 --serial tty --console off \ 160 --cpus boot=2 --memory size=4G \ 161 --kernel /var/images/linux \ 162 --initramfs /var/images/initrd \ 163 --cmdline "console=ttyS0" \ 164 --api-socket /tmp/api 165``` 166 167After a few seconds the VM should be up and you can interact with it. 168 169#### Performing the Migration 170 171Initiate the Migration over TCP: 172 173```console 174src $ ch-remote --api-socket=/tmp/api send-migration tcp:{dst}:{port} 175``` 176 177> Replace {dst}:{port} with the actual IP address and port of your destination host. 178 179After completing the above commands, the source VM will be migrated to 180the destination host and continue running there. The source VM instance 181will terminate normally. All ongoing processes and connections within 182the VM should remain intact after the migration. 183