1CheckPoint and Restart (CPR) 2============================ 3 4CPR is the umbrella name for a set of migration modes in which the 5VM is migrated to a new QEMU instance on the same host. It is 6intended for use when the goal is to update host software components 7that run the VM, such as QEMU or even the host kernel. At this time, 8the cpr-reboot and cpr-transfer modes are available. 9 10Because QEMU is restarted on the same host, with access to the same 11local devices, CPR is allowed in certain cases where normal migration 12would be blocked. However, the user must not modify the contents of 13guest block devices between quitting old QEMU and starting new QEMU. 14 15CPR unconditionally stops VM execution before memory is saved, and 16thus does not depend on any form of dirty page tracking. 17 18cpr-reboot mode 19--------------- 20 21In this mode, QEMU stops the VM, and writes VM state to the migration 22URI, which will typically be a file. After quitting QEMU, the user 23resumes by running QEMU with the ``-incoming`` option. Because the 24old and new QEMU instances are not active concurrently, the URI cannot 25be a type that streams data from one instance to the other. 26 27Guest RAM can be saved in place if backed by shared memory, or can be 28copied to a file. The former is more efficient and is therefore 29preferred. 30 31After state and memory are saved, the user may update userland host 32software before restarting QEMU and resuming the VM. Further, if 33the RAM is backed by persistent shared memory, such as a DAX device, 34then the user may reboot to a new host kernel before restarting QEMU. 35 36This mode supports VFIO devices provided the user first puts the 37guest in the suspended runstate, such as by issuing the 38``guest-suspend-ram`` command to the QEMU guest agent. The agent 39must be pre-installed in the guest, and the guest must support 40suspend to RAM. Beware that suspension can take a few seconds, so 41the user should poll to see the suspended state before proceeding 42with the CPR operation. 43 44Usage 45^^^^^ 46 47It is recommended that guest RAM be backed with some type of shared 48memory, such as ``memory-backend-file,share=on``, and that the 49``x-ignore-shared`` capability be set. This combination allows memory 50to be saved in place. Otherwise, after QEMU stops the VM, all guest 51RAM is copied to the migration URI. 52 53Outgoing: 54 * Set the migration mode parameter to ``cpr-reboot``. 55 * Set the ``x-ignore-shared`` capability if desired. 56 * Issue the ``migrate`` command. It is recommended the URI be a 57 ``file`` type, but one can use other types such as ``exec``, 58 provided the command captures all the data from the outgoing side, 59 and provides all the data to the incoming side. 60 * Quit when QEMU reaches the postmigrate state. 61 62Incoming: 63 * Start QEMU with the ``-incoming defer`` option. 64 * Set the migration mode parameter to ``cpr-reboot``. 65 * Set the ``x-ignore-shared`` capability if desired. 66 * Issue the ``migrate-incoming`` command. 67 * If the VM was running when the outgoing ``migrate`` command was 68 issued, then QEMU automatically resumes VM execution. 69 70Example 1 71^^^^^^^^^ 72:: 73 74 # qemu-kvm -monitor stdio 75 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G 76 ... 77 78 (qemu) info status 79 VM status: running 80 (qemu) migrate_set_parameter mode cpr-reboot 81 (qemu) migrate_set_capability x-ignore-shared on 82 (qemu) migrate -d file:vm.state 83 (qemu) info status 84 VM status: paused (postmigrate) 85 (qemu) quit 86 87 ### optionally update kernel and reboot 88 # systemctl kexec 89 kexec_core: Starting new kernel 90 ... 91 92 # qemu-kvm ... -incoming defer 93 (qemu) info status 94 VM status: paused (inmigrate) 95 (qemu) migrate_set_parameter mode cpr-reboot 96 (qemu) migrate_set_capability x-ignore-shared on 97 (qemu) migrate_incoming file:vm.state 98 (qemu) info status 99 VM status: running 100 101Example 2: VFIO 102^^^^^^^^^^^^^^^ 103:: 104 105 # qemu-kvm -monitor stdio 106 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G 107 -device vfio-pci, ... 108 -chardev socket,id=qga0,path=qga.sock,server=on,wait=off 109 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 110 ... 111 112 (qemu) info status 113 VM status: running 114 115 # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock 116 117 (qemu) info status 118 VM status: paused (suspended) 119 (qemu) migrate_set_parameter mode cpr-reboot 120 (qemu) migrate_set_capability x-ignore-shared on 121 (qemu) migrate -d file:vm.state 122 (qemu) info status 123 VM status: paused (postmigrate) 124 (qemu) quit 125 126 ### optionally update kernel and reboot 127 # systemctl kexec 128 kexec_core: Starting new kernel 129 ... 130 131 # qemu-kvm ... -incoming defer 132 (qemu) info status 133 VM status: paused (inmigrate) 134 (qemu) migrate_set_parameter mode cpr-reboot 135 (qemu) migrate_set_capability x-ignore-shared on 136 (qemu) migrate_incoming file:vm.state 137 (qemu) info status 138 VM status: paused (suspended) 139 (qemu) system_wakeup 140 (qemu) info status 141 VM status: running 142 143Caveats 144^^^^^^^ 145 146cpr-reboot mode may not be used with postcopy, background-snapshot, 147or COLO. 148 149cpr-transfer mode 150----------------- 151 152This mode allows the user to transfer a guest to a new QEMU instance 153on the same host with minimal guest pause time, by preserving guest 154RAM in place, albeit with new virtual addresses in new QEMU. Devices 155and their pinned memory pages will also be preserved in a future QEMU 156release. 157 158The user starts new QEMU on the same host as old QEMU, with command- 159line arguments to create the same machine, plus the ``-incoming`` 160option for the main migration channel, like normal live migration. 161In addition, the user adds a second -incoming option with channel 162type ``cpr``. This CPR channel must support file descriptor transfer 163with SCM_RIGHTS, i.e. it must be a UNIX domain socket. 164 165To initiate CPR, the user issues a migrate command to old QEMU, 166adding a second migration channel of type ``cpr`` in the channels 167argument. Old QEMU stops the VM, saves state to the migration 168channels, and enters the postmigrate state. Execution resumes in 169new QEMU. 170 171New QEMU reads the CPR channel before opening a monitor, hence 172the CPR channel cannot be specified in the list of channels for a 173migrate-incoming command. It may only be specified on the command 174line. 175 176Usage 177^^^^^ 178 179Memory backend objects must have the ``share=on`` attribute. 180 181The VM must be started with the ``-machine aux-ram-share=on`` 182option. This causes implicit RAM blocks (those not described by 183a memory-backend object) to be allocated by mmap'ing a memfd. 184Examples include VGA and ROM. 185 186Outgoing: 187 * Set the migration mode parameter to ``cpr-transfer``. 188 * Issue the ``migrate`` command, containing a main channel and 189 a cpr channel. 190 191Incoming: 192 * Start new QEMU with two ``-incoming`` options. 193 * If the VM was running when the outgoing ``migrate`` command was 194 issued, then QEMU automatically resumes VM execution. 195 196Caveats 197^^^^^^^ 198 199cpr-transfer mode may not be used with postcopy, background-snapshot, 200or COLO. 201 202memory-backend-epc is not supported. 203 204The main incoming migration channel address cannot be a file type. 205 206If the main incoming channel address is an inet socket, then the port 207cannot be 0 (meaning dynamically choose a port). 208 209When using ``-incoming defer``, you must issue the migrate command to 210old QEMU before issuing any monitor commands to new QEMU, because new 211QEMU blocks waiting to read from the cpr channel before starting its 212monitor, and old QEMU does not write to the channel until the migrate 213command is issued. However, new QEMU does not open and read the 214main migration channel until you issue the migrate incoming command. 215 216Example 1: incoming channel 217^^^^^^^^^^^^^^^^^^^^^^^^^^^ 218 219In these examples, we simply restart the same version of QEMU, but 220in a real scenario one would start new QEMU on the incoming side. 221Note that new QEMU does not print the monitor prompt until old QEMU 222has issued the migrate command. The outgoing side uses QMP because 223HMP cannot specify a CPR channel. Some QMP responses are omitted for 224brevity. 225 226:: 227 228 Outgoing: Incoming: 229 230 # qemu-kvm -qmp stdio 231 -object memory-backend-file,id=ram0,size=4G, 232 mem-path=/dev/shm/ram0,share=on -m 4G 233 -machine memory-backend=ram0 234 -machine aux-ram-share=on 235 ... 236 # qemu-kvm -monitor stdio 237 -incoming tcp:0:44444 238 -incoming '{"channel-type": "cpr", 239 "addr": { "transport": "socket", 240 "type": "unix", "path": "cpr.sock"}}' 241 ... 242 {"execute":"qmp_capabilities"} 243 244 {"execute": "query-status"} 245 {"return": {"status": "running", 246 "running": true}} 247 248 {"execute":"migrate-set-parameters", 249 "arguments":{"mode":"cpr-transfer"}} 250 251 {"execute": "migrate", "arguments": { "channels": [ 252 {"channel-type": "main", 253 "addr": { "transport": "socket", "type": "inet", 254 "host": "0", "port": "44444" }}, 255 {"channel-type": "cpr", 256 "addr": { "transport": "socket", "type": "unix", 257 "path": "cpr.sock" }}]}} 258 259 QEMU 10.0.50 monitor 260 (qemu) info status 261 VM status: running 262 263 {"execute": "query-status"} 264 {"return": {"status": "postmigrate", 265 "running": false}} 266 267Example 2: incoming defer 268^^^^^^^^^^^^^^^^^^^^^^^^^ 269 270This example uses ``-incoming defer`` to hot plug a device before 271accepting the main migration channel. Again note you must issue the 272migrate command to old QEMU before you can issue any monitor 273commands to new QEMU. 274 275 276:: 277 278 Outgoing: Incoming: 279 280 # qemu-kvm -monitor stdio 281 -object memory-backend-file,id=ram0,size=4G, 282 mem-path=/dev/shm/ram0,share=on -m 4G 283 -machine memory-backend=ram0 284 -machine aux-ram-share=on 285 ... 286 # qemu-kvm -monitor stdio 287 -incoming defer 288 -incoming '{"channel-type": "cpr", 289 "addr": { "transport": "socket", 290 "type": "unix", "path": "cpr.sock"}}' 291 ... 292 {"execute":"qmp_capabilities"} 293 294 {"execute": "device_add", 295 "arguments": {"driver": "pcie-root-port"}} 296 297 {"execute":"migrate-set-parameters", 298 "arguments":{"mode":"cpr-transfer"}} 299 300 {"execute": "migrate", "arguments": { "channels": [ 301 {"channel-type": "main", 302 "addr": { "transport": "socket", "type": "inet", 303 "host": "0", "port": "44444" }}, 304 {"channel-type": "cpr", 305 "addr": { "transport": "socket", "type": "unix", 306 "path": "cpr.sock" }}]}} 307 308 QEMU 10.0.50 monitor 309 (qemu) info status 310 VM status: paused (inmigrate) 311 (qemu) device_add pcie-root-port 312 (qemu) migrate_incoming tcp:0:44444 313 (qemu) info status 314 VM status: running 315 316 {"execute": "query-status"} 317 {"return": {"status": "postmigrate", 318 "running": false}} 319 320Futures 321^^^^^^^ 322 323cpr-transfer mode is based on a capability to transfer open file 324descriptors from old to new QEMU. In the future, descriptors for 325vfio, iommufd, vhost, and char devices could be transferred, 326preserving those devices and their kernel state without interruption, 327even if they do not explicitly support live migration. 328