1# Snapshot and Restore 2 3The goal for the snapshot/restore feature is to provide the user with the 4ability to take a snapshot of a previously paused virtual machine. This 5snapshot can be used as the base for creating new identical virtual machines, 6without the need to boot them from scratch. The restore codepath takes the 7snapshot and creates the exact same virtual machine, restoring the previously 8saved states. The new virtual machine is restored in a paused state, as it was 9before the snapshot was performed. 10 11This feature is important for the project as it establishes the first step 12towards the support for live migration. 13 14## Snapshot a Cloud-Hypervisor VM 15 16First thing, we must run a Cloud-Hypervisor VM: 17 18```bash 19./cloud-hypervisor \ 20 --api-socket /tmp/cloud-hypervisor.sock \ 21 --cpus boot=4 \ 22 --memory size=4G \ 23 --kernel vmlinux \ 24 --cmdline "root=/dev/vda1 console=hvc0 rw" \ 25 --disk path=focal-server-cloudimg-amd64.raw 26``` 27 28At any point in time when the VM is running, one might choose to pause it: 29 30```bash 31./ch-remote --api-socket=/tmp/cloud-hypervisor.sock pause 32``` 33 34Once paused, the VM can be safely snapshot into the specified directory and 35using the following command: 36 37```bash 38./ch-remote --api-socket=/tmp/cloud-hypervisor.sock snapshot file:///home/foo/snapshot 39``` 40 41Given the directory was present on the system, the snapshot will succeed and 42it should contain the following files: 43 44```bash 45ll /home/foo/snapshot/ 46total 4194536 47drwxrwxr-x 2 foo bar 4096 Jul 22 11:50 ./ 48drwxr-xr-x 47 foo bar 4096 Jul 22 11:47 ../ 49-rw------- 1 foo bar 3221225472 Jul 22 11:19 memory-region-0 50-rw------- 1 foo bar 1073741824 Jul 22 11:19 memory-region-1 51-rw------- 1 foo bar 217853 Jul 22 11:19 vm.json 52``` 53 54In this particular example, we can observe that 2 memory region files were 55created. That is explained by the size of the guest RAM, which is 4GiB in this 56case. Because it exceeds 3GiB (which is where we can find a ~1GiB memory hole), 57Cloud-Hypervisor needs 2 distinct memory regions to be created. Each memory 58region's content is stored through a dedicated file, which explains why we end 59up with 2 different files, the first one containing the guest RAM range 0-3GiB 60and the second one containing the guest RAM range 3-4GiB. 61 62`vm.json` gathers all information related to the virtual machine configuration 63and state. The configuration bits are used to create a similar virtual machine 64with the correct amount of CPUs, RAM, and other expected devices. The state 65bits are used to restore each component in the state it was left before the 66snapshot occurred. 67 68## Restore a Cloud-Hypervisor VM 69 70Given that one has access to an existing snapshot in `/home/foo/snapshot`, 71it is possible to create a new VM based on this snapshot with the following 72command: 73 74```bash 75./cloud-hypervisor \ 76 --api-socket /tmp/cloud-hypervisor.sock \ 77 --restore source_url=file:///home/foo/snapshot 78``` 79 80Or using two different commands from two terminals: 81 82```bash 83# First terminal 84./cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 85 86# Second terminal 87./ch-remote --api-socket=/tmp/cloud-hypervisor.sock restore source_url=file:///home/foo/snapshot 88``` 89 90Remember the VM is restored in a `paused` state, which was the VM's state when 91it was snapshot. For this reason, one must explicitly `resume` the VM before to 92start using it. 93 94```bash 95./ch-remote --api-socket=/tmp/cloud-hypervisor.sock resume 96``` 97 98At this point, the VM is fully restored and is identical to the VM which was 99snapshot earlier. 100 101## Limitations 102 103The support of snapshot/restore feature is still experimental, meaning one 104might still find some bugs associated with it. 105 106Additionally, some devices and features don't support to be snapshot and 107restored yet: 108- `vhost-user` devices 109- `virtio-mem` 110- Intel SGX 111 112VFIO devices are out of scope. 113