1- [1. What is Cloud Hypervisor?](#1-what-is-cloud-hypervisor) 2 - [Objectives](#objectives) 3 - [High Level](#high-level) 4 - [Architectures](#architectures) 5 - [Guest OS](#guest-os) 6- [2. Getting Started](#2-getting-started) 7 - [Host OS](#host-os) 8 - [Preparation](#preparation) 9 - [Install prerequisites](#install-prerequisites) 10 - [Clone and build](#clone-and-build) 11 - [Containerized builds and tests](#containerized-builds-and-tests) 12 - [Use Prebuilt Binaries](#use-prebuilt-binaries) 13 - [Run](#run) 14 - [Cloud image](#cloud-image) 15 - [Custom kernel and disk image](#custom-kernel-and-disk-image) 16 - [Building your kernel](#building-your-kernel) 17 - [Disk image](#disk-image) 18 - [Booting the guest VM](#booting-the-guest-vm) 19- [3. Status](#3-status) 20 - [Hot Plug](#hot-plug) 21 - [Device Model](#device-model) 22 - [TODO](#todo) 23 - [Roadmap](#roadmap) 24- [4. `rust-vmm` project dependency](#4-rust-vmm-project-dependency) 25 - [Firecracker and crosvm](#firecracker-and-crosvm) 26- [5. Community](#5-community) 27 - [Contribute](#contribute) 28 - [Join us](#join-us) 29 - [Security issues](#security-issues) 30 31# 1. What is Cloud Hypervisor? 32 33Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on 34top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt) 35hypervisor and Microsoft Hypervisor (MSHV). 36 37The project focuses on exclusively running modern, cloud workloads, on top of 38a limited set of hardware architectures and platforms. Cloud workloads refers 39to those that are usually run by customers inside a cloud provider. For our 40purposes this means modern operating systems with most I/O handled by 41paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 4264-bit CPUs. 43 44Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is 45based on the [rust-vmm](https://github.com/rust-vmm) crates. 46 47## Objectives 48 49### High Level 50 51- Runs on KVM or MSHV 52- Minimal emulation 53- Low latency 54- Low memory footprint 55- Low complexity 56- High performance 57- Small attack surface 58- 64-bit support only 59- CPU, memory, PCI hotplug 60- Machine to machine migration 61 62### Architectures 63 64Cloud Hypervisor supports the `x86-64` and `AArch64` architectures. There are 65some small differences in functionality between the two architectures 66(see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)). 67 68### Guest OS 69 70Cloud Hypervisor supports `64-bit Linux` and Windows 10/Windows Server 2019. 71 72# 2. Getting Started 73 74Below sections describe how to build and run Cloud Hypervisor on the `x86_64` 75platform. For getting started on the `AArch64` platform, please refer to the 76[Arm64 documentation](docs/arm64.md). 77 78## Host OS 79 80Based on required KVM functionality the minimum host kernel version is 4.11. 81For adequate peformance the minimum recommended host kernel vesion is 5.6. The 82majority of the CI currently tests with kernel version 5.15. 83 84## Preparation 85 86We create a folder to build and run `cloud-hypervisor` at `$HOME/cloud-hypervisor` 87 88```shell 89$ export CLOUDH=$HOME/cloud-hypervisor 90$ mkdir $CLOUDH 91``` 92 93## Install prerequisites 94 95You need to install some prerequisite packages in order to build and test Cloud 96Hypervisor. Here, all the steps are based on Ubuntu, for other Linux 97distributions please replace the package manager and package name. 98 99```shell 100# Install build-essential, git, and qemu-utils 101$ sudo apt install git build-essential qemu-utils 102# Install rust tool chain 103$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh 104# If you want to build statically linked binary please add musl target 105$ rustup target add x86_64-unknown-linux-musl 106``` 107 108## Clone and build 109 110First you need to clone and build the cloud-hypervisor repo: 111 112```shell 113$ pushd $CLOUDH 114$ git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git 115$ cd cloud-hypervisor 116$ cargo build --release 117 118# We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host. 119$ sudo setcap cap_net_admin+ep ./target/release/cloud-hypervisor 120 121# If you want to build statically linked binary 122$ cargo build --release --target=x86_64-unknown-linux-musl --all 123$ popd 124``` 125 126This will build a `cloud-hypervisor` binary under 127`$CLOUDH/cloud-hypervisor/target/release/cloud-hypervisor`. 128 129### Containerized builds and tests 130 131If you want to build and test Cloud Hypervisor without having to install all the 132required dependencies (The rust toolchain, cargo tools, etc), you can also use 133Cloud Hypervisor's development script: `dev_cli.sh`. Please note that upon its 134first invocation, this script will pull a fairly large container image. 135 136For example, to build the Cloud Hypervisor release binary: 137 138```shell 139$ pushd $CLOUDH 140$ cd cloud-hypervisor 141$ ./scripts/dev_cli.sh build --release 142``` 143 144With `dev_cli.sh`, one can also run the Cloud Hypervisor CI locally. This can be 145very convenient for debugging CI errors without having to fully rely on the 146Cloud Hypervisor CI infrastructure. 147 148For example, to run the Cloud Hypervisor unit tests: 149 150```shell 151$ ./scripts/dev_cli.sh tests --unit 152``` 153 154Run the `./scripts/dev_cli.sh --help` command to view all the supported 155development script commands and their related options. 156 157## Use Prebuilt Binaries 158 159Cloud Hypervisor packages targeting some popular Linux distributions are available 160thanks to the [Open Build Service](https://build.opensuse.org). The 161[OBS README](https://github.com/cloud-hypervisor/obs-packaging) explains how to 162enable the repository in a supported Linux distribution and install Cloud Hypervisor 163and accompanying packages. Please report any packaging issues in the 164[obs-packaging](https://github.com/cloud-hypervisor/obs-packaging) repository. 165 166## Run 167 168You can run a guest VM by either using an existing cloud image or booting into 169your own kernel and disk image. 170 171### Cloud image 172 173Cloud Hypervisor supports booting disk images containing all needed 174components to run cloud workloads, a.k.a. cloud images. To do that we rely on 175the [Rust Hypervisor 176Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) project 177to provide an ELF formatted KVM firmware for `cloud-hypervisor` to directly 178boot into. 179 180We need to get the latest `rust-hypervisor-firmware` release and also a working 181cloud image. Here we will use a Ubuntu image: 182 183```shell 184$ pushd $CLOUDH 185$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img 186$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw 187$ wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.4.1/hypervisor-fw 188$ popd 189``` 190 191```shell 192$ pushd $CLOUDH 193$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor 194$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 195 --kernel ./hypervisor-fw \ 196 --disk path=focal-server-cloudimg-amd64.raw \ 197 --cpus boot=4 \ 198 --memory size=1024M \ 199 --net "tap=,mac=,ip=,mask=" 200$ popd 201``` 202 203Multiple arguments can be given to the `--disk` parameter. 204 205### Custom kernel and disk image 206 207#### Building your kernel 208 209Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel. 210In order to support virtio-watchdog we have our own development branch. You are 211of course able to use your own kernel but these instructions will continue with 212the version that we develop and test against. 213 214To build the kernel: 215 216```shell 217 218# Clone the Cloud Hypervisor Linux branch 219$ pushd $CLOUDH 220$ git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-5.15.12 linux-cloud-hypervisor 221$ pushd linux-cloud-hypervisor 222 223# Use the cloud-hypervisor kernel config to build your kernel 224$ cp $CLOUDH/cloud-hypervisor/resources/linux-config-x86_64 .config 225$ KCFLAGS="-Wa,-mx86-used-note=no" make bzImage -j `nproc` 226$ popd 227``` 228 229The `vmlinux` kernel image will then be located at 230`linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin`. 231 232#### Disk image 233 234For the disk image, we will use a Ubuntu cloud image that contains a root 235partition: 236 237```shell 238$ pushd $CLOUDH 239$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img 240$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw 241$ popd 242``` 243 244#### Booting the guest VM 245 246Now we can directly boot into our custom kernel and make it use the Ubuntu root 247partition. If we want to have 4 vCPUs and 1024 MBytes of memory: 248 249```shell 250$ pushd $CLOUDH 251$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor 252$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 253 --kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \ 254 --disk path=focal-server-cloudimg-amd64.raw \ 255 --cmdline "console=hvc0 root=/dev/vda1 rw" \ 256 --cpus boot=4 \ 257 --memory size=1024M \ 258 --net "tap=,mac=,ip=,mask=" 259``` 260 261The above example use the `virtio-console` device as the guest console, and this 262device may not be enabled soon enough by the guest kernel to get early kernel 263debug messages. 264 265When in need for earlier debug messages, using the legacy serial device based 266console is preferred: 267 268``` 269$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 270 --kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \ 271 --console off \ 272 --serial tty \ 273 --disk path=focal-server-cloudimg-amd64.raw \ 274 --cmdline "console=ttyS0 root=/dev/vda1 rw" \ 275 --cpus boot=4 \ 276 --memory size=1024M \ 277 --net "tap=,mac=,ip=,mask=" 278``` 279 280# 3. Status 281 282Cloud Hypervisor is under active development. The following stability guarantees 283are currently made: 284 285* The API (including command line options) will not be removed or changed in a 286 breaking way without a minimum of 2 major releases notice. Where possible 287 warnings will be given about the use of deprecated functionality and the 288 deprecations will be documented in the release notes. 289 290* Point releases will be made between individual releases where there are 291 substantial bug fixes or security issues that need to be fixed. These point 292 releases will only include bug fixes. 293 294Currently the following items are **not** guaranteed across updates: 295 296* Snapshot/restore is not supported across different versions 297* Live migration is not supported across different versions 298* The following features are considered experimental and may change 299 substantially between releases: TDX, vfio-user, vDPA. 300 301As of 2022-04-05, the following cloud images are supported: 302 303- [Ubuntu Bionic](https://cloud-images.ubuntu.com/bionic/current/) (cloudimg) 304- [Ubuntu Focal](https://cloud-images.ubuntu.com/focal/current/) (cloudimg) 305- [Ubuntu Jammy](https://cloud-images.ubuntu.com/jammy/current/) (cloudimg) 306 307Direct kernel boot to userspace should work with a rootfs from most 308distributions. 309 310Further details can be found in the [release documentation](docs/releases.md). 311 312## Hot Plug 313 314Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO), 315`virtio-{net,block,pmem,fs,vsock}` and memory resizing. This 316[document](docs/hotplug.md) details how to add devices to a running VM. 317 318## Device Model 319 320Details of the device model can be found in this 321[documentation](docs/device_model.md). 322 323## TODO 324 325We are not tracking the Cloud Hypervisor TODO list from a specific git tracked 326file but through 327[github issues](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/new) 328instead. 329 330## Roadmap 331 332The project roadmap is tracked through a [GitHub project](https://github.com/orgs/cloud-hypervisor/projects/6). 333 334# 4. `rust-vmm` project dependency 335 336In order to satisfy the design goal of having a high-performance, 337security-focused hypervisor the decision was made to use the 338[Rust](https://www.rust-lang.org/) programming language. The language's strong 339focus on memory and thread safety makes it an ideal candidate for implementing 340VMMs. 341 342Instead of implementing the VMM components from scratch, Cloud Hypervisor is 343importing the [rust-vmm](https://github.com/rust-vmm) crates, and sharing code 344and architecture together with other VMMs like e.g. Amazon's 345[Firecracker](https://firecracker-microvm.github.io/) and Google's 346[crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm/). 347 348Cloud Hypervisor embraces the rust-vmm project goals, which is to be able to 349share and re-use as many virtualization crates as possible. As such, the Cloud 350Hypervisor relationship with the rust-vmm project is twofold: 351 3521. It will use as much of the rust-vmm code as possible. Any new rust-vmm crate 353 that's relevant to the project goals will be integrated as soon as possible. 3542. As it is likely that the rust-vmm project will lack some of the features that 355 Cloud Hypervisor needs (e.g. ACPI, VFIO, vhost-user, etc), we will be using 356 the Cloud Hypervisor VMM to implement and test them, and contribute them back 357 to the rust-vmm project. 358 359## Firecracker and crosvm 360 361A large part of the Cloud Hypervisor code is based on either the Firecracker or 362the crosvm projects implementations. Both of these are VMMs written in Rust with 363a focus on safety and security, like Cloud Hypervisor. 364 365However we want to emphasize that the Cloud Hypervisor project is neither a fork 366nor a reimplementation of any of those projects. The goals and use cases we're 367trying to meet are different. We're aiming at supporting cloud workloads, i.e. 368those modern, full Linux distribution images currently being run by Cloud 369Service Provider (CSP) tenants. 370 371Our primary target is not to support client or serverless use cases, and as such 372our code base already diverges from the crosvm and Firecracker ones. As we add 373more features to support our use cases, we believe that the divergence will 374increase while at the same time sharing as much of the fundamental 375virtualization code through the rust-vmm project crates as possible. 376 377# 5. Community 378 379The Cloud Hypervisor project follows the governance, and community guidelines 380described in the [Community](https://github.com/cloud-hypervisor/community) 381repository. 382 383## Contribute 384 385We are working on building a global, diverse and collaborative community around 386the Cloud Hypervisor project. Anyone who is interested in 387[contributing](CONTRIBUTING.md) to the project is welcome to participate. 388 389We believe that contributing to a open source project like Cloud Hypervisor 390covers a lot more than just sending code. Testing, documentation, pull request 391reviews, bug reports, feature requests, project improvement suggestions, etc, 392are all equal and welcome means of contribution. See the 393[CONTRIBUTING](CONTRIBUTING.md) document for more details. 394 395## Join us 396 397Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI) 398and [join us on Slack](https://cloud-hypervisor.slack.com/). 399 400## Security issues 401 402Please contact the maintainers listed in the MAINTAINERS.md file with security issues. 403