1- [1. What is Cloud Hypervisor?](#1-what-is-cloud-hypervisor) 2 - [Objectives](#objectives) 3 - [High Level](#high-level) 4 - [Architectures](#architectures) 5 - [Guest OS](#guest-os) 6- [2. Getting Started](#2-getting-started) 7 - [Preparation](#preparation) 8 - [Install prerequisites](#install-prerequisites) 9 - [Clone and build](#clone-and-build) 10 - [Containerized builds and tests](#containerized-builds-and-tests) 11 - [Run](#run) 12 - [Cloud image](#cloud-image) 13 - [Custom kernel and disk image](#custom-kernel-and-disk-image) 14 - [Building your kernel](#building-your-kernel) 15 - [Disk image](#disk-image) 16 - [Booting the guest VM](#booting-the-guest-vm) 17- [3. Status](#3-status) 18 - [Hot Plug](#hot-plug) 19 - [Device Model](#device-model) 20 - [TODO](#todo) 21- [4. `rust-vmm` project dependency](#4-rust-vmm-project-dependency) 22 - [Firecracker and crosvm](#firecracker-and-crosvm) 23- [5. Community](#5-community) 24 - [Contribute](#contribute) 25 - [Join us](#join-us) 26 - [Security issues](#security-issues) 27 28# 1. What is Cloud Hypervisor? 29 30Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on 31top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt) 32hypervisor and Microsoft Hypervisor (MSHV). 33 34The project focuses on exclusively running modern, cloud workloads, on top of 35a limited set of hardware architectures and platforms. Cloud workloads refers 36to those that are usually run by customers inside a cloud provider. For our 37purposes this means modern operating systems with most I/O handled by 38paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 3964-bit CPUs. 40 41Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is 42based on the [rust-vmm](https://github.com/rust-vmm) crates. 43 44## Objectives 45 46### High Level 47 48- Runs on KVM or MSHV 49- Minimal emulation 50- Low latency 51- Low memory footprint 52- Low complexity 53- High performance 54- Small attack surface 55- 64-bit support only 56- CPU, memory, PCI hotplug 57- Machine to machine migration 58 59### Architectures 60 61Cloud Hypervisor supports the `x86-64` and `AArch64` architectures. There are 62some small differences in functionality between the two architectures 63(see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)). 64 65### Guest OS 66 67Cloud Hypervisor supports `64-bit Linux` and Windows 10/Windows Server 2019. 68 69# 2. Getting Started 70 71Below sections describe how to build and run Cloud Hypervisor on the `x86_64` 72platform. For getting started on the `AArch64` platform, please refer to the 73[Arm64 documentation](docs/arm64.md). 74 75## Preparation 76 77We create a folder to build and run `cloud-hypervisor` at `$HOME/cloud-hypervisor` 78 79```shell 80$ export CLOUDH=$HOME/cloud-hypervisor 81$ mkdir $CLOUDH 82``` 83 84## Install prerequisites 85 86You need to install some prerequisite packages in order to build and test Cloud 87Hypervisor. Here, all the steps are based on Ubuntu, for other Linux 88distributions please replace the package manager and package name. 89 90```shell 91# Install git 92$ sudo apt install git 93# Install rust tool chain 94$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh 95# Install build-essential 96$ sudo apt install build-essential 97# If you want to build statically linked binary please add musl target 98$ rustup target add x86_64-unknown-linux-musl 99``` 100 101## Clone and build 102 103First you need to clone and build the cloud-hypervisor repo: 104 105```shell 106$ pushd $CLOUDH 107$ git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git 108$ cd cloud-hypervisor 109$ cargo build --release 110 111# We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host. 112$ sudo setcap cap_net_admin+ep ./target/release/cloud-hypervisor 113 114# If you want to build statically linked binary 115$ cargo build --release --target=x86_64-unknown-linux-musl --all 116$ popd 117``` 118 119This will build a `cloud-hypervisor` binary under 120`$CLOUDH/cloud-hypervisor/target/release/cloud-hypervisor`. 121 122### Containerized builds and tests 123 124If you want to build and test Cloud Hypervisor without having to install all the 125required dependencies (The rust toolchain, cargo tools, etc), you can also use 126Cloud Hypervisor's development script: `dev_cli.sh`. Please note that upon its 127first invocation, this script will pull a fairly large container image. 128 129For example, to build the Cloud Hypervisor release binary: 130 131```shell 132$ pushd $CLOUDH 133$ cd cloud-hypervisor 134$ ./scripts/dev_cli.sh build --release 135``` 136 137With `dev_cli.sh`, one can also run the Cloud Hypervisor CI locally. This can be 138very convenient for debugging CI errors without having to fully rely on the 139Cloud Hypervisor CI infrastructure. 140 141For example, to run the Cloud Hypervisor unit tests: 142 143```shell 144$ ./scripts/dev_cli.sh tests --unit 145``` 146 147Run the `./scripts/dev_cli.sh --help` command to view all the supported 148development script commands and their related options. 149 150## Run 151 152You can run a guest VM by either using an existing cloud image or booting into 153your own kernel and disk image. 154 155### Cloud image 156 157Cloud Hypervisor supports booting disk images containing all needed 158components to run cloud workloads, a.k.a. cloud images. To do that we rely on 159the [Rust Hypervisor 160Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) project 161to provide an ELF formatted KVM firmware for `cloud-hypervisor` to directly 162boot into. 163 164We need to get the latest `rust-hypervisor-firmware` release and also a working 165cloud image. Here we will use a Ubuntu image: 166 167```shell 168$ pushd $CLOUDH 169$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img 170$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw 171$ wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.3.2/hypervisor-fw 172$ popd 173``` 174 175```shell 176$ pushd $CLOUDH 177$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor 178$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 179 --kernel ./hypervisor-fw \ 180 --disk path=focal-server-cloudimg-amd64.raw \ 181 --cpus boot=4 \ 182 --memory size=1024M \ 183 --net "tap=,mac=,ip=,mask=" 184$ popd 185``` 186 187Multiple arguments can be given to the `--disk` parameter. 188 189### Custom kernel and disk image 190 191#### Building your kernel 192 193Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel. 194In order to support virtio-watchdog we have our own development branch. You are 195of course able to use your own kernel but these instructions will continue with 196the version that we develop and test against. 197 198To build the kernel: 199 200```shell 201 202# Clone the Cloud Hypervisor Linux branch 203$ pushd $CLOUDH 204$ git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-5.15.12 linux-cloud-hypervisor 205$ pushd linux-cloud-hypervisor 206 207# Use the cloud-hypervisor kernel config to build your kernel 208$ cp $CLOUDH/cloud-hypervisor/resources/linux-config-x86_64 .config 209$ KCFLAGS="-Wa,-mx86-used-note=no" make bzImage -j `nproc` 210$ popd 211``` 212 213The `vmlinux` kernel image will then be located at 214`linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin`. 215 216#### Disk image 217 218For the disk image, we will use a Ubuntu cloud image that contains a root 219partition: 220 221```shell 222$ pushd $CLOUDH 223$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img 224$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw 225$ popd 226``` 227 228#### Booting the guest VM 229 230Now we can directly boot into our custom kernel and make it use the Ubuntu root 231partition. If we want to have 4 vCPUs and 1024 MBytes of memory: 232 233```shell 234$ pushd $CLOUDH 235$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor 236$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 237 --kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \ 238 --disk path=focal-server-cloudimg-amd64.raw \ 239 --cmdline "console=hvc0 root=/dev/vda1 rw" \ 240 --cpus boot=4 \ 241 --memory size=1024M \ 242 --net "tap=,mac=,ip=,mask=" 243``` 244 245The above example use the `virtio-console` device as the guest console, and this 246device may not be enabled soon enough by the guest kernel to get early kernel 247debug messages. 248 249When in need for earlier debug messages, using the legacy serial device based 250console is preferred: 251 252``` 253$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 254 --kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \ 255 --console off \ 256 --serial tty \ 257 --disk path=focal-server-cloudimg-amd64.raw \ 258 --cmdline "console=ttyS0 root=/dev/vda1 rw" \ 259 --cpus boot=4 \ 260 --memory size=1024M \ 261 --net "tap=,mac=,ip=,mask=" 262``` 263 264# 3. Status 265 266Cloud Hypervisor is under active development. The following stability guarantees 267are currently made: 268 269* The API (including command line options) will not be removed or changed in a 270 breaking way without a minimum of 2 major releases notice. Where possible 271 warnings will be given about the use of deprecated functionality and the 272 deprecations will be documented in the release notes. 273 274* Point releases will be made between individual releases where there are 275 substantial bug fixes or security issues that need to be fixed. These point 276 releases will only include bug fixes. 277 278Currently the following items are **not** guaranteed across updates: 279 280* Snapshot/restore is not supported across different versions 281* Live migration is not supported across different versions 282* The following features are considered experimental and may change 283 substantially between releases: TDX, vfio-user, vDPA. 284 285As of 2022-04-05, the following cloud images are supported: 286 287- [Ubuntu Bionic](https://cloud-images.ubuntu.com/bionic/current/) (cloudimg) 288- [Ubuntu Focal](https://cloud-images.ubuntu.com/focal/current/) (cloudimg) 289- [Ubuntu Jammy](https://cloud-images.ubuntu.com/jammy/current/) (cloudimg) 290 291Direct kernel boot to userspace should work with a rootfs from most 292distributions. 293 294## Hot Plug 295 296Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO), 297`virtio-{net,block,pmem,fs,vsock}` and memory resizing. This 298[document](docs/hotplug.md) details how to add devices to a running VM. 299 300## Device Model 301 302Details of the device model can be found in this 303[documentation](docs/device_model.md). 304 305## TODO 306 307We are not tracking the Cloud Hypervisor TODO list from a specific git tracked 308file but through 309[github issues](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/new) 310instead. 311 312# 4. `rust-vmm` project dependency 313 314In order to satisfy the design goal of having a high-performance, 315security-focused hypervisor the decision was made to use the 316[Rust](https://www.rust-lang.org/) programming language. The language's strong 317focus on memory and thread safety makes it an ideal candidate for implementing 318VMMs. 319 320Instead of implementing the VMM components from scratch, Cloud Hypervisor is 321importing the [rust-vmm](https://github.com/rust-vmm) crates, and sharing code 322and architecture together with other VMMs like e.g. Amazon's 323[Firecracker](https://firecracker-microvm.github.io/) and Google's 324[crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm/). 325 326Cloud Hypervisor embraces the rust-vmm project goals, which is to be able to 327share and re-use as many virtualization crates as possible. As such, the Cloud 328Hypervisor relationship with the rust-vmm project is twofold: 329 3301. It will use as much of the rust-vmm code as possible. Any new rust-vmm crate 331 that's relevant to the project goals will be integrated as soon as possible. 3322. As it is likely that the rust-vmm project will lack some of the features that 333 Cloud Hypervisor needs (e.g. ACPI, VFIO, vhost-user, etc), we will be using 334 the Cloud Hypervisor VMM to implement and test them, and contribute them back 335 to the rust-vmm project. 336 337## Firecracker and crosvm 338 339A large part of the Cloud Hypervisor code is based on either the Firecracker or 340the crosvm projects implementations. Both of these are VMMs written in Rust with 341a focus on safety and security, like Cloud Hypervisor. 342 343However we want to emphasize that the Cloud Hypervisor project is neither a fork 344nor a reimplementation of any of those projects. The goals and use cases we're 345trying to meet are different. We're aiming at supporting cloud workloads, i.e. 346those modern, full Linux distribution images currently being run by Cloud 347Service Provider (CSP) tenants. 348 349Our primary target is not to support client or serverless use cases, and as such 350our code base already diverges from the crosvm and Firecracker ones. As we add 351more features to support our use cases, we believe that the divergence will 352increase while at the same time sharing as much of the fundamental 353virtualization code through the rust-vmm project crates as possible. 354 355# 5. Community 356 357The Cloud Hypervisor project follows the governance, and community guidelines 358described in the [Community](https://github.com/cloud-hypervisor/community) 359repository. 360 361## Contribute 362 363We are working on building a global, diverse and collaborative community around 364the Cloud Hypervisor project. Anyone who is interested in 365[contributing](CONTRIBUTING.md) to the project is welcome to participate. 366 367We believe that contributing to a open source project like Cloud Hypervisor 368covers a lot more than just sending code. Testing, documentation, pull request 369reviews, bug reports, feature requests, project improvement suggestions, etc, 370are all equal and welcome means of contribution. See the 371[CONTRIBUTING](CONTRIBUTING.md) document for more details. 372 373## Join us 374 375Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI) 376and [join us on Slack](https://cloud-hypervisor.slack.com/). 377 378## Security issues 379 380Please contact the maintainers listed in the MAINTAINERS.md file with security issues. 381