1[](https://travis-ci.com/cloud-hypervisor/cloud-hypervisor) 2 3- [1. What is Cloud Hypervisor?](#1-what-is-cloud-hypervisor) 4 - [Objectives](#objectives) 5 - [High Level](#high-level) 6 - [Architectures](#architectures) 7 - [Guest OS](#guest-os) 8- [2. Getting Started](#2-getting-started) 9 - [Install prerequisites](#install-prerequisites) 10 - [Clone and build](#clone-and-build) 11 - [Containerized builds and tests](#containerized-builds-and-tests) 12 - [Run](#run) 13 - [Cloud image](#cloud-image) 14 - [Custom kernel and disk image](#custom-kernel-and-disk-image) 15 - [Building your kernel](#building-your-kernel) 16 - [Disk image](#disk-image) 17 - [Booting the guest VM](#booting-the-guest-vm) 18- [3. Status](#3-status) 19 - [Hot Plug](#hot-plug) 20 - [Device Model](#device-model) 21 - [TODO](#todo) 22- [4. `rust-vmm` project dependency](#4-rust-vmm-project-dependency) 23 - [Firecracker and crosvm](#firecracker-and-crosvm) 24- [5. Community](#5-community) 25 - [Contribute](#contribute) 26 - [Join us](#join-us) 27 - [Security issues](#security-issues) 28 29# 1. What is Cloud Hypervisor? 30 31Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt) and the MSHV hypervisors . 32 33The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms. 34Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern operating systems with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 64-bit CPUs. 35 36Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is based on the [rust-vmm](https://github.com/rust-vmm) crates. 37 38## Objectives 39 40### High Level 41 42- Runs on KVM or MSHV 43- Minimal emulation 44- Low latency 45- Low memory footprint 46- Low complexity 47- High performance 48- Small attack surface 49- 64-bit support only 50- CPU, memory, PCI hotplug 51- Machine to machine migration 52 53### Architectures 54 55Cloud Hypervisor supports the `x86-64` and `AArch64` architectures. There are some small differences in functionality between the two architectures (see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)). 56 57### Guest OS 58 59Cloud Hypervisor supports `64-bit Linux` and Windows 10/Windows Server 2019. 60 61# 2. Getting Started 62 63We create a folder to build and run `cloud-hypervisor` at `$HOME/cloud-hypervisor` 64 65```shell 66$ export CLOUDH=$HOME/cloud-hypervisor 67$ mkdir $CLOUDH 68``` 69 70## Install prerequisites 71 72You need to install some prerequisite packages in order to build and test Cloud Hypervisor. 73Here, all the steps are based on Ubuntu, for other Linux distributions please replace the 74package manager and package name. 75 76```shell 77# Install git 78$ sudo apt install git 79# Install rust tool chain 80$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh 81# Install build-essential 82$ sudo apt install build-essential 83# If you want to build statically linked binary please add musl target 84$ rustup target add x86_64-unknown-linux-musl 85``` 86 87## Clone and build 88 89First you need to clone and build the cloud-hypervisor repo: 90 91```shell 92$ pushd $CLOUDH 93$ git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git 94$ cd cloud-hypervisor 95$ cargo build --release 96 97# We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host. 98$ sudo setcap cap_net_admin+ep ./target/release/cloud-hypervisor 99 100# If you want to build statically linked binary 101$ cargo build --release --target=x86_64-unknown-linux-musl --all 102$ popd 103``` 104 105This will build a `cloud-hypervisor` binary under `$CLOUDH/cloud-hypervisor/target/release/cloud-hypervisor`. 106 107### Containerized builds and tests 108 109If you want to build and test Cloud Hypervisor without having to install all the 110required dependencies (The rust toolchain, cargo tools, etc), you can also use 111Cloud Hypervisor's development script: `dev_cli.sh`. Please note that upon its 112first invocation, this script will pull a fairly large container image. 113 114For example, to build the Cloud Hypervisor release binary: 115 116```shell 117$ pushd $CLOUDH 118$ cd cloud-hypervisor 119$ ./scripts/dev_cli.sh build --release 120``` 121 122With `dev_cli.sh`, one can also run the Cloud Hypervisor CI locally. This can be 123very convenient for debugging CI errors without having to fully rely on the 124Cloud Hypervisor CI infrastructure. 125 126For example, to run the Cloud Hypervisor unit tests: 127 128```shell 129$ ./scripts/dev_cli.sh tests --unit 130``` 131 132Run the `./scripts/dev_cli.sh --help` command to view all the supported 133development script commands and their related options. 134 135## Run 136 137You can run a guest VM by either using an existing cloud image or booting into your own kernel and disk image. 138 139### Cloud image 140 141Cloud Hypervisor supports booting disk images containing all needed 142components to run cloud workloads, a.k.a. cloud images. To do that we rely on 143the [Rust Hypervisor 144Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) project to provide 145an ELF 146formatted KVM firmware for `cloud-hypervisor` to directly boot into. 147 148We need to get the latest `rust-hypervisor-firmware` release and also a working cloud image. Here we will use a Ubuntu image: 149 150```shell 151$ pushd $CLOUDH 152$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img 153$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw 154$ wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.3.2/hypervisor-fw 155$ popd 156``` 157 158```shell 159$ pushd $CLOUDH 160$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor 161$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 162 --kernel ./hypervisor-fw \ 163 --disk path=focal-server-cloudimg-amd64.raw \ 164 --cpus boot=4 \ 165 --memory size=1024M \ 166 --net "tap=,mac=,ip=,mask=" \ 167 --rng 168$ popd 169``` 170 171Multiple arguments can be given to the `--disk` parameter. 172 173### Custom kernel and disk image 174 175#### Building your kernel 176 177Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel. In order to support virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against. 178 179To build the kernel: 180 181```shell 182 183# Clone the Cloud Hypervisor Linux branch 184$ pushd $CLOUDH 185$ git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-5.13-rc5 linux-cloud-hypervisor 186$ pushd linux-cloud-hypervisor 187 188# Use the cloud-hypervisor kernel config to build your kernel 189$ cp $CLOUDH/cloud-hypervisor/resources/linux-config-x86_64 .config 190$ make bzImage -j `nproc` 191$ popd 192``` 193 194The `vmlinux` kernel image will then be located at `linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin`. 195 196#### Disk image 197 198For the disk image, we will use a Ubuntu cloud image that contains a root partition: 199 200```shell 201$ pushd $CLOUDH 202$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img 203$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw 204$ popd 205``` 206 207#### Booting the guest VM 208 209Now we can directly boot into our custom kernel and make it use the Ubuntu root partition. 210If we want to have 4 vCPUs and 1024 MBytes of memory: 211 212```shell 213$ pushd $CLOUDH 214$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor 215$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 216 --kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \ 217 --disk path=focal-server-cloudimg-amd64.raw \ 218 --cmdline "console=hvc0 root=/dev/vda1 rw" \ 219 --cpus boot=4 \ 220 --memory size=1024M \ 221 --net "tap=,mac=,ip=,mask=" \ 222 --rng 223``` 224 225The above example use the `virtio-console` device as the guest console, and this 226device may not be enabled soon enough by the guest kernel to get early kernel 227debug messages. 228 229When in need for earlier debug messages, using the legacy serial device based 230console is preferred: 231 232``` 233$ ./cloud-hypervisor/target/release/cloud-hypervisor \ 234 --kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \ 235 --console off \ 236 --serial tty \ 237 --disk path=focal-server-cloudimg-amd64.raw \ 238 --cmdline "console=ttyS0 root=/dev/vda1 rw" \ 239 --cpus boot=4 \ 240 --memory size=1024M \ 241 --net "tap=,mac=,ip=,mask=" \ 242 --rng 243``` 244 245# 3. Status 246 247Cloud Hypervisor is under active development. The following stability guarantees are currently made: 248 249* The API (including command line options) will not be removed or changed in a 250 breaking way without a minimum of 2 releases notice. Where possible warnings 251 will be given about the use of deprecated functionality and the deprecations 252 will be documented in the release notes. 253* Point releases will be made between individual releases where there are 254 substantial bug fixes or security issues that need to be fixed. 255 256Currently the following items are **not** guaranteed across updates: 257 258* Snapshot/restore is not supported across different versions 259* Live migration is not supported across different versions 260* The following features are considered experimental and may change 261 substantially between releases: TDX, SGX. 262 263 264As of 2021-04-29, the following cloud images are supported: 265 266- [Ubuntu Bionic](https://cloud-images.ubuntu.com/bionic/current/) (cloudimg) 267- [Ubuntu Focal](https://cloud-images.ubuntu.com/focal/current/) (cloudimg) 268- [Ubuntu Groovy](https://cloud-images.ubuntu.com/groovy/current/) (cloudimg) 269- [Ubuntu Hirsute](https://cloud-images.ubuntu.com/hirsute/current/) (cloudimg) 270 271Direct kernel boot to userspace should work with a rootfs from most distributions. 272 273## Hot Plug 274 275Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO), `virtio-{net,block,pmem,fs,vsock}` and memory resizing. This [document](docs/hotplug.md) details how to add devices to 276a running VM. 277 278## Device Model 279 280Details of the device model can be found in this [documentation](docs/device_model.md). 281 282## TODO 283 284We are not tracking the Cloud Hypervisor TODO list from a specific git tracked file but through 285[github issues](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/new) instead. 286 287# 4. `rust-vmm` project dependency 288 289In order to satisfy the design goal of having a high-performance, security-focused hypervisor the decision 290was made to use the [Rust](https://www.rust-lang.org/) programming language. 291The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs. 292 293Instead of implementing the VMM components from scratch, Cloud Hypervisor is importing the [rust-vmm](https://github.com/rust-vmm) 294crates, and sharing code and architecture together with other VMMs like e.g. Amazon's [Firecracker](https://firecracker-microvm.github.io/) 295and Google's [crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm/). 296 297Cloud Hypervisor embraces the rust-vmm project goals, which is to be able to share and re-use 298as many virtualization crates as possible. As such, the Cloud Hypervisor relationship with the rust-vmm 299project is twofold: 300 3011. It will use as much of the rust-vmm code as possible. Any new rust-vmm crate that's relevant to the project 302 goals will be integrated as soon as possible. 3032. As it is likely that the rust-vmm project will lack some of the features that Cloud Hypervisor needs (e.g. ACPI, 304 VFIO, vhost-user, etc), we will be using the Cloud Hypervisor VMM to implement and test them, and contribute them 305 back to the rust-vmm project. 306 307## Firecracker and crosvm 308 309A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm projects implementations. 310Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor. 311 312However we want to emphasize that the Cloud Hypervisor project is neither a fork nor a reimplementation of any of those 313projects. The goals and use cases we're trying to meet are different. 314We're aiming at supporting cloud workloads, i.e. those modern, full Linux distribution images currently being run by 315Cloud Service Provider (CSP) tenants. 316 317Our primary target is not to support client or serverless use cases, and as such our code base already diverges 318from the crosvm and Firecracker ones. As we add more features to support our use cases, we believe that the divergence 319will increase while at the same time sharing as much of the fundamental virtualization code through the rust-vmm project 320crates as possible. 321 322# 5. Community 323 324The Cloud Hypervisor project follows the governance, and community guidelines described in 325the [Community](https://github.com/cloud-hypervisor/community) repository. 326 327## Contribute 328 329We are working on building a global, diverse and collaborative community around the Cloud Hypervisor project. 330Anyone who is interested in [contributing](CONTRIBUTING.md) to the project is welcome to participate. 331 332We believe that contributing to a open source project like Cloud Hypervisor covers a lot more than just sending 333code. Testing, documentation, pull request reviews, bug reports, feature requests, project improvement suggestions, 334etc, are all equal and welcome means of contribution. See the [CONTRIBUTING](CONTRIBUTING.md) document for more details. 335 336## Join us 337 338Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI) 339and [join us on Slack](https://cloud-hypervisor.slack.com/). 340 341## Security issues 342 343Please use the GitHub security advisories feature for reporting issues: https://github.com/cloud-hypervisor/cloud-hypervisor/security/advisories/new 344
){kind=link}