1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 * [External API](#external-api) 3 + [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 * [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 * [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 * [Create a Virtual Machine](#create-a-virtual-machine) 10 * [Boot a Virtual Machine](#boot-a-virtual-machine) 11 * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 * [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 + [Command Line Interface](#command-line-interface) 15 + [REST API and CLI Architecural Relationship](#rest-api-and-cli-architectural-relationship) 16 * [Internal API](#internal-api) 17 + [Goals and Design](#goals-and-design) 18 * [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74Action | Endpoint | Request Body | Response Body | Prerequisites 75------------------------------------|-----------------|--------------|----------------------------|--------------------------- 76Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A 77Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running 78 79#### Virtual Machine (VM) Actions 80 81Action | Endpoint | Request Body | Response Body | Prerequisites 82-----------------------------------|---------------------|---------------------------|-------------------|--------------------------- 83Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet 84Delete the VM | `/vm.delete` | N/A | N/A | The VM is created but not booted 85Boot the VM | `/vm.boot` | N/A | N/A | The VM is created 86Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted 87Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted 88Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted 89Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused 90Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 91Remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 92Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created 93Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | N/A | The VM is booted 94Remove VFIO PCI device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted 95Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | N/A | The VM is booted 96Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | N/A | The VM is booted 97Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | N/A | The VM is booted 98 99### REST API Examples 100 101For the following set of examples, we assume Cloud Hypervisor is started with 102the REST API available at `/tmp/cloud-hypervisor.sock`: 103 104``` 105$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 106Cloud Hypervisor Guest 107 API server: /tmp/cloud-hypervisor.sock 108 vCPUs: 1 109 Memory: 512 MB 110 Kernel: None 111 Kernel cmdline: 112 Disk(s): None 113``` 114 115#### Create a Virtual Machine 116 117We want to create a virtual machine with the following characteristics: 118 119* 4 vCPUs 120* 1 GB of RAM 121* 1 virtio based networking interface 122* Direct kernel boot from a custom 5.5.0 Linux kernel located at 123 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 124* Using a Clear Linux image as its root filesystem, located at 125 `/opt/clh/images/clear-30080-kvm.img` 126 127```shell 128#!/bin/bash 129 130curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 131 -X PUT 'http://localhost/api/v1/vm.create' \ 132 -H 'Accept: application/json' \ 133 -H 'Content-Type: application/json' \ 134 -d '{ 135 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 136 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 137 "cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"}, 138 "disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}], 139 "rng":{"src":"/dev/urandom"}, 140 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 141 }' 142``` 143 144#### Boot a Virtual Machine 145 146Once the VM is created, we can boot it: 147 148```shell 149#!/bin/bash 150 151curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 152``` 153 154#### Dump a Virtual Machine Information 155 156We can fetch information about any VM, as soon as it's created: 157 158```shell 159#!/bin/bash 160 161curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 162 -X GET 'http://localhost/api/v1/vm.info' \ 163 -H 'Accept: application/json' 164``` 165 166#### Reboot a Virtual Machine 167 168We can reboot a VM that's already booted: 169 170```shell 171#!/bin/bash 172 173curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 174``` 175 176#### Shut a Virtual Machine Down 177 178Once booted, we can shut a VM down from the REST API: 179 180```shell 181#!/bin/bash 182 183curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 184``` 185 186### Command Line Interface 187 188The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 189the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 190the launched VM once they're up and running. 191 192If you want to inspect the VMM, or control the VM after launching Cloud 193Hypervisor from the CLI, you must use the [REST API](#rest-api). 194 195From the CLI, one can either: 196 1971. Create and boot a complete virtual machine by using the CLI options to build 198 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 199 options. As soon as the `cloud-hypervisor` binary is launched, the 200 [REST API](#rest-api) is available for controlling and managing the VM. 2011. Start the [REST API](#rest-api) server only, by not passing any VM 202 configuration options. The VM can then be asynchronously created and booted 203 by sending HTTP commands to the [REST API](#rest-api). Check the 204 [REST API examples](#rest-api-examples) section for more details. 205 206### REST API and CLI Architectural Relationship 207 208The REST API and the CLI both rely on a common, [internal API](#internal-api). 209 210The CLI options are parsed by the 211[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into 212[internal API](#internal-api) commands. 213 214The REST API is processed by an HTTP thread using the 215[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 216crate. As with the CLI, the HTTP requests eventually get translated into 217[internal API](#internal-api) commands. 218 219As a summary, the REST API and the CLI are essentially frontends for the 220[internal API](#internal-api): 221 222``` 223 +------------------+ 224 REST API | | 225 +--------->+ micro_http +--------+ 226 | | | | 227 | +------------------+ | 228 | | +------------------------+ 229 | | | | 230+------------+ | | | | 231| | | | | +--------------+ | 232| User +---------+ +------> | Internal API | | 233| | | | | +--------------+ | 234+------------+ | | | | 235 | | | | 236 | | +------------------------+ 237 | +----------+ | VMM 238 | CLI | | | 239 +----------->+ clap +--------------+ 240 | | 241 +----------+ 242 243 244``` 245 246## Internal API 247 248The Cloud Hypervisor internal API, as its name suggests, is used internally 249by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 250send commands and responses to each others. 251 252It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 253and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 254control loop. 255 256API producers are the HTTP thread handling the [REST API](#rest-api) and the 257main thread that initially parses the [CLI](#command-line-interface). 258 259### Goals and Design 260 261The internal API is designed for controlling, managing and inspecting a Cloud 262Hypervisor VMM and its guest. It is a backend for handling external, user 263visible requests through either the [REST API](#rest-api) or the 264[CLI](#command-line-interface) interfaces. 265 266The API follows a command-response scheme that closely maps the [REST API](#rest-api). 267Any command must be replied to with a response. 268 269Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 270are received and processed by the VMM control loop. 271 272In order for the VMM control loop to respond to any internal API command, it 273must be able to send a response back to the MPSC sender. For that purpose, all 274internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 275end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 276 277The sender of any internal API command is therefore responsible for: 278 2791. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 280 channel. 2811. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 282 end of the response channel as part of the internal API command payload. 2831. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 284 end of the response channel. 285 286## End to End Example 287 288In order to further understand how the external and internal Cloud Hypervisor 289APIs work together, let's look at a complete VM creation flow, from the 290[REST API](#rest-api) call, to the reply the external user will receive: 291 2921. A user or operator sends an HTTP request to the Cloud Hypervisor 293 [REST API](#rest-api) in order to creates a virtual machine: 294 ``` 295 shell 296 #!/bin/bash 297 298 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 299 -X PUT 'http://localhost/api/v1/vm.create' \ 300 -H 'Accept: application/json' \ 301 -H 'Content-Type: application/json' \ 302 -d '{ 303 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 304 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 305 "cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"}, 306 "disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}], 307 "rng":{"src":"/dev/urandom"}, 308 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 309 }' 310 ``` 3111. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 312 HTTP request JSON body into an internal `VmConfig` structure. 3131. The Cloud Hypervisor HTTP thread creates an 314 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 315 server to send its response back. 3161. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 317 virtual machine. The command's payload is made of the de-serialized 318 `VmConfig` structure and the response channel: 319 ```Rust 320 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 321 ``` 3221. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 323 for the response: 324 ```Rust 325 // Send the VM creation request. 326 api_sender 327 .send(ApiRequest::VmCreate(config, response_sender)) 328 .map_err(ApiError::RequestSend)?; 329 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 330 331 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 332 ``` 3331. The Cloud Hypervisor control loop receives the command, as it listens on the 334 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 335 ```Rust 336 // Read from the API receiver channel 337 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 338 ``` 3391. The Cloud Hypervisor control loop matches the received internal API against 340 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 341 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 342 command payload. It stores the `VmConfig` structure and replies back to the 343 sender ((The HTTP thread): 344 ```Rust 345 match api_request { 346 ApiRequest::VmCreate(config, sender) => { 347 // We only store the passed VM config. 348 // The VM will be created when being asked to boot it. 349 let response = if self.vm_config.is_none() { 350 self.vm_config = Some(config); 351 Ok(ApiResponsePayload::Empty) 352 } else { 353 Err(ApiError::VmAlreadyCreated) 354 }; 355 356 sender.send(response).map_err(Error::ApiResponseSend)?; 357 } 358 ``` 3591. The Cloud Hypervisor HTTP thread receives the internal API command response 360 as the return value from its `VmCreate` HTTP handler. Depending on the 361 control loop internal API response, it generates the appropriate HTTP 362 response: 363 ```Rust 364 // Call vm_create() 365 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 366 .map_err(HttpError::VmCreate) 367 { 368 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 369 Err(e) => error_response(e, StatusCode::InternalServerError), 370 } 371 ``` 3721. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 373 user. This is abstracted by the 374 [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 375 crate. 376 377