1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 * [External API](#external-api) 3 + [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 * [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 * [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 * [Create a Virtual Machine](#create-a-virtual-machine) 10 * [Boot a Virtual Machine](#boot-a-virtual-machine) 11 * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 * [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 + [Command Line Interface](#command-line-interface) 15 + [REST API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 16 * [Internal API](#internal-api) 17 + [Goals and Design](#goals-and-design) 18 * [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74Action | Endpoint | Request Body | Response Body | Prerequisites 75------------------------------------|-----------------|--------------|----------------------------|--------------------------- 76Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A 77Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running 78 79#### Virtual Machine (VM) Actions 80 81Action | Endpoint | Request Body | Response Body | Prerequisites 82-----------------------------------|----------------------|---------------------------|--------------------------|--------------------------- 83Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet 84Delete the VM | `/vm.delete` | N/A | N/A | N/A 85Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted 86Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted 87Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted 88Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted 89Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused 90Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 91Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 92Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted 93Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created 94Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted 95Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted 96Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted 97Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted 98Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted 99Add userspace PCI device to the VM | `/vm.add-user-device`| `/schemas/VmAddUserDevice`| `/schemas/PciDeviceInfo` | The VM is booted 100Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted 101Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted 102Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted 103 104### REST API Examples 105 106For the following set of examples, we assume Cloud Hypervisor is started with 107the REST API available at `/tmp/cloud-hypervisor.sock`: 108 109``` 110$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 111Cloud Hypervisor Guest 112 API server: /tmp/cloud-hypervisor.sock 113 vCPUs: 1 114 Memory: 512 MB 115 Kernel: None 116 Kernel cmdline: 117 Disk(s): None 118``` 119 120#### Create a Virtual Machine 121 122We want to create a virtual machine with the following characteristics: 123 124* 4 vCPUs 125* 1 GB of RAM 126* 1 virtio based networking interface 127* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 128 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 129* Using a Ubuntu image as its root filesystem, located at 130 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 131 132```shell 133#!/bin/bash 134 135curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 136 -X PUT 'http://localhost/api/v1/vm.create' \ 137 -H 'Accept: application/json' \ 138 -H 'Content-Type: application/json' \ 139 -d '{ 140 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 141 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 142 "cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 143 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 144 "rng":{"src":"/dev/urandom"}, 145 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 146 }' 147``` 148 149#### Boot a Virtual Machine 150 151Once the VM is created, we can boot it: 152 153```shell 154#!/bin/bash 155 156curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 157``` 158 159#### Dump a Virtual Machine Information 160 161We can fetch information about any VM, as soon as it's created: 162 163```shell 164#!/bin/bash 165 166curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 167 -X GET 'http://localhost/api/v1/vm.info' \ 168 -H 'Accept: application/json' 169``` 170 171#### Reboot a Virtual Machine 172 173We can reboot a VM that's already booted: 174 175```shell 176#!/bin/bash 177 178curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 179``` 180 181#### Shut a Virtual Machine Down 182 183Once booted, we can shut a VM down from the REST API: 184 185```shell 186#!/bin/bash 187 188curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 189``` 190 191### Command Line Interface 192 193The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 194the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 195the launched VM once they're up and running. 196 197If you want to inspect the VMM, or control the VM after launching Cloud 198Hypervisor from the CLI, you must use the [REST API](#rest-api). 199 200From the CLI, one can either: 201 2021. Create and boot a complete virtual machine by using the CLI options to build 203 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 204 options. As soon as the `cloud-hypervisor` binary is launched, the 205 [REST API](#rest-api) is available for controlling and managing the VM. 2061. Start the [REST API](#rest-api) server only, by not passing any VM 207 configuration options. The VM can then be asynchronously created and booted 208 by sending HTTP commands to the [REST API](#rest-api). Check the 209 [REST API examples](#rest-api-examples) section for more details. 210 211### REST API and CLI Architectural Relationship 212 213The REST API and the CLI both rely on a common, [internal API](#internal-api). 214 215The CLI options are parsed by the 216[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into 217[internal API](#internal-api) commands. 218 219The REST API is processed by an HTTP thread using the 220[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 221crate. As with the CLI, the HTTP requests eventually get translated into 222[internal API](#internal-api) commands. 223 224As a summary, the REST API and the CLI are essentially frontends for the 225[internal API](#internal-api): 226 227``` 228 +------------------+ 229 REST API | | 230 +--------->+ micro_http +--------+ 231 | | | | 232 | +------------------+ | 233 | | +------------------------+ 234 | | | | 235+------------+ | | | | 236| | | | | +--------------+ | 237| User +---------+ +------> | Internal API | | 238| | | | | +--------------+ | 239+------------+ | | | | 240 | | | | 241 | | +------------------------+ 242 | +----------+ | VMM 243 | CLI | | | 244 +----------->+ clap +--------------+ 245 | | 246 +----------+ 247 248 249``` 250 251## Internal API 252 253The Cloud Hypervisor internal API, as its name suggests, is used internally 254by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 255send commands and responses to each others. 256 257It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 258and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 259control loop. 260 261API producers are the HTTP thread handling the [REST API](#rest-api) and the 262main thread that initially parses the [CLI](#command-line-interface). 263 264### Goals and Design 265 266The internal API is designed for controlling, managing and inspecting a Cloud 267Hypervisor VMM and its guest. It is a backend for handling external, user 268visible requests through either the [REST API](#rest-api) or the 269[CLI](#command-line-interface) interfaces. 270 271The API follows a command-response scheme that closely maps the [REST API](#rest-api). 272Any command must be replied to with a response. 273 274Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 275are received and processed by the VMM control loop. 276 277In order for the VMM control loop to respond to any internal API command, it 278must be able to send a response back to the MPSC sender. For that purpose, all 279internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 280end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 281 282The sender of any internal API command is therefore responsible for: 283 2841. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 285 channel. 2861. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 287 end of the response channel as part of the internal API command payload. 2881. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 289 end of the response channel. 290 291## End to End Example 292 293In order to further understand how the external and internal Cloud Hypervisor 294APIs work together, let's look at a complete VM creation flow, from the 295[REST API](#rest-api) call, to the reply the external user will receive: 296 2971. A user or operator sends an HTTP request to the Cloud Hypervisor 298 [REST API](#rest-api) in order to creates a virtual machine: 299 ``` 300 shell 301 #!/bin/bash 302 303 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 304 -X PUT 'http://localhost/api/v1/vm.create' \ 305 -H 'Accept: application/json' \ 306 -H 'Content-Type: application/json' \ 307 -d '{ 308 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 309 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 310 "cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 311 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 312 "rng":{"src":"/dev/urandom"}, 313 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 314 }' 315 ``` 3161. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 317 HTTP request JSON body into an internal `VmConfig` structure. 3181. The Cloud Hypervisor HTTP thread creates an 319 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 320 server to send its response back. 3211. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 322 virtual machine. The command's payload is made of the de-serialized 323 `VmConfig` structure and the response channel: 324 ```Rust 325 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 326 ``` 3271. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 328 for the response: 329 ```Rust 330 // Send the VM creation request. 331 api_sender 332 .send(ApiRequest::VmCreate(config, response_sender)) 333 .map_err(ApiError::RequestSend)?; 334 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 335 336 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 337 ``` 3381. The Cloud Hypervisor control loop receives the command, as it listens on the 339 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 340 ```Rust 341 // Read from the API receiver channel 342 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 343 ``` 3441. The Cloud Hypervisor control loop matches the received internal API against 345 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 346 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 347 command payload. It stores the `VmConfig` structure and replies back to the 348 sender ((The HTTP thread): 349 ```Rust 350 match api_request { 351 ApiRequest::VmCreate(config, sender) => { 352 // We only store the passed VM config. 353 // The VM will be created when being asked to boot it. 354 let response = if self.vm_config.is_none() { 355 self.vm_config = Some(config); 356 Ok(ApiResponsePayload::Empty) 357 } else { 358 Err(ApiError::VmAlreadyCreated) 359 }; 360 361 sender.send(response).map_err(Error::ApiResponseSend)?; 362 } 363 ``` 3641. The Cloud Hypervisor HTTP thread receives the internal API command response 365 as the return value from its `VmCreate` HTTP handler. Depending on the 366 control loop internal API response, it generates the appropriate HTTP 367 response: 368 ```Rust 369 // Call vm_create() 370 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 371 .map_err(HttpError::VmCreate) 372 { 373 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 374 Err(e) => error_response(e, StatusCode::InternalServerError), 375 } 376 ``` 3771. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 378 user. This is abstracted by the 379 [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 380 crate. 381