1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 - [External API](#external-api) 3 - [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 - [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 - [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 - [Create a Virtual Machine](#create-a-virtual-machine) 10 - [Boot a Virtual Machine](#boot-a-virtual-machine) 11 - [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 - [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 - [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 - [Command Line Interface](#command-line-interface) 15 - [REST API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 16 - [Internal API](#internal-api) 17 - [Goals and Design](#goals-and-design) 18 - [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74| Action | Endpoint | Request Body | Response Body | Prerequisites | 75| ----------------------------------- | --------------- | ------------ | -------------------------- | ------------------ | 76| Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A | 77| Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running | 78 79#### Virtual Machine (VM) Actions 80 81| Action | Endpoint | Request Body | Response Body | Prerequisites | 82| ---------------------------------- | --------------------- | --------------------------- | ------------------------ | -------------------------------- | 83| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | 84| Delete the VM | `/vm.delete` | N/A | N/A | N/A | 85| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | 86| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | 87| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | 88| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | 89| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | 90| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | 91| Task a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | 92| Perform a coredump of the VM | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | 93| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | 94| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 95| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 96| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | 97| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | 98| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 99| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 100| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 101| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 102| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 103| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 104| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 105| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 106| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | 107| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | 108 109### REST API Examples 110 111For the following set of examples, we assume Cloud Hypervisor is started with 112the REST API available at `/tmp/cloud-hypervisor.sock`: 113 114``` 115$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 116Cloud Hypervisor Guest 117 API server: /tmp/cloud-hypervisor.sock 118 vCPUs: 1 119 Memory: 512 MB 120 Kernel: None 121 Kernel cmdline: 122 Disk(s): None 123``` 124 125#### Create a Virtual Machine 126 127We want to create a virtual machine with the following characteristics: 128 129* 4 vCPUs 130* 1 GB of RAM 131* 1 virtio based networking interface 132* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 133 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 134* Using a Ubuntu image as its root filesystem, located at 135 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 136 137```shell 138#!/bin/bash 139 140curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 141 -X PUT 'http://localhost/api/v1/vm.create' \ 142 -H 'Accept: application/json' \ 143 -H 'Content-Type: application/json' \ 144 -d '{ 145 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 146 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 147 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 148 "rng":{"src":"/dev/urandom"}, 149 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 150 }' 151``` 152 153#### Boot a Virtual Machine 154 155Once the VM is created, we can boot it: 156 157```shell 158#!/bin/bash 159 160curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 161``` 162 163#### Dump a Virtual Machine Information 164 165We can fetch information about any VM, as soon as it's created: 166 167```shell 168#!/bin/bash 169 170curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 171 -X GET 'http://localhost/api/v1/vm.info' \ 172 -H 'Accept: application/json' 173``` 174 175#### Reboot a Virtual Machine 176 177We can reboot a VM that's already booted: 178 179```shell 180#!/bin/bash 181 182curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 183``` 184 185#### Shut a Virtual Machine Down 186 187Once booted, we can shut a VM down from the REST API: 188 189```shell 190#!/bin/bash 191 192curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 193``` 194 195### Command Line Interface 196 197The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 198the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 199the launched VM once they're up and running. 200 201If you want to inspect the VMM, or control the VM after launching Cloud 202Hypervisor from the CLI, you must use the [REST API](#rest-api). 203 204From the CLI, one can either: 205 2061. Create and boot a complete virtual machine by using the CLI options to build 207 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 208 options. As soon as the `cloud-hypervisor` binary is launched, the 209 [REST API](#rest-api) is available for controlling and managing the VM. 2101. Start the [REST API](#rest-api) server only, by not passing any VM 211 configuration options. The VM can then be asynchronously created and booted 212 by sending HTTP commands to the [REST API](#rest-api). Check the 213 [REST API examples](#rest-api-examples) section for more details. 214 215### REST API and CLI Architectural Relationship 216 217The REST API and the CLI both rely on a common, [internal API](#internal-api). 218 219The CLI options are parsed by the 220[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into 221[internal API](#internal-api) commands. 222 223The REST API is processed by an HTTP thread using the 224[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 225crate. As with the CLI, the HTTP requests eventually get translated into 226[internal API](#internal-api) commands. 227 228As a summary, the REST API and the CLI are essentially frontends for the 229[internal API](#internal-api): 230 231``` 232 +------------------+ 233 REST API | | 234 +--------->+ micro_http +--------+ 235 | | | | 236 | +------------------+ | 237 | | +------------------------+ 238 | | | | 239+------------+ | | | | 240| | | | | +--------------+ | 241| User +---------+ +------> | Internal API | | 242| | | | | +--------------+ | 243+------------+ | | | | 244 | | | | 245 | | +------------------------+ 246 | +----------+ | VMM 247 | CLI | | | 248 +----------->+ clap +--------------+ 249 | | 250 +----------+ 251 252 253``` 254 255## Internal API 256 257The Cloud Hypervisor internal API, as its name suggests, is used internally 258by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 259send commands and responses to each others. 260 261It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 262and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 263control loop. 264 265API producers are the HTTP thread handling the [REST API](#rest-api) and the 266main thread that initially parses the [CLI](#command-line-interface). 267 268### Goals and Design 269 270The internal API is designed for controlling, managing and inspecting a Cloud 271Hypervisor VMM and its guest. It is a backend for handling external, user 272visible requests through either the [REST API](#rest-api) or the 273[CLI](#command-line-interface) interfaces. 274 275The API follows a command-response scheme that closely maps the [REST API](#rest-api). 276Any command must be replied to with a response. 277 278Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 279are received and processed by the VMM control loop. 280 281In order for the VMM control loop to respond to any internal API command, it 282must be able to send a response back to the MPSC sender. For that purpose, all 283internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 284end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 285 286The sender of any internal API command is therefore responsible for: 287 2881. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 289 channel. 2901. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 291 end of the response channel as part of the internal API command payload. 2921. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 293 end of the response channel. 294 295## End to End Example 296 297In order to further understand how the external and internal Cloud Hypervisor 298APIs work together, let's look at a complete VM creation flow, from the 299[REST API](#rest-api) call, to the reply the external user will receive: 300 3011. A user or operator sends an HTTP request to the Cloud Hypervisor 302 [REST API](#rest-api) in order to creates a virtual machine: 303 ``` 304 shell 305 #!/bin/bash 306 307 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 308 -X PUT 'http://localhost/api/v1/vm.create' \ 309 -H 'Accept: application/json' \ 310 -H 'Content-Type: application/json' \ 311 -d '{ 312 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 313 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 314 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 315 "rng":{"src":"/dev/urandom"}, 316 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 317 }' 318 ``` 3191. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 320 HTTP request JSON body into an internal `VmConfig` structure. 3211. The Cloud Hypervisor HTTP thread creates an 322 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 323 server to send its response back. 3241. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 325 virtual machine. The command's payload is made of the de-serialized 326 `VmConfig` structure and the response channel: 327 ```Rust 328 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 329 ``` 3301. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 331 for the response: 332 ```Rust 333 // Send the VM creation request. 334 api_sender 335 .send(ApiRequest::VmCreate(config, response_sender)) 336 .map_err(ApiError::RequestSend)?; 337 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 338 339 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 340 ``` 3411. The Cloud Hypervisor control loop receives the command, as it listens on the 342 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 343 ```Rust 344 // Read from the API receiver channel 345 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 346 ``` 3471. The Cloud Hypervisor control loop matches the received internal API against 348 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 349 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 350 command payload. It stores the `VmConfig` structure and replies back to the 351 sender ((The HTTP thread): 352 ```Rust 353 match api_request { 354 ApiRequest::VmCreate(config, sender) => { 355 // We only store the passed VM config. 356 // The VM will be created when being asked to boot it. 357 let response = if self.vm_config.is_none() { 358 self.vm_config = Some(config); 359 Ok(ApiResponsePayload::Empty) 360 } else { 361 Err(ApiError::VmAlreadyCreated) 362 }; 363 364 sender.send(response).map_err(Error::ApiResponseSend)?; 365 } 366 ``` 3671. The Cloud Hypervisor HTTP thread receives the internal API command response 368 as the return value from its `VmCreate` HTTP handler. Depending on the 369 control loop internal API response, it generates the appropriate HTTP 370 response: 371 ```Rust 372 // Call vm_create() 373 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 374 .map_err(HttpError::VmCreate) 375 { 376 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 377 Err(e) => error_response(e, StatusCode::InternalServerError), 378 } 379 ``` 3801. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 381 user. This is abstracted by the 382 [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 383 crate. 384