1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 - [External API](#external-api) 3 - [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 - [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 - [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 - [Create a Virtual Machine](#create-a-virtual-machine) 10 - [Boot a Virtual Machine](#boot-a-virtual-machine) 11 - [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 - [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 - [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 - [Command Line Interface](#command-line-interface) 15 - [REST API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 16 - [Internal API](#internal-api) 17 - [Goals and Design](#goals-and-design) 18 - [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74| Action | Endpoint | Request Body | Response Body | Prerequisites | 75| ----------------------------------- | --------------- | ------------ | -------------------------- | ------------------ | 76| Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A | 77| Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running | 78 79#### Virtual Machine (VM) Actions 80 81| Action | Endpoint | Request Body | Response Body | Prerequisites | 82| ---------------------------------- | ----------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------ | 83| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | 84| Delete the VM | `/vm.delete` | N/A | N/A | N/A | 85| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | 86| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | 87| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | 88| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | 89| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | 90| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | 91| Task a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | 92| Perform a coredump of the VM* | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | 93| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | 94| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 95| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 96| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | 97| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | 98| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 99| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 100| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 101| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 102| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 103| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 104| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 105| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 106| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | 107| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | 108| Prepare to receive a migration | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A | N/A | 109| Start to send migration to target | `/vm.send-migration` | `/schemas/SendMigrationData` | N/A | The VM is booted and (shared mem or hugepages enabled) | 110 111* The `vmcoredump` action is available exclusively for the `x86_64` 112architecture and can be executed only when the `guest_debug` feature is 113enabled. Without this feature, the corresponding REST API endpoint is not 114available. 115 116### REST API Examples 117 118For the following set of examples, we assume Cloud Hypervisor is started with 119the REST API available at `/tmp/cloud-hypervisor.sock`: 120 121``` 122$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 123Cloud Hypervisor Guest 124 API server: /tmp/cloud-hypervisor.sock 125 vCPUs: 1 126 Memory: 512 MB 127 Kernel: None 128 Kernel cmdline: 129 Disk(s): None 130``` 131 132#### Create a Virtual Machine 133 134We want to create a virtual machine with the following characteristics: 135 136* 4 vCPUs 137* 1 GB of RAM 138* 1 virtio based networking interface 139* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 140 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 141* Using a Ubuntu image as its root filesystem, located at 142 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 143 144```shell 145#!/bin/bash 146 147curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 148 -X PUT 'http://localhost/api/v1/vm.create' \ 149 -H 'Accept: application/json' \ 150 -H 'Content-Type: application/json' \ 151 -d '{ 152 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 153 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 154 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 155 "rng":{"src":"/dev/urandom"}, 156 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 157 }' 158``` 159 160#### Boot a Virtual Machine 161 162Once the VM is created, we can boot it: 163 164```shell 165#!/bin/bash 166 167curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 168``` 169 170#### Dump a Virtual Machine Information 171 172We can fetch information about any VM, as soon as it's created: 173 174```shell 175#!/bin/bash 176 177curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 178 -X GET 'http://localhost/api/v1/vm.info' \ 179 -H 'Accept: application/json' 180``` 181 182#### Reboot a Virtual Machine 183 184We can reboot a VM that's already booted: 185 186```shell 187#!/bin/bash 188 189curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 190``` 191 192#### Shut a Virtual Machine Down 193 194Once booted, we can shut a VM down from the REST API: 195 196```shell 197#!/bin/bash 198 199curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 200``` 201 202### Command Line Interface 203 204The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 205the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 206the launched VM once they're up and running. 207 208If you want to inspect the VMM, or control the VM after launching Cloud 209Hypervisor from the CLI, you must use the [REST API](#rest-api). 210 211From the CLI, one can either: 212 2131. Create and boot a complete virtual machine by using the CLI options to build 214 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 215 options. As soon as the `cloud-hypervisor` binary is launched, the 216 [REST API](#rest-api) is available for controlling and managing the VM. 2171. Start the [REST API](#rest-api) server only, by not passing any VM 218 configuration options. The VM can then be asynchronously created and booted 219 by sending HTTP commands to the [REST API](#rest-api). Check the 220 [REST API examples](#rest-api-examples) section for more details. 221 222### REST API and CLI Architectural Relationship 223 224The REST API and the CLI both rely on a common, [internal API](#internal-api). 225 226The CLI options are parsed by the 227[argh crate](https://docs.rs/argh/latest/argh/) and then translated into 228[internal API](#internal-api) commands. 229 230The REST API is processed by an HTTP thread using the 231[Firecracker's `micro_http`](https://github.com/firecracker-microvm/micro-http) 232crate. As with the CLI, the HTTP requests eventually get translated into 233[internal API](#internal-api) commands. 234 235As a summary, the REST API and the CLI are essentially frontends for the 236[internal API](#internal-api): 237 238``` 239 +------------------+ 240 REST API | | 241 +--------->+ micro_http +--------+ 242 | | | | 243 | +------------------+ | 244 | | +------------------------+ 245 | | | | 246+------------+ | | | | 247| | | | | +--------------+ | 248| User +---------+ +------> | Internal API | | 249| | | | | +--------------+ | 250+------------+ | | | | 251 | | | | 252 | | +------------------------+ 253 | +----------+ | VMM 254 | CLI | | | 255 +----------->+ argh +--------------+ 256 | | 257 +----------+ 258 259 260``` 261 262## Internal API 263 264The Cloud Hypervisor internal API, as its name suggests, is used internally 265by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 266send commands and responses to each others. 267 268It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 269and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 270control loop. 271 272API producers are the HTTP thread handling the [REST API](#rest-api) and the 273main thread that initially parses the [CLI](#command-line-interface). 274 275### Goals and Design 276 277The internal API is designed for controlling, managing and inspecting a Cloud 278Hypervisor VMM and its guest. It is a backend for handling external, user 279visible requests through either the [REST API](#rest-api) or the 280[CLI](#command-line-interface) interfaces. 281 282The API follows a command-response scheme that closely maps the [REST API](#rest-api). 283Any command must be replied to with a response. 284 285Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 286are received and processed by the VMM control loop. 287 288In order for the VMM control loop to respond to any internal API command, it 289must be able to send a response back to the MPSC sender. For that purpose, all 290internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 291end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 292 293The sender of any internal API command is therefore responsible for: 294 2951. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 296 channel. 2971. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 298 end of the response channel as part of the internal API command payload. 2991. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 300 end of the response channel. 301 302## End to End Example 303 304In order to further understand how the external and internal Cloud Hypervisor 305APIs work together, let's look at a complete VM creation flow, from the 306[REST API](#rest-api) call, to the reply the external user will receive: 307 3081. A user or operator sends an HTTP request to the Cloud Hypervisor 309 [REST API](#rest-api) in order to creates a virtual machine: 310 ``` 311 shell 312 #!/bin/bash 313 314 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 315 -X PUT 'http://localhost/api/v1/vm.create' \ 316 -H 'Accept: application/json' \ 317 -H 'Content-Type: application/json' \ 318 -d '{ 319 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 320 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 321 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 322 "rng":{"src":"/dev/urandom"}, 323 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 324 }' 325 ``` 3261. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 327 HTTP request JSON body into an internal `VmConfig` structure. 3281. The Cloud Hypervisor HTTP thread creates an 329 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 330 server to send its response back. 3311. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 332 virtual machine. The command's payload is made of the de-serialized 333 `VmConfig` structure and the response channel: 334 ```Rust 335 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 336 ``` 3371. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 338 for the response: 339 ```Rust 340 // Send the VM creation request. 341 api_sender 342 .send(ApiRequest::VmCreate(config, response_sender)) 343 .map_err(ApiError::RequestSend)?; 344 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 345 346 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 347 ``` 3481. The Cloud Hypervisor control loop receives the command, as it listens on the 349 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 350 ```Rust 351 // Read from the API receiver channel 352 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 353 ``` 3541. The Cloud Hypervisor control loop matches the received internal API against 355 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 356 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 357 command payload. It stores the `VmConfig` structure and replies back to the 358 sender ((The HTTP thread): 359 ```Rust 360 match api_request { 361 ApiRequest::VmCreate(config, sender) => { 362 // We only store the passed VM config. 363 // The VM will be created when being asked to boot it. 364 let response = if self.vm_config.is_none() { 365 self.vm_config = Some(config); 366 Ok(ApiResponsePayload::Empty) 367 } else { 368 Err(ApiError::VmAlreadyCreated) 369 }; 370 371 sender.send(response).map_err(Error::ApiResponseSend)?; 372 } 373 ``` 3741. The Cloud Hypervisor HTTP thread receives the internal API command response 375 as the return value from its `VmCreate` HTTP handler. Depending on the 376 control loop internal API response, it generates the appropriate HTTP 377 response: 378 ```Rust 379 // Call vm_create() 380 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 381 .map_err(HttpError::VmCreate) 382 { 383 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 384 Err(e) => error_response(e, StatusCode::InternalServerError), 385 } 386 ``` 3871. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 388 user. This is abstracted by the 389 [micro_http](https://github.com/firecracker-microvm/micro-http) 390 crate. 391