1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 - [External API](#external-api) 3 - [REST API](#rest-api) 4 - [REST API Location and availability](#rest-api-location-and-availability) 5 - [REST API Endpoints](#rest-api-endpoints) 6 - [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 - [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 - [Create a Virtual Machine](#create-a-virtual-machine) 10 - [Boot a Virtual Machine](#boot-a-virtual-machine) 11 - [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 - [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 - [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 - [D-Bus API](#d-bus-api) 15 - [D-Bus API Location and availability](#d-bus-api-location-and-availability) 16 - [D-Bus API Interface](#d-bus-api-interface) 17 - [Command Line Interface](#command-line-interface) 18 - [REST API, D-Bus API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 19 - [Internal API](#internal-api) 20 - [Goals and Design](#goals-and-design) 21 - [End to End Example](#end-to-end-example) 22 23# Cloud Hypervisor API 24 25The Cloud Hypervisor API is made of 2 distinct interfaces: 26 271. **The External API** This is the user facing API. Users and operators 28 can control and manage the Cloud Hypervisor through various options 29 including a REST API, a Command Line Interface (CLI) or a D-Bus based API, 30 which is not compiled into Cloud Hypervisor by default. 31 321. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 33 module. This API is used internally by the Cloud Hypervisor threads to 34 communicate between each others. 35 36The goal of this document is to describe the Cloud Hypervisor API as a whole, 37and to outline how the internal and external APIs are architecturally related. 38 39## External API 40 41### REST API 42 43The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 44API triggers VM and VMM specific actions, and as such it is designed as a 45collection of RPC-style, static methods. 46 47The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 48compliant. Please consult the [Cloud Hypervisor OpenAPI Document](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 49for more details about the API payloads and responses. 50 51#### REST API Location and availability 52 53The REST API is available as soon as the Cloud Hypervisor binary is started, 54through a local UNIX socket. 55By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 56For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 57123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 58 59The REST API default URL can be overridden through the Cloud Hypervisor 60option `--api-socket`: 61 62``` 63$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 64Cloud Hypervisor Guest 65 API server: /tmp/cloud-hypervisor.sock 66 vCPUs: 1 67 Memory: 512 MB 68 Kernel: None 69 Kernel cmdline: 70 Disk(s): None 71``` 72 73#### REST API Endpoints 74 75The Cloud Hypervisor API exposes the following actions through its endpoints: 76 77##### Virtual Machine Manager (VMM) Actions 78 79| Action | Endpoint | Request Body | Response Body | Prerequisites | 80| ----------------------------------- | --------------- | ------------ | -------------------------- | ------------------ | 81| Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A | 82| Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running | 83 84##### Virtual Machine (VM) Actions 85 86| Action | Endpoint | Request Body | Response Body | Prerequisites | 87| ---------------------------------- | ----------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------ | 88| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | 89| Delete the VM | `/vm.delete` | N/A | N/A | N/A | 90| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | 91| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | 92| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | 93| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | 94| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | 95| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | 96| Task a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | 97| Perform a coredump of the VM* | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | 98| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | 99| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 100| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 101| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | 102| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | 103| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 104| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 105| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 106| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 107| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 108| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 109| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 110| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 111| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | 112| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | 113| Prepare to receive a migration | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A | N/A | 114| Start to send migration to target | `/vm.send-migration` | `/schemas/SendMigrationData` | N/A | The VM is booted and (shared mem or hugepages enabled) | 115 116* The `vmcoredump` action is available exclusively for the `x86_64` 117architecture and can be executed only when the `guest_debug` feature is 118enabled. Without this feature, the corresponding [REST API](#rest-api) or 119[D-Bus API](#d-bus-api) endpoints are not available. 120 121#### REST API Examples 122 123For the following set of examples, we assume Cloud Hypervisor is started with 124the REST API available at `/tmp/cloud-hypervisor.sock`: 125 126``` 127$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 128Cloud Hypervisor Guest 129 API server: /tmp/cloud-hypervisor.sock 130 vCPUs: 1 131 Memory: 512 MB 132 Kernel: None 133 Kernel cmdline: 134 Disk(s): None 135``` 136 137##### Create a Virtual Machine 138 139We want to create a virtual machine with the following characteristics: 140 141* 4 vCPUs 142* 1 GB of RAM 143* 1 virtio based networking interface 144* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 145 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 146* Using a Ubuntu image as its root filesystem, located at 147 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 148 149```shell 150#!/bin/bash 151 152curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 153 -X PUT 'http://localhost/api/v1/vm.create' \ 154 -H 'Accept: application/json' \ 155 -H 'Content-Type: application/json' \ 156 -d '{ 157 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 158 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 159 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 160 "rng":{"src":"/dev/urandom"}, 161 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 162 }' 163``` 164 165##### Boot a Virtual Machine 166 167Once the VM is created, we can boot it: 168 169```shell 170#!/bin/bash 171 172curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 173``` 174 175##### Dump a Virtual Machine Information 176 177We can fetch information about any VM, as soon as it's created: 178 179```shell 180#!/bin/bash 181 182curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 183 -X GET 'http://localhost/api/v1/vm.info' \ 184 -H 'Accept: application/json' 185``` 186 187##### Reboot a Virtual Machine 188 189We can reboot a VM that's already booted: 190 191```shell 192#!/bin/bash 193 194curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 195``` 196 197##### Shut a Virtual Machine Down 198 199Once booted, we can shut a VM down from the REST API: 200 201```shell 202#!/bin/bash 203 204curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 205``` 206 207### D-Bus API 208 209Cloud Hypervisor offers a D-Bus API as an alternative to its REST API. As of 210writing this document, the D-Bus API mirrors the functionality of the REST 211API and shares the same set of endpoints, meaning that it supports every call 212that is supported by the REST API and can be a drop-in replacement since it 213also consumes/produces JSON. 214 215#### D-Bus API Location and availability 216 217This feature is not compiled into Cloud Hypervisor by default. Users who 218wish to use the D-Bus API, must explicitly enable it with the `dbus_api` 219feature flag when compiling Cloud Hypervisor. 220 221```sh 222$ ./scripts/dev_cli.sh build --release --libc musl -- --features dbus_api 223``` 224 225Once this feature is enabled, it can be configured with the following 226CLI options: 227 228``` 229 --dbus-service-name 230 well known name of the service 231 --dbus-object-path 232 object path to serve the dbus interface 233 --dbus-system-bus use the system bus instead of a session bus 234``` 235 236Example invocation: 237 238```sh 239$ ./cloud-hypervisor --dbus-service-name "org.cloudhypervisor.DBusApi" \ 240 --dbus-object-path "/org/cloudhypervisor/DBusApi" 241``` 242 243This will start serving a service with the name `org.cloudhypervisor.DBusApi1` 244which in turn can be used to control and manage Cloud Hypervisor. 245 246#### D-Bus API Interface 247 248Please refer to the [REST API](#rest-api) documentation. As previously 249mentioned, the D-Bus API currently mirrors the behaviour of the REST API. 250 251### Command Line Interface 252 253The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 254the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 255the launched VM once they're up and running. 256 257If you want to inspect the VMM, or control the VM after launching Cloud 258Hypervisor from the CLI, you must use either the [REST API](#rest-api) 259or the [D-Bus API](#d-bus-api). 260 261From the CLI, one can: 262 2631. Create and boot a complete virtual machine by using the CLI options to build 264 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 265 options. As soon as the `cloud-hypervisor` binary is launched, contrary 266 to the [D-Bus API](#d-bus-api), the [REST API](#rest-api) is available 267 for controlling and managing the VM. The [D-Bus API](#d-bus-api) doesn't start 268 automatically and needs to be explicitly configured in order to be run. 2691. Start either the REST API, D-Bus API or both simultaneously without passing 270 any VM configuration options. The VM can then be asynchronously created and 271 booted by calling API methods of choice. It should be noted that one external 272 API does not exclude another; it is possible to have both the REST and D-Bus 273 APIs running simultaneously. 274 275### REST API, D-Bus API and CLI Architectural Relationship 276 277The REST API, D-Bus API and the CLI all rely on a common, [internal API](#internal-api). 278 279The CLI options are parsed by the 280[argh crate](https://docs.rs/argh/latest/argh/) and then translated into 281[internal API](#internal-api) commands. 282 283The REST API is processed by an HTTP thread using the 284[Firecracker's `micro_http`](https://github.com/firecracker-microvm/micro-http) 285crate. As with the CLI, the HTTP requests eventually get translated into 286[internal API](#internal-api) commands. 287 288The D-Bus API is implemented using the [zbus](https://github.com/dbus2/zbus) 289crate and runs in its own thread. Whenever it needs to call the [internal API](#internal-api), 290the [blocking](https://github.com/smol-rs/blocking) crate is used perform the call in zbus' async context. 291 292As a summary, the REST API, the D-Bus API and the CLI are essentially frontends for the 293[internal API](#internal-api): 294 295``` 296 +------------------+ 297 REST API | | 298 +--------->+ micro_http +--------+ 299 | | | | 300 | +------------------+ | 301 | | +------------------------+ 302 | | | | 303+------------+ | +----------+ | | | 304| | | D-Bus API | | | | +--------------+ | 305| User +---------+----------->+ zbus +--------------+------> | Internal API | | 306| | | | | | | +--------------+ | 307+------------+ | +----------+ | | | 308 | | | | 309 | | +------------------------+ 310 | +----------+ | VMM 311 | CLI | | | 312 +----------->+ argh +--------------+ 313 | | 314 +----------+ 315 316 317``` 318 319## Internal API 320 321The Cloud Hypervisor internal API, as its name suggests, is used internally 322by the different Cloud Hypervisor threads (VMM, HTTP, D-Bus, control loop, 323etc) to send commands and responses to each others. 324 325It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 326and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 327control loop. 328 329API producers are the HTTP thread handling the [REST API](#rest-api), the 330D-Bus thread handling the [D-Bus API](#d-bus-api) and the main thread that 331initially parses the [CLI](#command-line-interface). 332 333### Goals and Design 334 335The internal API is designed for controlling, managing and inspecting a Cloud 336Hypervisor VMM and its guest. It is a backend for handling external, user 337visible requests through the [REST API](#rest-api), the [D-Bus API](#d-bus-api) 338or the [CLI](#command-line-interface) interfaces. 339 340The API follows a command-response scheme that closely maps the [REST API](#rest-api). 341Any command must be replied to with a response. 342 343Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 344are received and processed by the VMM control loop. 345 346In order for the VMM control loop to respond to any internal API command, it 347must be able to send a response back to the MPSC sender. For that purpose, all 348internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 349end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 350 351The sender of any internal API command is therefore responsible for: 352 3531. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 354 channel. 3551. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 356 end of the response channel as part of the internal API command payload. 3571. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 358 end of the response channel. 359 360## End to End Example 361 362In order to further understand how the external and internal Cloud Hypervisor 363APIs work together, let's look at a complete VM creation flow, from the 364[REST API](#rest-api) call, to the reply the external user will receive: 365 3661. A user or operator sends an HTTP request to the Cloud Hypervisor 367 [REST API](#rest-api) in order to creates a virtual machine: 368 ``` 369 shell 370 #!/bin/bash 371 372 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 373 -X PUT 'http://localhost/api/v1/vm.create' \ 374 -H 'Accept: application/json' \ 375 -H 'Content-Type: application/json' \ 376 -d '{ 377 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 378 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 379 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 380 "rng":{"src":"/dev/urandom"}, 381 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 382 }' 383 ``` 3841. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 385 HTTP request JSON body into an internal `VmConfig` structure. 3861. The Cloud Hypervisor HTTP thread creates an 387 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 388 server to send its response back. 3891. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 390 virtual machine. The command's payload is made of the de-serialized 391 `VmConfig` structure and the response channel: 392 ```Rust 393 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 394 ``` 3951. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 396 for the response: 397 ```Rust 398 // Send the VM creation request. 399 api_sender 400 .send(ApiRequest::VmCreate(config, response_sender)) 401 .map_err(ApiError::RequestSend)?; 402 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 403 404 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 405 ``` 4061. The Cloud Hypervisor control loop receives the command, as it listens on the 407 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 408 ```Rust 409 // Read from the API receiver channel 410 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 411 ``` 4121. The Cloud Hypervisor control loop matches the received internal API against 413 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 414 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 415 command payload. It stores the `VmConfig` structure and replies back to the 416 sender ((The HTTP thread): 417 ```Rust 418 match api_request { 419 ApiRequest::VmCreate(config, sender) => { 420 // We only store the passed VM config. 421 // The VM will be created when being asked to boot it. 422 let response = if self.vm_config.is_none() { 423 self.vm_config = Some(config); 424 Ok(ApiResponsePayload::Empty) 425 } else { 426 Err(ApiError::VmAlreadyCreated) 427 }; 428 429 sender.send(response).map_err(Error::ApiResponseSend)?; 430 } 431 ``` 4321. The Cloud Hypervisor HTTP thread receives the internal API command response 433 as the return value from its `VmCreate` HTTP handler. Depending on the 434 control loop internal API response, it generates the appropriate HTTP 435 response: 436 ```Rust 437 // Call vm_create() 438 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 439 .map_err(HttpError::VmCreate) 440 { 441 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 442 Err(e) => error_response(e, StatusCode::InternalServerError), 443 } 444 ``` 4451. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 446 user. This is abstracted by the 447 [micro_http](https://github.com/firecracker-microvm/micro-http) 448 crate. 449