1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 - [External API](#external-api) 3 - [REST API](#rest-api) 4 - [REST API Location and availability](#rest-api-location-and-availability) 5 - [REST API Endpoints](#rest-api-endpoints) 6 - [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 - [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 - [Create a Virtual Machine](#create-a-virtual-machine) 10 - [Boot a Virtual Machine](#boot-a-virtual-machine) 11 - [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 - [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 - [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 - [D-Bus API](#d-bus-api) 15 - [D-Bus API Location and availability](#d-bus-api-location-and-availability) 16 - [D-Bus API Interface](#d-bus-api-interface) 17 - [Command Line Interface](#command-line-interface) 18 - [REST API, D-Bus API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 19 - [Internal API](#internal-api) 20 - [Goals and Design](#goals-and-design) 21 - [End to End Example](#end-to-end-example) 22 23# Cloud Hypervisor API 24 25The Cloud Hypervisor API is made of 2 distinct interfaces: 26 271. **The External API** This is the user facing API. Users and operators 28 can control and manage the Cloud Hypervisor through various options 29 including a REST API, a Command Line Interface (CLI) or a D-Bus based API, 30 which is not compiled into Cloud Hypervisor by default. 31 321. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 33 module. This API is used internally by the Cloud Hypervisor threads to 34 communicate between each others. 35 36The goal of this document is to describe the Cloud Hypervisor API as a whole, 37and to outline how the internal and external APIs are architecturally related. 38 39## External API 40 41### REST API 42 43The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 44API triggers VM and VMM specific actions, and as such it is designed as a 45collection of RPC-style, static methods. 46 47The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 48compliant. Please consult the [Cloud Hypervisor OpenAPI Document](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 49for more details about the API payloads and responses. 50 51#### REST API Location and availability 52 53The REST API is available as soon as the Cloud Hypervisor binary is started, 54through a local UNIX socket. 55By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 56For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 57123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 58 59The REST API default URL can be overridden through the Cloud Hypervisor 60option `--api-socket`: 61 62``` 63$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 64Cloud Hypervisor Guest 65 API server: /tmp/cloud-hypervisor.sock 66 vCPUs: 1 67 Memory: 512 MB 68 Kernel: None 69 Kernel cmdline: 70 Disk(s): None 71``` 72 73#### REST API Endpoints 74 75The Cloud Hypervisor API exposes the following actions through its endpoints: 76 77##### Virtual Machine Manager (VMM) Actions 78 79| Action | Endpoint | Request Body | Response Body | Prerequisites | 80| ----------------------------------- | --------------- | ------------ | -------------------------- | ------------------ | 81| Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A | 82| Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running | 83 84##### Virtual Machine (VM) Actions 85 86| Action | Endpoint | Request Body | Response Body | Prerequisites | 87| ---------------------------------- | ----------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------ | 88| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | 89| Delete the VM | `/vm.delete` | N/A | N/A | N/A | 90| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | 91| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | 92| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | 93| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | 94| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | 95| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | 96| Task a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | 97| Perform a coredump of the VM* | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | 98| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | 99| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 100| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 101| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | 102| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | 103| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 104| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 105| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 106| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 107| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 108| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 109| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 110| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 111| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | 112| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | 113| Inject an NMI | `/vm.nmi` | N/A | N/A | The VM is booted | 114| Prepare to receive a migration | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A | N/A | 115| Start to send migration to target | `/vm.send-migration` | `/schemas/SendMigrationData` | N/A | The VM is booted and (shared mem or hugepages enabled) | 116 117* The `vmcoredump` action is available exclusively for the `x86_64` 118architecture and can be executed only when the `guest_debug` feature is 119enabled. Without this feature, the corresponding [REST API](#rest-api) or 120[D-Bus API](#d-bus-api) endpoints are not available. 121 122#### REST API Examples 123 124For the following set of examples, we assume Cloud Hypervisor is started with 125the REST API available at `/tmp/cloud-hypervisor.sock`: 126 127``` 128$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 129Cloud Hypervisor Guest 130 API server: /tmp/cloud-hypervisor.sock 131 vCPUs: 1 132 Memory: 512 MB 133 Kernel: None 134 Kernel cmdline: 135 Disk(s): None 136``` 137 138##### Create a Virtual Machine 139 140We want to create a virtual machine with the following characteristics: 141 142* 4 vCPUs 143* 1 GB of RAM 144* 1 virtio based networking interface 145* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 146 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 147* Using a Ubuntu image as its root filesystem, located at 148 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 149 150```shell 151#!/usr/bin/env bash 152 153curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 154 -X PUT 'http://localhost/api/v1/vm.create' \ 155 -H 'Accept: application/json' \ 156 -H 'Content-Type: application/json' \ 157 -d '{ 158 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 159 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 160 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 161 "rng":{"src":"/dev/urandom"}, 162 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 163 }' 164``` 165 166##### Boot a Virtual Machine 167 168Once the VM is created, we can boot it: 169 170```shell 171#!/usr/bin/env bash 172 173curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 174``` 175 176##### Dump a Virtual Machine Information 177 178We can fetch information about any VM, as soon as it's created: 179 180```shell 181#!/usr/bin/env bash 182 183curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 184 -X GET 'http://localhost/api/v1/vm.info' \ 185 -H 'Accept: application/json' 186``` 187 188##### Reboot a Virtual Machine 189 190We can reboot a VM that's already booted: 191 192```shell 193#!/usr/bin/env bash 194 195curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 196``` 197 198##### Shut a Virtual Machine Down 199 200Once booted, we can shut a VM down from the REST API: 201 202```shell 203#!/usr/bin/env bash 204 205curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 206``` 207 208### D-Bus API 209 210Cloud Hypervisor offers a D-Bus API as an alternative to its REST API. This 211D-Bus API fully reflects the functionality of the REST API, exposing the 212same group of endpoints. It can be a drop-in replacement since it also 213consumes/produces JSON. 214 215In addition, the D-Bus API also exposes events from `event-monitor` in the 216form of a D-Bus signal to which users can subscribe. For more information, 217see [D-Bus API Interface](#d-bus-api-interface). 218 219#### D-Bus API Location and availability 220 221This feature is not compiled into Cloud Hypervisor by default. Users who 222wish to use the D-Bus API, must explicitly enable it with the `dbus_api` 223feature flag when compiling Cloud Hypervisor. 224 225```sh 226$ ./scripts/dev_cli.sh build --release --libc musl -- --features dbus_api 227``` 228 229Once this feature is enabled, it can be configured with the following 230CLI options: 231 232``` 233 --dbus-service-name 234 well known name of the service 235 --dbus-object-path 236 object path to serve the dbus interface 237 --dbus-system-bus use the system bus instead of a session bus 238``` 239 240Example invocation: 241 242```sh 243$ ./cloud-hypervisor --dbus-service-name "org.cloudhypervisor.DBusApi" \ 244 --dbus-object-path "/org/cloudhypervisor/DBusApi" 245``` 246 247This will start serving a service with the name `org.cloudhypervisor.DBusApi1` 248which in turn can be used to control and manage Cloud Hypervisor. 249 250#### D-Bus API Interface 251 252Please refer to the [REST API](#rest-api) documentation for everything that 253is in common with the REST API. As previously mentioned, the D-Bus API can 254be used as a drop-in replacement for the [REST API](#rest-api). 255 256The D-Bus interface also exposes a signal, named `Event`, which is emitted 257whenever a new event is published from the `event-monitor` crate. Here is its 258definition in XML format: 259 260```xml 261<node> 262 <interface name="org.cloudhypervisor.DBusApi1"> 263 <signal name="Event"> 264 <arg name="event" type="s"/> 265 </signal> 266 </interface> 267</node> 268``` 269 270### Command Line Interface 271 272The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 273the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 274the launched VM once they're up and running. 275 276If you want to inspect the VMM, or control the VM after launching Cloud 277Hypervisor from the CLI, you must use either the [REST API](#rest-api) 278or the [D-Bus API](#d-bus-api). 279 280From the CLI, one can: 281 2821. Create and boot a complete virtual machine by using the CLI options to build 283 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 284 options. As soon as the `cloud-hypervisor` binary is launched, contrary 285 to the [D-Bus API](#d-bus-api), the [REST API](#rest-api) is available 286 for controlling and managing the VM. The [D-Bus API](#d-bus-api) doesn't start 287 automatically and needs to be explicitly configured in order to be run. 2881. Start either the REST API, D-Bus API or both simultaneously without passing 289 any VM configuration options. The VM can then be asynchronously created and 290 booted by calling API methods of choice. It should be noted that one external 291 API does not exclude another; it is possible to have both the REST and D-Bus 292 APIs running simultaneously. 293 294### REST API, D-Bus API and CLI Architectural Relationship 295 296The REST API, D-Bus API and the CLI all rely on a common, [internal API](#internal-api). 297 298The CLI options are parsed by the 299[clap crate](https://docs.rs/clap/4.3.11/clap/) and then translated into 300[internal API](#internal-api) commands. 301 302The REST API is processed by an HTTP thread using the 303[Firecracker's `micro_http`](https://github.com/firecracker-microvm/micro-http) 304crate. As with the CLI, the HTTP requests eventually get translated into 305[internal API](#internal-api) commands. 306 307The D-Bus API is implemented using the [zbus](https://github.com/dbus2/zbus) 308crate and runs in its own thread. Whenever it needs to call the [internal API](#internal-api), 309the [blocking](https://github.com/smol-rs/blocking) crate is used perform the call in zbus' async context. 310 311As a summary, the REST API, the D-Bus API and the CLI are essentially frontends for the 312[internal API](#internal-api): 313 314``` 315 +------------------+ 316 REST API | | 317 +--------->+ micro_http +--------+ 318 | | | | 319 | +------------------+ | 320 | | +------------------------+ 321 | | | | 322+------------+ | +----------+ | | | 323| | | D-Bus API | | | | +--------------+ | 324| User +---------+----------->+ zbus +--------------+------> | Internal API | | 325| | | | | | | +--------------+ | 326+------------+ | +----------+ | | | 327 | | | | 328 | | +------------------------+ 329 | +----------+ | VMM 330 | CLI | | | 331 +----------->+ clap +--------------+ 332 | | 333 +----------+ 334 335 336``` 337 338## Internal API 339 340The Cloud Hypervisor internal API, as its name suggests, is used internally 341by the different Cloud Hypervisor threads (VMM, HTTP, D-Bus, control loop, 342etc) to send commands and responses to each others. 343 344It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 345and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 346control loop. 347 348API producers are the HTTP thread handling the [REST API](#rest-api), the 349D-Bus thread handling the [D-Bus API](#d-bus-api) and the main thread that 350initially parses the [CLI](#command-line-interface). 351 352### Goals and Design 353 354The internal API is designed for controlling, managing and inspecting a Cloud 355Hypervisor VMM and its guest. It is a backend for handling external, user 356visible requests through the [REST API](#rest-api), the [D-Bus API](#d-bus-api) 357or the [CLI](#command-line-interface) interfaces. 358 359The API follows a command-response scheme that closely maps the [REST API](#rest-api). 360Any command must be replied to with a response. 361 362Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 363are received and processed by the VMM control loop. 364 365In order for the VMM control loop to respond to any internal API command, it 366must be able to send a response back to the MPSC sender. For that purpose, all 367internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 368end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 369 370The sender of any internal API command is therefore responsible for: 371 3721. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 373 channel. 3741. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 375 end of the response channel as part of the internal API command payload. 3761. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 377 end of the response channel. 378 379## End to End Example 380 381In order to further understand how the external and internal Cloud Hypervisor 382APIs work together, let's look at a complete VM creation flow, from the 383[REST API](#rest-api) call, to the reply the external user will receive: 384 3851. A user or operator sends an HTTP request to the Cloud Hypervisor 386 [REST API](#rest-api) in order to creates a virtual machine: 387 ``` 388 shell 389 #!/usr/bin/env bash 390 391 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 392 -X PUT 'http://localhost/api/v1/vm.create' \ 393 -H 'Accept: application/json' \ 394 -H 'Content-Type: application/json' \ 395 -d '{ 396 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 397 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 398 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 399 "rng":{"src":"/dev/urandom"}, 400 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 401 }' 402 ``` 4031. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 404 HTTP request JSON body into an internal `VmConfig` structure. 4051. The Cloud Hypervisor HTTP thread creates an 406 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 407 server to send its response back. 4081. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 409 virtual machine. The command's payload is made of the de-serialized 410 `VmConfig` structure and the response channel: 411 ```Rust 412 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 413 ``` 4141. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 415 for the response: 416 ```Rust 417 // Send the VM creation request. 418 api_sender 419 .send(ApiRequest::VmCreate(config, response_sender)) 420 .map_err(ApiError::RequestSend)?; 421 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 422 423 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 424 ``` 4251. The Cloud Hypervisor control loop receives the command, as it listens on the 426 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 427 ```Rust 428 // Read from the API receiver channel 429 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 430 ``` 4311. The Cloud Hypervisor control loop matches the received internal API against 432 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 433 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 434 command payload. It stores the `VmConfig` structure and replies back to the 435 sender ((The HTTP thread): 436 ```Rust 437 match api_request { 438 ApiRequest::VmCreate(config, sender) => { 439 // We only store the passed VM config. 440 // The VM will be created when being asked to boot it. 441 let response = if self.vm_config.is_none() { 442 self.vm_config = Some(config); 443 Ok(ApiResponsePayload::Empty) 444 } else { 445 Err(ApiError::VmAlreadyCreated) 446 }; 447 448 sender.send(response).map_err(Error::ApiResponseSend)?; 449 } 450 ``` 4511. The Cloud Hypervisor HTTP thread receives the internal API command response 452 as the return value from its `VmCreate` HTTP handler. Depending on the 453 control loop internal API response, it generates the appropriate HTTP 454 response: 455 ```Rust 456 // Call vm_create() 457 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 458 .map_err(HttpError::VmCreate) 459 { 460 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 461 Err(e) => error_response(e, StatusCode::InternalServerError), 462 } 463 ``` 4641. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 465 user. This is abstracted by the 466 [micro_http](https://github.com/firecracker-microvm/micro-http) 467 crate. 468