1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 - [External API](#external-api) 3 - [REST API](#rest-api) 4 - [REST API Location and availability](#rest-api-location-and-availability) 5 - [REST API Endpoints](#rest-api-endpoints) 6 - [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 - [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 - [Create a Virtual Machine](#create-a-virtual-machine) 10 - [Boot a Virtual Machine](#boot-a-virtual-machine) 11 - [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 - [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 - [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 - [D-Bus API](#d-bus-api) 15 - [D-Bus API Location and availability](#d-bus-api-location-and-availability) 16 - [D-Bus API Interface](#d-bus-api-interface) 17 - [Command Line Interface](#command-line-interface) 18 - [REST API, D-Bus API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 19 - [Internal API](#internal-api) 20 - [Goals and Design](#goals-and-design) 21 - [End to End Example](#end-to-end-example) 22 23# Cloud Hypervisor API 24 25The Cloud Hypervisor API is made of 2 distinct interfaces: 26 271. **The External API** This is the user facing API. Users and operators 28 can control and manage the Cloud Hypervisor through various options 29 including a REST API, a Command Line Interface (CLI) or a D-Bus based API, 30 which is not compiled into Cloud Hypervisor by default. 31 321. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 33 module. This API is used internally by the Cloud Hypervisor threads to 34 communicate between each others. 35 36The goal of this document is to describe the Cloud Hypervisor API as a whole, 37and to outline how the internal and external APIs are architecturally related. 38 39## External API 40 41### REST API 42 43The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 44API triggers VM and VMM specific actions, and as such it is designed as a 45collection of RPC-style, static methods. 46 47The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 48compliant. Please consult the [Cloud Hypervisor OpenAPI Document](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 49for more details about the API payloads and responses. 50 51#### REST API Location and availability 52 53The REST API is available as soon as the Cloud Hypervisor binary is started, 54through a local UNIX socket. 55By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 56For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 57123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 58 59The REST API default URL can be overridden through the Cloud Hypervisor 60option `--api-socket`: 61 62``` 63$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 64Cloud Hypervisor Guest 65 API server: /tmp/cloud-hypervisor.sock 66 vCPUs: 1 67 Memory: 512 MB 68 Kernel: None 69 Kernel cmdline: 70 Disk(s): None 71``` 72 73#### REST API Endpoints 74 75The Cloud Hypervisor API exposes the following actions through its endpoints: 76 77##### Virtual Machine Manager (VMM) Actions 78 79| Action | Endpoint | Request Body | Response Body | Prerequisites | 80| ----------------------------------- | --------------- | ------------ | -------------------------- | ------------------ | 81| Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A | 82| Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running | 83 84##### Virtual Machine (VM) Actions 85 86| Action | Endpoint | Request Body | Response Body | Prerequisites | 87| ---------------------------------- | ----------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------ | 88| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | 89| Delete the VM | `/vm.delete` | N/A | N/A | N/A | 90| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | 91| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | 92| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | 93| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | 94| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | 95| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | 96| Task a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | 97| Perform a coredump of the VM* | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | 98| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | 99| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 100| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 101| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | 102| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | 103| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 104| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 105| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 106| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 107| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 108| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 109| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 110| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 111| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | 112| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | 113| Prepare to receive a migration | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A | N/A | 114| Start to send migration to target | `/vm.send-migration` | `/schemas/SendMigrationData` | N/A | The VM is booted and (shared mem or hugepages enabled) | 115 116* The `vmcoredump` action is available exclusively for the `x86_64` 117architecture and can be executed only when the `guest_debug` feature is 118enabled. Without this feature, the corresponding [REST API](#rest-api) or 119[D-Bus API](#d-bus-api) endpoints are not available. 120 121#### REST API Examples 122 123For the following set of examples, we assume Cloud Hypervisor is started with 124the REST API available at `/tmp/cloud-hypervisor.sock`: 125 126``` 127$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 128Cloud Hypervisor Guest 129 API server: /tmp/cloud-hypervisor.sock 130 vCPUs: 1 131 Memory: 512 MB 132 Kernel: None 133 Kernel cmdline: 134 Disk(s): None 135``` 136 137##### Create a Virtual Machine 138 139We want to create a virtual machine with the following characteristics: 140 141* 4 vCPUs 142* 1 GB of RAM 143* 1 virtio based networking interface 144* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 145 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 146* Using a Ubuntu image as its root filesystem, located at 147 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 148 149```shell 150#!/usr/bin/env bash 151 152curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 153 -X PUT 'http://localhost/api/v1/vm.create' \ 154 -H 'Accept: application/json' \ 155 -H 'Content-Type: application/json' \ 156 -d '{ 157 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 158 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 159 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 160 "rng":{"src":"/dev/urandom"}, 161 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 162 }' 163``` 164 165##### Boot a Virtual Machine 166 167Once the VM is created, we can boot it: 168 169```shell 170#!/usr/bin/env bash 171 172curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 173``` 174 175##### Dump a Virtual Machine Information 176 177We can fetch information about any VM, as soon as it's created: 178 179```shell 180#!/usr/bin/env bash 181 182curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 183 -X GET 'http://localhost/api/v1/vm.info' \ 184 -H 'Accept: application/json' 185``` 186 187##### Reboot a Virtual Machine 188 189We can reboot a VM that's already booted: 190 191```shell 192#!/usr/bin/env bash 193 194curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 195``` 196 197##### Shut a Virtual Machine Down 198 199Once booted, we can shut a VM down from the REST API: 200 201```shell 202#!/usr/bin/env bash 203 204curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 205``` 206 207### D-Bus API 208 209Cloud Hypervisor offers a D-Bus API as an alternative to its REST API. This 210D-Bus API fully reflects the functionality of the REST API, exposing the 211same group of endpoints. It can be a drop-in replacement since it also 212consumes/produces JSON. 213 214In addition, the D-Bus API also exposes events from `event-monitor` in the 215form of a D-Bus signal to which users can subscribe. For more information, 216see [D-Bus API Interface](#d-bus-api-interface). 217 218#### D-Bus API Location and availability 219 220This feature is not compiled into Cloud Hypervisor by default. Users who 221wish to use the D-Bus API, must explicitly enable it with the `dbus_api` 222feature flag when compiling Cloud Hypervisor. 223 224```sh 225$ ./scripts/dev_cli.sh build --release --libc musl -- --features dbus_api 226``` 227 228Once this feature is enabled, it can be configured with the following 229CLI options: 230 231``` 232 --dbus-service-name 233 well known name of the service 234 --dbus-object-path 235 object path to serve the dbus interface 236 --dbus-system-bus use the system bus instead of a session bus 237``` 238 239Example invocation: 240 241```sh 242$ ./cloud-hypervisor --dbus-service-name "org.cloudhypervisor.DBusApi" \ 243 --dbus-object-path "/org/cloudhypervisor/DBusApi" 244``` 245 246This will start serving a service with the name `org.cloudhypervisor.DBusApi1` 247which in turn can be used to control and manage Cloud Hypervisor. 248 249#### D-Bus API Interface 250 251Please refer to the [REST API](#rest-api) documentation for everything that 252is in common with the REST API. As previously mentioned, the D-Bus API can 253be used as a drop-in replacement for the [REST API](#rest-api). 254 255The D-Bus interface also exposes a signal, named `Event`, which is emitted 256whenever a new event is published from the `event-monitor` crate. Here is its 257definition in XML format: 258 259```xml 260<node> 261 <interface name="org.cloudhypervisor.DBusApi1"> 262 <signal name="Event"> 263 <arg name="event" type="s"/> 264 </signal> 265 </interface> 266</node> 267``` 268 269### Command Line Interface 270 271The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 272the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 273the launched VM once they're up and running. 274 275If you want to inspect the VMM, or control the VM after launching Cloud 276Hypervisor from the CLI, you must use either the [REST API](#rest-api) 277or the [D-Bus API](#d-bus-api). 278 279From the CLI, one can: 280 2811. Create and boot a complete virtual machine by using the CLI options to build 282 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 283 options. As soon as the `cloud-hypervisor` binary is launched, contrary 284 to the [D-Bus API](#d-bus-api), the [REST API](#rest-api) is available 285 for controlling and managing the VM. The [D-Bus API](#d-bus-api) doesn't start 286 automatically and needs to be explicitly configured in order to be run. 2871. Start either the REST API, D-Bus API or both simultaneously without passing 288 any VM configuration options. The VM can then be asynchronously created and 289 booted by calling API methods of choice. It should be noted that one external 290 API does not exclude another; it is possible to have both the REST and D-Bus 291 APIs running simultaneously. 292 293### REST API, D-Bus API and CLI Architectural Relationship 294 295The REST API, D-Bus API and the CLI all rely on a common, [internal API](#internal-api). 296 297The CLI options are parsed by the 298[clap crate](https://docs.rs/clap/4.3.11/clap/) and then translated into 299[internal API](#internal-api) commands. 300 301The REST API is processed by an HTTP thread using the 302[Firecracker's `micro_http`](https://github.com/firecracker-microvm/micro-http) 303crate. As with the CLI, the HTTP requests eventually get translated into 304[internal API](#internal-api) commands. 305 306The D-Bus API is implemented using the [zbus](https://github.com/dbus2/zbus) 307crate and runs in its own thread. Whenever it needs to call the [internal API](#internal-api), 308the [blocking](https://github.com/smol-rs/blocking) crate is used perform the call in zbus' async context. 309 310As a summary, the REST API, the D-Bus API and the CLI are essentially frontends for the 311[internal API](#internal-api): 312 313``` 314 +------------------+ 315 REST API | | 316 +--------->+ micro_http +--------+ 317 | | | | 318 | +------------------+ | 319 | | +------------------------+ 320 | | | | 321+------------+ | +----------+ | | | 322| | | D-Bus API | | | | +--------------+ | 323| User +---------+----------->+ zbus +--------------+------> | Internal API | | 324| | | | | | | +--------------+ | 325+------------+ | +----------+ | | | 326 | | | | 327 | | +------------------------+ 328 | +----------+ | VMM 329 | CLI | | | 330 +----------->+ clap +--------------+ 331 | | 332 +----------+ 333 334 335``` 336 337## Internal API 338 339The Cloud Hypervisor internal API, as its name suggests, is used internally 340by the different Cloud Hypervisor threads (VMM, HTTP, D-Bus, control loop, 341etc) to send commands and responses to each others. 342 343It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 344and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 345control loop. 346 347API producers are the HTTP thread handling the [REST API](#rest-api), the 348D-Bus thread handling the [D-Bus API](#d-bus-api) and the main thread that 349initially parses the [CLI](#command-line-interface). 350 351### Goals and Design 352 353The internal API is designed for controlling, managing and inspecting a Cloud 354Hypervisor VMM and its guest. It is a backend for handling external, user 355visible requests through the [REST API](#rest-api), the [D-Bus API](#d-bus-api) 356or the [CLI](#command-line-interface) interfaces. 357 358The API follows a command-response scheme that closely maps the [REST API](#rest-api). 359Any command must be replied to with a response. 360 361Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 362are received and processed by the VMM control loop. 363 364In order for the VMM control loop to respond to any internal API command, it 365must be able to send a response back to the MPSC sender. For that purpose, all 366internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 367end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 368 369The sender of any internal API command is therefore responsible for: 370 3711. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 372 channel. 3731. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 374 end of the response channel as part of the internal API command payload. 3751. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 376 end of the response channel. 377 378## End to End Example 379 380In order to further understand how the external and internal Cloud Hypervisor 381APIs work together, let's look at a complete VM creation flow, from the 382[REST API](#rest-api) call, to the reply the external user will receive: 383 3841. A user or operator sends an HTTP request to the Cloud Hypervisor 385 [REST API](#rest-api) in order to creates a virtual machine: 386 ``` 387 shell 388 #!/usr/bin/env bash 389 390 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 391 -X PUT 'http://localhost/api/v1/vm.create' \ 392 -H 'Accept: application/json' \ 393 -H 'Content-Type: application/json' \ 394 -d '{ 395 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 396 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 397 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 398 "rng":{"src":"/dev/urandom"}, 399 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 400 }' 401 ``` 4021. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 403 HTTP request JSON body into an internal `VmConfig` structure. 4041. The Cloud Hypervisor HTTP thread creates an 405 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 406 server to send its response back. 4071. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 408 virtual machine. The command's payload is made of the de-serialized 409 `VmConfig` structure and the response channel: 410 ```Rust 411 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 412 ``` 4131. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 414 for the response: 415 ```Rust 416 // Send the VM creation request. 417 api_sender 418 .send(ApiRequest::VmCreate(config, response_sender)) 419 .map_err(ApiError::RequestSend)?; 420 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 421 422 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 423 ``` 4241. The Cloud Hypervisor control loop receives the command, as it listens on the 425 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 426 ```Rust 427 // Read from the API receiver channel 428 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 429 ``` 4301. The Cloud Hypervisor control loop matches the received internal API against 431 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 432 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 433 command payload. It stores the `VmConfig` structure and replies back to the 434 sender ((The HTTP thread): 435 ```Rust 436 match api_request { 437 ApiRequest::VmCreate(config, sender) => { 438 // We only store the passed VM config. 439 // The VM will be created when being asked to boot it. 440 let response = if self.vm_config.is_none() { 441 self.vm_config = Some(config); 442 Ok(ApiResponsePayload::Empty) 443 } else { 444 Err(ApiError::VmAlreadyCreated) 445 }; 446 447 sender.send(response).map_err(Error::ApiResponseSend)?; 448 } 449 ``` 4501. The Cloud Hypervisor HTTP thread receives the internal API command response 451 as the return value from its `VmCreate` HTTP handler. Depending on the 452 control loop internal API response, it generates the appropriate HTTP 453 response: 454 ```Rust 455 // Call vm_create() 456 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 457 .map_err(HttpError::VmCreate) 458 { 459 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 460 Err(e) => error_response(e, StatusCode::InternalServerError), 461 } 462 ``` 4631. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 464 user. This is abstracted by the 465 [micro_http](https://github.com/firecracker-microvm/micro-http) 466 crate. 467