1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 - [External API](#external-api) 3 - [REST API](#rest-api) 4 - [REST API Location and availability](#rest-api-location-and-availability) 5 - [REST API Endpoints](#rest-api-endpoints) 6 - [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 - [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 - [Create a Virtual Machine](#create-a-virtual-machine) 10 - [Boot a Virtual Machine](#boot-a-virtual-machine) 11 - [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 - [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 - [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 - [D-Bus API](#d-bus-api) 15 - [D-Bus API Location and availability](#d-bus-api-location-and-availability) 16 - [D-Bus API Interface](#d-bus-api-interface) 17 - [Command Line Interface](#command-line-interface) 18 - [REST API, D-Bus API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 19 - [Internal API](#internal-api) 20 - [Goals and Design](#goals-and-design) 21 - [End to End Example](#end-to-end-example) 22 23# Cloud Hypervisor API 24 25The Cloud Hypervisor API is made of 2 distinct interfaces: 26 271. **The External API** This is the user facing API. Users and operators 28 can control and manage the Cloud Hypervisor through various options 29 including a REST API, a Command Line Interface (CLI) or a D-Bus based API, 30 which is not compiled into Cloud Hypervisor by default. 31 321. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 33 module. This API is used internally by the Cloud Hypervisor threads to 34 communicate between each others. 35 36The goal of this document is to describe the Cloud Hypervisor API as a whole, 37and to outline how the internal and external APIs are architecturally related. 38 39## External API 40 41### REST API 42 43The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 44API triggers VM and VMM specific actions, and as such it is designed as a 45collection of RPC-style, static methods. 46 47The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 48compliant. Please consult the [Cloud Hypervisor OpenAPI Document](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 49for more details about the API payloads and responses. 50 51#### REST API Location and availability 52 53The REST API, if enabled, is available as soon as the Cloud Hypervisor binary is started, 54through either a local UNIX socket as given in the Cloud Hypervisor option `--api-socket path=...` 55or a fd with `--api-socket fd=...`. 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket path=/tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68#### REST API Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72##### Virtual Machine Manager (VMM) Actions 73 74| Action | Endpoint | Request Body | Response Body | Prerequisites | 75| ----------------------------------- | --------------- | ------------ | -------------------------- | ------------------ | 76| Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A | 77| Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running | 78 79##### Virtual Machine (VM) Actions 80 81| Action | Endpoint | Request Body | Response Body | Prerequisites | 82| ---------------------------------- | ----------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------ | 83| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | 84| Delete the VM | `/vm.delete` | N/A | N/A | N/A | 85| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | 86| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | 87| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | 88| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | 89| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | 90| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | 91| Task a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | 92| Perform a coredump of the VM* | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | 93| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | 94| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 95| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | 96| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | 97| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | 98| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 99| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 100| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 101| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 102| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 103| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | 104| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 105| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | 106| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | 107| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | 108| Inject an NMI | `/vm.nmi` | N/A | N/A | The VM is booted | 109| Prepare to receive a migration | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A | N/A | 110| Start to send migration to target | `/vm.send-migration` | `/schemas/SendMigrationData` | N/A | The VM is booted and (shared mem or hugepages enabled) | 111 112* The `vmcoredump` action is available exclusively for the `x86_64` 113architecture and can be executed only when the `guest_debug` feature is 114enabled. Without this feature, the corresponding [REST API](#rest-api) or 115[D-Bus API](#d-bus-api) endpoints are not available. 116 117#### REST API Examples 118 119For the following set of examples, we assume Cloud Hypervisor is started with 120the REST API available at `/tmp/cloud-hypervisor.sock`: 121 122``` 123$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 124Cloud Hypervisor Guest 125 API server: /tmp/cloud-hypervisor.sock 126 vCPUs: 1 127 Memory: 512 MB 128 Kernel: None 129 Kernel cmdline: 130 Disk(s): None 131``` 132 133##### Create a Virtual Machine 134 135We want to create a virtual machine with the following characteristics: 136 137* 4 vCPUs 138* 1 GB of RAM 139* 1 virtio based networking interface 140* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 141 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 142* Using a Ubuntu image as its root filesystem, located at 143 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 144 145```shell 146#!/usr/bin/env bash 147 148curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 149 -X PUT 'http://localhost/api/v1/vm.create' \ 150 -H 'Accept: application/json' \ 151 -H 'Content-Type: application/json' \ 152 -d '{ 153 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 154 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 155 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 156 "rng":{"src":"/dev/urandom"}, 157 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 158 }' 159``` 160 161##### Boot a Virtual Machine 162 163Once the VM is created, we can boot it: 164 165```shell 166#!/usr/bin/env bash 167 168curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 169``` 170 171##### Dump a Virtual Machine Information 172 173We can fetch information about any VM, as soon as it's created: 174 175```shell 176#!/usr/bin/env bash 177 178curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 179 -X GET 'http://localhost/api/v1/vm.info' \ 180 -H 'Accept: application/json' 181``` 182 183##### Reboot a Virtual Machine 184 185We can reboot a VM that's already booted: 186 187```shell 188#!/usr/bin/env bash 189 190curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 191``` 192 193##### Shut a Virtual Machine Down 194 195Once booted, we can shut a VM down from the REST API: 196 197```shell 198#!/usr/bin/env bash 199 200curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 201``` 202 203### D-Bus API 204 205Cloud Hypervisor offers a D-Bus API as an alternative to its REST API. This 206D-Bus API fully reflects the functionality of the REST API, exposing the 207same group of endpoints. It can be a drop-in replacement since it also 208consumes/produces JSON. 209 210In addition, the D-Bus API also exposes events from `event-monitor` in the 211form of a D-Bus signal to which users can subscribe. For more information, 212see [D-Bus API Interface](#d-bus-api-interface). 213 214#### D-Bus API Location and availability 215 216This feature is not compiled into Cloud Hypervisor by default. Users who 217wish to use the D-Bus API, must explicitly enable it with the `dbus_api` 218feature flag when compiling Cloud Hypervisor. 219 220```sh 221$ ./scripts/dev_cli.sh build --release --libc musl -- --features dbus_api 222``` 223 224Once this feature is enabled, it can be configured with the following 225CLI options: 226 227``` 228 --dbus-service-name 229 well known name of the service 230 --dbus-object-path 231 object path to serve the dbus interface 232 --dbus-system-bus use the system bus instead of a session bus 233``` 234 235Example invocation: 236 237```sh 238$ ./cloud-hypervisor --dbus-service-name "org.cloudhypervisor.DBusApi" \ 239 --dbus-object-path "/org/cloudhypervisor/DBusApi" 240``` 241 242This will start serving a service with the name `org.cloudhypervisor.DBusApi1` 243which in turn can be used to control and manage Cloud Hypervisor. 244 245#### D-Bus API Interface 246 247Please refer to the [REST API](#rest-api) documentation for everything that 248is in common with the REST API. As previously mentioned, the D-Bus API can 249be used as a drop-in replacement for the [REST API](#rest-api). 250 251The D-Bus interface also exposes a signal, named `Event`, which is emitted 252whenever a new event is published from the `event-monitor` crate. Here is its 253definition in XML format: 254 255```xml 256<node> 257 <interface name="org.cloudhypervisor.DBusApi1"> 258 <signal name="Event"> 259 <arg name="event" type="s"/> 260 </signal> 261 </interface> 262</node> 263``` 264 265### Command Line Interface 266 267The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 268the Cloud Hypervisor binary, i.e. it cannot be used for controlling the VMM or 269the launched VM once they're up and running. 270 271If you want to inspect the VMM, or control the VM after launching Cloud 272Hypervisor from the CLI, you must use either the [REST API](#rest-api) 273or the [D-Bus API](#d-bus-api). 274 275From the CLI, one can: 276 2771. Create and boot a complete virtual machine by using the CLI options to build 278 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 279 options. As soon as the `cloud-hypervisor` binary is launched, contrary 280 to the [D-Bus API](#d-bus-api), the [REST API](#rest-api) is available 281 for controlling and managing the VM. The [D-Bus API](#d-bus-api) doesn't start 282 automatically and needs to be explicitly configured in order to be run. 2831. Start either the REST API, D-Bus API or both simultaneously without passing 284 any VM configuration options. The VM can then be asynchronously created and 285 booted by calling API methods of choice. It should be noted that one external 286 API does not exclude another; it is possible to have both the REST and D-Bus 287 APIs running simultaneously. 288 289### REST API, D-Bus API and CLI Architectural Relationship 290 291The REST API, D-Bus API and the CLI all rely on a common, [internal API](#internal-api). 292 293The CLI options are parsed by the 294[clap crate](https://docs.rs/clap/4.3.11/clap/) and then translated into 295[internal API](#internal-api) commands. 296 297The REST API is processed by an HTTP thread using the 298[Firecracker's `micro_http`](https://github.com/firecracker-microvm/micro-http) 299crate. As with the CLI, the HTTP requests eventually get translated into 300[internal API](#internal-api) commands. 301 302The D-Bus API is implemented using the [zbus](https://github.com/dbus2/zbus) 303crate and runs in its own thread. Whenever it needs to call the [internal API](#internal-api), 304the [blocking](https://github.com/smol-rs/blocking) crate is used perform the call in zbus' async context. 305 306As a summary, the REST API, the D-Bus API and the CLI are essentially frontends for the 307[internal API](#internal-api): 308 309``` 310 +------------------+ 311 REST API | | 312 +--------->+ micro_http +--------+ 313 | | | | 314 | +------------------+ | 315 | | +------------------------+ 316 | | | | 317+------------+ | +----------+ | | | 318| | | D-Bus API | | | | +--------------+ | 319| User +---------+----------->+ zbus +--------------+------> | Internal API | | 320| | | | | | | +--------------+ | 321+------------+ | +----------+ | | | 322 | | | | 323 | | +------------------------+ 324 | +----------+ | VMM 325 | CLI | | | 326 +----------->+ clap +--------------+ 327 | | 328 +----------+ 329 330 331``` 332 333## Internal API 334 335The Cloud Hypervisor internal API, as its name suggests, is used internally 336by the different Cloud Hypervisor threads (VMM, HTTP, D-Bus, control loop, 337etc) to send commands and responses to each others. 338 339It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 340and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 341control loop. 342 343API producers are the HTTP thread handling the [REST API](#rest-api), the 344D-Bus thread handling the [D-Bus API](#d-bus-api) and the main thread that 345initially parses the [CLI](#command-line-interface). 346 347### Goals and Design 348 349The internal API is designed for controlling, managing and inspecting a Cloud 350Hypervisor VMM and its guest. It is a backend for handling external, user 351visible requests through the [REST API](#rest-api), the [D-Bus API](#d-bus-api) 352or the [CLI](#command-line-interface) interfaces. 353 354The API follows a command-response scheme that closely maps the [REST API](#rest-api). 355Any command must be replied to with a response. 356 357Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 358are received and processed by the VMM control loop. 359 360In order for the VMM control loop to respond to any internal API command, it 361must be able to send a response back to the MPSC sender. For that purpose, all 362internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 363end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 364 365The sender of any internal API command is therefore responsible for: 366 3671. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 368 channel. 3691. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 370 end of the response channel as part of the internal API command payload. 3711. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 372 end of the response channel. 373 374## End to End Example 375 376In order to further understand how the external and internal Cloud Hypervisor 377APIs work together, let's look at a complete VM creation flow, from the 378[REST API](#rest-api) call, to the reply the external user will receive: 379 3801. A user or operator sends an HTTP request to the Cloud Hypervisor 381 [REST API](#rest-api) in order to creates a virtual machine: 382 ``` 383 shell 384 #!/usr/bin/env bash 385 386 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 387 -X PUT 'http://localhost/api/v1/vm.create' \ 388 -H 'Accept: application/json' \ 389 -H 'Content-Type: application/json' \ 390 -d '{ 391 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 392 "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 393 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 394 "rng":{"src":"/dev/urandom"}, 395 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 396 }' 397 ``` 3981. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 399 HTTP request JSON body into an internal `VmConfig` structure. 4001. The Cloud Hypervisor HTTP thread creates an 401 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 402 server to send its response back. 4031. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 404 virtual machine. The command's payload is made of the de-serialized 405 `VmConfig` structure and the response channel: 406 ```Rust 407 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 408 ``` 4091. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 410 for the response: 411 ```Rust 412 // Send the VM creation request. 413 api_sender 414 .send(ApiRequest::VmCreate(config, response_sender)) 415 .map_err(ApiError::RequestSend)?; 416 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 417 418 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 419 ``` 4201. The Cloud Hypervisor control loop receives the command, as it listens on the 421 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 422 ```Rust 423 // Read from the API receiver channel 424 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 425 ``` 4261. The Cloud Hypervisor control loop matches the received internal API against 427 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 428 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 429 command payload. It stores the `VmConfig` structure and replies back to the 430 sender ((The HTTP thread): 431 ```Rust 432 match api_request { 433 ApiRequest::VmCreate(config, sender) => { 434 // We only store the passed VM config. 435 // The VM will be created when being asked to boot it. 436 let response = if self.vm_config.is_none() { 437 self.vm_config = Some(config); 438 Ok(ApiResponsePayload::Empty) 439 } else { 440 Err(ApiError::VmAlreadyCreated) 441 }; 442 443 sender.send(response).map_err(Error::ApiResponseSend)?; 444 } 445 ``` 4461. The Cloud Hypervisor HTTP thread receives the internal API command response 447 as the return value from its `VmCreate` HTTP handler. Depending on the 448 control loop internal API response, it generates the appropriate HTTP 449 response: 450 ```Rust 451 // Call vm_create() 452 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 453 .map_err(HttpError::VmCreate) 454 { 455 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 456 Err(e) => error_response(e, StatusCode::InternalServerError), 457 } 458 ``` 4591. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 460 user. This is abstracted by the 461 [micro_http](https://github.com/firecracker-microvm/micro-http) 462 crate. 463