1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 * [External API](#external-api) 3 + [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 * [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 * [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 * [Create a Virtual Machine](#create-a-virtual-machine) 10 * [Boot a Virtual Machine](#boot-a-virtual-machine) 11 * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 * [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 + [Command Line Interface](#command-line-interface) 15 + [REST API and CLI Architecural Relationship](#rest-api-and-cli-architectural-relationship) 16 * [Internal API](#internal-api) 17 + [Goals and Design](#goals-and-design) 18 * [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74Action | Endpoint | Request Body | Response Body | Prerequisites 75------------------------------------|-----------------|--------------|----------------------------|--------------------------- 76Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A 77Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running 78 79#### Virtual Machine (VM) Actions 80 81Action | Endpoint | Request Body | Response Body | Prerequisites 82-----------------------------------|---------------------|---------------------------|--------------------------|--------------------------- 83Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet 84Delete the VM | `/vm.delete` | N/A | N/A | N/A 85Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted 86Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted 87Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted 88Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted 89Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused 90Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 91Remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 92Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created 93Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted 94Remove VFIO PCI device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted 95Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted 96Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted 97Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted 98Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted 99Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted 100Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted 101 102### REST API Examples 103 104For the following set of examples, we assume Cloud Hypervisor is started with 105the REST API available at `/tmp/cloud-hypervisor.sock`: 106 107``` 108$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 109Cloud Hypervisor Guest 110 API server: /tmp/cloud-hypervisor.sock 111 vCPUs: 1 112 Memory: 512 MB 113 Kernel: None 114 Kernel cmdline: 115 Disk(s): None 116``` 117 118#### Create a Virtual Machine 119 120We want to create a virtual machine with the following characteristics: 121 122* 4 vCPUs 123* 1 GB of RAM 124* 1 virtio based networking interface 125* Direct kernel boot from a custom 5.5.0 Linux kernel located at 126 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 127* Using a Clear Linux image as its root filesystem, located at 128 `/opt/clh/images/clear-30080-kvm.img` 129 130```shell 131#!/bin/bash 132 133curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 134 -X PUT 'http://localhost/api/v1/vm.create' \ 135 -H 'Accept: application/json' \ 136 -H 'Content-Type: application/json' \ 137 -d '{ 138 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 139 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 140 "cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"}, 141 "disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}], 142 "rng":{"src":"/dev/urandom"}, 143 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 144 }' 145``` 146 147#### Boot a Virtual Machine 148 149Once the VM is created, we can boot it: 150 151```shell 152#!/bin/bash 153 154curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 155``` 156 157#### Dump a Virtual Machine Information 158 159We can fetch information about any VM, as soon as it's created: 160 161```shell 162#!/bin/bash 163 164curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 165 -X GET 'http://localhost/api/v1/vm.info' \ 166 -H 'Accept: application/json' 167``` 168 169#### Reboot a Virtual Machine 170 171We can reboot a VM that's already booted: 172 173```shell 174#!/bin/bash 175 176curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 177``` 178 179#### Shut a Virtual Machine Down 180 181Once booted, we can shut a VM down from the REST API: 182 183```shell 184#!/bin/bash 185 186curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 187``` 188 189### Command Line Interface 190 191The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 192the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 193the launched VM once they're up and running. 194 195If you want to inspect the VMM, or control the VM after launching Cloud 196Hypervisor from the CLI, you must use the [REST API](#rest-api). 197 198From the CLI, one can either: 199 2001. Create and boot a complete virtual machine by using the CLI options to build 201 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 202 options. As soon as the `cloud-hypervisor` binary is launched, the 203 [REST API](#rest-api) is available for controlling and managing the VM. 2041. Start the [REST API](#rest-api) server only, by not passing any VM 205 configuration options. The VM can then be asynchronously created and booted 206 by sending HTTP commands to the [REST API](#rest-api). Check the 207 [REST API examples](#rest-api-examples) section for more details. 208 209### REST API and CLI Architectural Relationship 210 211The REST API and the CLI both rely on a common, [internal API](#internal-api). 212 213The CLI options are parsed by the 214[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into 215[internal API](#internal-api) commands. 216 217The REST API is processed by an HTTP thread using the 218[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 219crate. As with the CLI, the HTTP requests eventually get translated into 220[internal API](#internal-api) commands. 221 222As a summary, the REST API and the CLI are essentially frontends for the 223[internal API](#internal-api): 224 225``` 226 +------------------+ 227 REST API | | 228 +--------->+ micro_http +--------+ 229 | | | | 230 | +------------------+ | 231 | | +------------------------+ 232 | | | | 233+------------+ | | | | 234| | | | | +--------------+ | 235| User +---------+ +------> | Internal API | | 236| | | | | +--------------+ | 237+------------+ | | | | 238 | | | | 239 | | +------------------------+ 240 | +----------+ | VMM 241 | CLI | | | 242 +----------->+ clap +--------------+ 243 | | 244 +----------+ 245 246 247``` 248 249## Internal API 250 251The Cloud Hypervisor internal API, as its name suggests, is used internally 252by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 253send commands and responses to each others. 254 255It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 256and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 257control loop. 258 259API producers are the HTTP thread handling the [REST API](#rest-api) and the 260main thread that initially parses the [CLI](#command-line-interface). 261 262### Goals and Design 263 264The internal API is designed for controlling, managing and inspecting a Cloud 265Hypervisor VMM and its guest. It is a backend for handling external, user 266visible requests through either the [REST API](#rest-api) or the 267[CLI](#command-line-interface) interfaces. 268 269The API follows a command-response scheme that closely maps the [REST API](#rest-api). 270Any command must be replied to with a response. 271 272Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 273are received and processed by the VMM control loop. 274 275In order for the VMM control loop to respond to any internal API command, it 276must be able to send a response back to the MPSC sender. For that purpose, all 277internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 278end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 279 280The sender of any internal API command is therefore responsible for: 281 2821. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 283 channel. 2841. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 285 end of the response channel as part of the internal API command payload. 2861. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 287 end of the response channel. 288 289## End to End Example 290 291In order to further understand how the external and internal Cloud Hypervisor 292APIs work together, let's look at a complete VM creation flow, from the 293[REST API](#rest-api) call, to the reply the external user will receive: 294 2951. A user or operator sends an HTTP request to the Cloud Hypervisor 296 [REST API](#rest-api) in order to creates a virtual machine: 297 ``` 298 shell 299 #!/bin/bash 300 301 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 302 -X PUT 'http://localhost/api/v1/vm.create' \ 303 -H 'Accept: application/json' \ 304 -H 'Content-Type: application/json' \ 305 -d '{ 306 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 307 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 308 "cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"}, 309 "disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}], 310 "rng":{"src":"/dev/urandom"}, 311 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 312 }' 313 ``` 3141. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 315 HTTP request JSON body into an internal `VmConfig` structure. 3161. The Cloud Hypervisor HTTP thread creates an 317 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 318 server to send its response back. 3191. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 320 virtual machine. The command's payload is made of the de-serialized 321 `VmConfig` structure and the response channel: 322 ```Rust 323 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 324 ``` 3251. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 326 for the response: 327 ```Rust 328 // Send the VM creation request. 329 api_sender 330 .send(ApiRequest::VmCreate(config, response_sender)) 331 .map_err(ApiError::RequestSend)?; 332 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 333 334 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 335 ``` 3361. The Cloud Hypervisor control loop receives the command, as it listens on the 337 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 338 ```Rust 339 // Read from the API receiver channel 340 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 341 ``` 3421. The Cloud Hypervisor control loop matches the received internal API against 343 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 344 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 345 command payload. It stores the `VmConfig` structure and replies back to the 346 sender ((The HTTP thread): 347 ```Rust 348 match api_request { 349 ApiRequest::VmCreate(config, sender) => { 350 // We only store the passed VM config. 351 // The VM will be created when being asked to boot it. 352 let response = if self.vm_config.is_none() { 353 self.vm_config = Some(config); 354 Ok(ApiResponsePayload::Empty) 355 } else { 356 Err(ApiError::VmAlreadyCreated) 357 }; 358 359 sender.send(response).map_err(Error::ApiResponseSend)?; 360 } 361 ``` 3621. The Cloud Hypervisor HTTP thread receives the internal API command response 363 as the return value from its `VmCreate` HTTP handler. Depending on the 364 control loop internal API response, it generates the appropriate HTTP 365 response: 366 ```Rust 367 // Call vm_create() 368 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 369 .map_err(HttpError::VmCreate) 370 { 371 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 372 Err(e) => error_response(e, StatusCode::InternalServerError), 373 } 374 ``` 3751. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 376 user. This is abstracted by the 377 [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 378 crate. 379