1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 * [External API](#external-api) 3 + [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 * [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 * [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 * [Create a Virtual Machine](#create-a-virtual-machine) 10 * [Boot a Virtual Machine](#boot-a-virtual-machine) 11 * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 * [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 + [Command Line Interface](#command-line-interface) 15 + [REST API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 16 * [Internal API](#internal-api) 17 + [Goals and Design](#goals-and-design) 18 * [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74Action | Endpoint | Request Body | Response Body | Prerequisites 75------------------------------------|-----------------|--------------|----------------------------|--------------------------- 76Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A 77Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running 78 79#### Virtual Machine (VM) Actions 80 81Action | Endpoint | Request Body | Response Body | Prerequisites 82-----------------------------------|----------------------|---------------------------|--------------------------|--------------------------- 83Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet 84Delete the VM | `/vm.delete` | N/A | N/A | N/A 85Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted 86Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted 87Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted 88Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted 89Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted 90Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused 91Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 92Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 93Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted 94Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created 95Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted 96Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted 97Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted 98Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted 99Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted 100Add userspace PCI device to the VM | `/vm.add-user-device`| `/schemas/VmAddUserDevice`| `/schemas/PciDeviceInfo` | The VM is booted 101Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted 102Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted 103Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted 104 105### REST API Examples 106 107For the following set of examples, we assume Cloud Hypervisor is started with 108the REST API available at `/tmp/cloud-hypervisor.sock`: 109 110``` 111$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 112Cloud Hypervisor Guest 113 API server: /tmp/cloud-hypervisor.sock 114 vCPUs: 1 115 Memory: 512 MB 116 Kernel: None 117 Kernel cmdline: 118 Disk(s): None 119``` 120 121#### Create a Virtual Machine 122 123We want to create a virtual machine with the following characteristics: 124 125* 4 vCPUs 126* 1 GB of RAM 127* 1 virtio based networking interface 128* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 129 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 130* Using a Ubuntu image as its root filesystem, located at 131 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 132 133```shell 134#!/bin/bash 135 136curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 137 -X PUT 'http://localhost/api/v1/vm.create' \ 138 -H 'Accept: application/json' \ 139 -H 'Content-Type: application/json' \ 140 -d '{ 141 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 142 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 143 "cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 144 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 145 "rng":{"src":"/dev/urandom"}, 146 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 147 }' 148``` 149 150#### Boot a Virtual Machine 151 152Once the VM is created, we can boot it: 153 154```shell 155#!/bin/bash 156 157curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 158``` 159 160#### Dump a Virtual Machine Information 161 162We can fetch information about any VM, as soon as it's created: 163 164```shell 165#!/bin/bash 166 167curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 168 -X GET 'http://localhost/api/v1/vm.info' \ 169 -H 'Accept: application/json' 170``` 171 172#### Reboot a Virtual Machine 173 174We can reboot a VM that's already booted: 175 176```shell 177#!/bin/bash 178 179curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 180``` 181 182#### Shut a Virtual Machine Down 183 184Once booted, we can shut a VM down from the REST API: 185 186```shell 187#!/bin/bash 188 189curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 190``` 191 192### Command Line Interface 193 194The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 195the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 196the launched VM once they're up and running. 197 198If you want to inspect the VMM, or control the VM after launching Cloud 199Hypervisor from the CLI, you must use the [REST API](#rest-api). 200 201From the CLI, one can either: 202 2031. Create and boot a complete virtual machine by using the CLI options to build 204 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 205 options. As soon as the `cloud-hypervisor` binary is launched, the 206 [REST API](#rest-api) is available for controlling and managing the VM. 2071. Start the [REST API](#rest-api) server only, by not passing any VM 208 configuration options. The VM can then be asynchronously created and booted 209 by sending HTTP commands to the [REST API](#rest-api). Check the 210 [REST API examples](#rest-api-examples) section for more details. 211 212### REST API and CLI Architectural Relationship 213 214The REST API and the CLI both rely on a common, [internal API](#internal-api). 215 216The CLI options are parsed by the 217[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into 218[internal API](#internal-api) commands. 219 220The REST API is processed by an HTTP thread using the 221[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 222crate. As with the CLI, the HTTP requests eventually get translated into 223[internal API](#internal-api) commands. 224 225As a summary, the REST API and the CLI are essentially frontends for the 226[internal API](#internal-api): 227 228``` 229 +------------------+ 230 REST API | | 231 +--------->+ micro_http +--------+ 232 | | | | 233 | +------------------+ | 234 | | +------------------------+ 235 | | | | 236+------------+ | | | | 237| | | | | +--------------+ | 238| User +---------+ +------> | Internal API | | 239| | | | | +--------------+ | 240+------------+ | | | | 241 | | | | 242 | | +------------------------+ 243 | +----------+ | VMM 244 | CLI | | | 245 +----------->+ clap +--------------+ 246 | | 247 +----------+ 248 249 250``` 251 252## Internal API 253 254The Cloud Hypervisor internal API, as its name suggests, is used internally 255by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 256send commands and responses to each others. 257 258It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 259and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 260control loop. 261 262API producers are the HTTP thread handling the [REST API](#rest-api) and the 263main thread that initially parses the [CLI](#command-line-interface). 264 265### Goals and Design 266 267The internal API is designed for controlling, managing and inspecting a Cloud 268Hypervisor VMM and its guest. It is a backend for handling external, user 269visible requests through either the [REST API](#rest-api) or the 270[CLI](#command-line-interface) interfaces. 271 272The API follows a command-response scheme that closely maps the [REST API](#rest-api). 273Any command must be replied to with a response. 274 275Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 276are received and processed by the VMM control loop. 277 278In order for the VMM control loop to respond to any internal API command, it 279must be able to send a response back to the MPSC sender. For that purpose, all 280internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 281end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 282 283The sender of any internal API command is therefore responsible for: 284 2851. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 286 channel. 2871. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 288 end of the response channel as part of the internal API command payload. 2891. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 290 end of the response channel. 291 292## End to End Example 293 294In order to further understand how the external and internal Cloud Hypervisor 295APIs work together, let's look at a complete VM creation flow, from the 296[REST API](#rest-api) call, to the reply the external user will receive: 297 2981. A user or operator sends an HTTP request to the Cloud Hypervisor 299 [REST API](#rest-api) in order to creates a virtual machine: 300 ``` 301 shell 302 #!/bin/bash 303 304 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 305 -X PUT 'http://localhost/api/v1/vm.create' \ 306 -H 'Accept: application/json' \ 307 -H 'Content-Type: application/json' \ 308 -d '{ 309 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 310 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 311 "cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 312 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 313 "rng":{"src":"/dev/urandom"}, 314 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 315 }' 316 ``` 3171. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 318 HTTP request JSON body into an internal `VmConfig` structure. 3191. The Cloud Hypervisor HTTP thread creates an 320 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 321 server to send its response back. 3221. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 323 virtual machine. The command's payload is made of the de-serialized 324 `VmConfig` structure and the response channel: 325 ```Rust 326 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 327 ``` 3281. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 329 for the response: 330 ```Rust 331 // Send the VM creation request. 332 api_sender 333 .send(ApiRequest::VmCreate(config, response_sender)) 334 .map_err(ApiError::RequestSend)?; 335 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 336 337 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 338 ``` 3391. The Cloud Hypervisor control loop receives the command, as it listens on the 340 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 341 ```Rust 342 // Read from the API receiver channel 343 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 344 ``` 3451. The Cloud Hypervisor control loop matches the received internal API against 346 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 347 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 348 command payload. It stores the `VmConfig` structure and replies back to the 349 sender ((The HTTP thread): 350 ```Rust 351 match api_request { 352 ApiRequest::VmCreate(config, sender) => { 353 // We only store the passed VM config. 354 // The VM will be created when being asked to boot it. 355 let response = if self.vm_config.is_none() { 356 self.vm_config = Some(config); 357 Ok(ApiResponsePayload::Empty) 358 } else { 359 Err(ApiError::VmAlreadyCreated) 360 }; 361 362 sender.send(response).map_err(Error::ApiResponseSend)?; 363 } 364 ``` 3651. The Cloud Hypervisor HTTP thread receives the internal API command response 366 as the return value from its `VmCreate` HTTP handler. Depending on the 367 control loop internal API response, it generates the appropriate HTTP 368 response: 369 ```Rust 370 // Call vm_create() 371 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 372 .map_err(HttpError::VmCreate) 373 { 374 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 375 Err(e) => error_response(e, StatusCode::InternalServerError), 376 } 377 ``` 3781. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 379 user. This is abstracted by the 380 [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 381 crate. 382