1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 * [External API](#external-api) 3 + [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 * [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 * [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 * [Create a Virtual Machine](#create-a-virtual-machine) 10 * [Boot a Virtual Machine](#boot-a-virtual-machine) 11 * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 * [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 + [Command Line Interface](#command-line-interface) 15 + [REST API and CLI Architecural Relationship](#rest-api-and-cli-architectural-relationship) 16 * [Internal API](#internal-api) 17 + [Goals and Design](#goals-and-design) 18 * [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74Action | Endpoint | Request Body | Response Body | Prerequisites 75------------------------------------|-----------------|--------------|----------------------------|--------------------------- 76Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A 77Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running 78 79#### Virtual Machine (VM) Actions 80 81Action | Endpoint | Request Body | Response Body | Prerequisites 82-----------------------------------|---------------------|---------------------------|-------------------|--------------------------- 83Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet 84Delete the VM | `/vm.delete` | N/A | N/A | The VM is created but not booted 85Boot the VM | `/vm.boot` | N/A | N/A | The VM is created 86Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted 87Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted 88Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted 89Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused 90Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 91Remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 92Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created 93Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | N/A | The VM is booted 94Remove VFIO PCI device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted 95 96### REST API Examples 97 98For the following set of examples, we assume Cloud Hypervisor is started with 99the REST API available at `/tmp/cloud-hypervisor.sock`: 100 101``` 102$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 103Cloud Hypervisor Guest 104 API server: /tmp/cloud-hypervisor.sock 105 vCPUs: 1 106 Memory: 512 MB 107 Kernel: None 108 Kernel cmdline: 109 Disk(s): None 110``` 111 112#### Create a Virtual Machine 113 114We want to create a virtual machine with the following characteristics: 115 116* 4 vCPUs 117* 1 GB of RAM 118* 1 virtio based networking interface 119* Direct kernel boot from a custom 5.5.0 Linux kernel located at 120 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 121* Using a Clear Linux image as its root filesystem, located at 122 `/opt/clh/images/clear-30080-kvm.img` 123 124```shell 125#!/bin/bash 126 127curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 128 -X PUT 'http://localhost/api/v1/vm.create' \ 129 -H 'Accept: application/json' \ 130 -H 'Content-Type: application/json' \ 131 -d '{ 132 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 133 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 134 "cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"}, 135 "disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}], 136 "rng":{"src":"/dev/urandom"}, 137 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 138 }' 139``` 140 141#### Boot a Virtual Machine 142 143Once the VM is created, we can boot it: 144 145```shell 146#!/bin/bash 147 148curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 149``` 150 151#### Dump a Virtual Machine Information 152 153We can fetch information about any VM, as soon as it's created: 154 155```shell 156#!/bin/bash 157 158curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 159 -X GET 'http://localhost/api/v1/vm.info' \ 160 -H 'Accept: application/json' 161``` 162 163#### Reboot a Virtual Machine 164 165We can reboot a VM that's already booted: 166 167```shell 168#!/bin/bash 169 170curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 171``` 172 173#### Shut a Virtual Machine Down 174 175Once booted, we can shut a VM down from the REST API: 176 177```shell 178#!/bin/bash 179 180curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 181``` 182 183### Command Line Interface 184 185The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 186the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 187the launched VM once they're up and running. 188 189If you want to inspect the VMM, or control the VM after launching Cloud 190Hypervisor from the CLI, you must use the [REST API](#rest-api). 191 192From the CLI, one can either: 193 1941. Create and boot a complete virtual machine by using the CLI options to build 195 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 196 options. As soon as the `cloud-hypervisor` binary is launched, the 197 [REST API](#rest-api) is available for controlling and managing the VM. 1981. Start the [REST API](#rest-api) server only, by not passing any VM 199 configuration options. The VM can then be asynchronously created and booted 200 by sending HTTP commands to the [REST API](#rest-api). Check the 201 [REST API examples](#rest-api-examples) section for more details. 202 203### REST API and CLI Architectural Relationship 204 205The REST API and the CLI both rely on a common, [internal API](#internal-api). 206 207The CLI options are parsed by the 208[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into 209[internal API](#internal-api) commands. 210 211The REST API is processed by an HTTP thread using the 212[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 213crate. As with the CLI, the HTTP requests eventually get translated into 214[internal API](#internal-api) commands. 215 216As a summary, the REST API and the CLI are essentially frontends for the 217[internal API](#internal-api): 218 219``` 220 +------------------+ 221 REST API | | 222 +--------->+ micro_http +--------+ 223 | | | | 224 | +------------------+ | 225 | | +------------------------+ 226 | | | | 227+------------+ | | | | 228| | | | | +--------------+ | 229| User +---------+ +------> | Internal API | | 230| | | | | +--------------+ | 231+------------+ | | | | 232 | | | | 233 | | +------------------------+ 234 | +----------+ | VMM 235 | CLI | | | 236 +----------->+ clap +--------------+ 237 | | 238 +----------+ 239 240 241``` 242 243## Internal API 244 245The Cloud Hypervisor internal API, as its name suggests, is used internally 246by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 247send commands and responses to each others. 248 249It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 250and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 251control loop. 252 253API producers are the HTTP thread handling the [REST API](#rest-api) and the 254main thread that initially parses the [CLI](#command-line-interface). 255 256### Goals and Design 257 258The internal API is designed for controlling, managing and inspecting a Cloud 259Hypervisor VMM and its guest. It is a backend for handling external, user 260visible requests through either the [REST API](#rest-api) or the 261[CLI](#command-line-interface) interfaces. 262 263The API follows a command-response scheme that closely maps the [REST API](#rest-api). 264Any command must be replied to with a response. 265 266Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 267are received and processed by the VMM control loop. 268 269In order for the VMM control loop to respond to any internal API command, it 270must be able to send a response back to the MPSC sender. For that purpose, all 271internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 272end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 273 274The sender of any internal API command is therefore responsible for: 275 2761. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 277 channel. 2781. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 279 end of the response channel as part of the internal API command payload. 2801. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 281 end of the response channel. 282 283## End to End Example 284 285In order to further understand how the external and internal Cloud Hypervisor 286APIs work together, let's look at a complete VM creation flow, from the 287[REST API](#rest-api) call, to the reply the external user will receive: 288 2891. A user or operator sends an HTTP request to the Cloud Hypervisor 290 [REST API](#rest-api) in order to creates a virtual machine: 291 ``` 292 shell 293 #!/bin/bash 294 295 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 296 -X PUT 'http://localhost/api/v1/vm.create' \ 297 -H 'Accept: application/json' \ 298 -H 'Content-Type: application/json' \ 299 -d '{ 300 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 301 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 302 "cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"}, 303 "disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}], 304 "rng":{"src":"/dev/urandom"}, 305 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 306 }' 307 ``` 3081. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 309 HTTP request JSON body into an internal `VmConfig` structure. 3101. The Cloud Hypervisor HTTP thread creates an 311 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 312 server to send its response back. 3131. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 314 virtual machine. The command's payload is made of the de-serialized 315 `VmConfig` structure and the response channel: 316 ```Rust 317 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 318 ``` 3191. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 320 for the response: 321 ```Rust 322 // Send the VM creation request. 323 api_sender 324 .send(ApiRequest::VmCreate(config, response_sender)) 325 .map_err(ApiError::RequestSend)?; 326 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 327 328 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 329 ``` 3301. The Cloud Hypervisor control loop receives the command, as it listens on the 331 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 332 ```Rust 333 // Read from the API receiver channel 334 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 335 ``` 3361. The Cloud Hypervisor control loop matches the received internal API against 337 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 338 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 339 command payload. It stores the `VmConfig` structure and replies back to the 340 sender ((The HTTP thread): 341 ```Rust 342 match api_request { 343 ApiRequest::VmCreate(config, sender) => { 344 // We only store the passed VM config. 345 // The VM will be created when being asked to boot it. 346 let response = if self.vm_config.is_none() { 347 self.vm_config = Some(config); 348 Ok(ApiResponsePayload::Empty) 349 } else { 350 Err(ApiError::VmAlreadyCreated) 351 }; 352 353 sender.send(response).map_err(Error::ApiResponseSend)?; 354 } 355 ``` 3561. The Cloud Hypervisor HTTP thread receives the internal API command response 357 as the return value from its `VmCreate` HTTP handler. Depending on the 358 control loop internal API response, it generates the appropriate HTTP 359 response: 360 ```Rust 361 // Call vm_create() 362 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 363 .map_err(HttpError::VmCreate) 364 { 365 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 366 Err(e) => error_response(e, StatusCode::InternalServerError), 367 } 368 ``` 3691. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 370 user. This is abstracted by the 371 [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 372 crate. 373 374