1- [Cloud Hypervisor API](#cloud-hypervisor-api) 2 * [External API](#external-api) 3 + [REST API](#rest-api) 4 - [Location and availability](#location-and-availability) 5 - [Endpoints](#endpoints) 6 * [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions) 7 * [Virtual Machine (VM) Actions](#virtual-machine-vm-actions) 8 - [REST API Examples](#rest-api-examples) 9 * [Create a Virtual Machine](#create-a-virtual-machine) 10 * [Boot a Virtual Machine](#boot-a-virtual-machine) 11 * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information) 12 * [Reboot a Virtual Machine](#reboot-a-virtual-machine) 13 * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down) 14 + [Command Line Interface](#command-line-interface) 15 + [REST API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship) 16 * [Internal API](#internal-api) 17 + [Goals and Design](#goals-and-design) 18 * [End to End Example](#end-to-end-example) 19 20# Cloud Hypervisor API 21 22The Cloud Hypervisor API is made of 2 distinct interfaces: 23 241. **The external API**. This is the user facing API. Users and operators can 25 control and manage Cloud Hypervisor through either a REST API or a Command 26 Line Interface (CLI). 271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/) 28 module. This API is used internally by the Cloud Hypervisor threads to 29 communicate between each others. 30 31The goal of this document is to describe the Cloud Hypervisor API as a whole, 32and to outline how the internal and external APIs are architecturally related. 33 34## External API 35 36### REST API 37 38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) 39API triggers VM and VMM specific actions, and as such it is designed as a 40collection of RPC-style, static methods. 41 42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md) 43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml) 44document for more details about the API payloads and responses. 45 46### Location and availability 47 48The REST API is available as soon as the Cloud Hypervisor binary is started, 49through a local UNIX socket. 50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`. 51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is 52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`. 53 54The REST API default URL can be overridden through the Cloud Hypervisor 55option `--api-socket`: 56 57``` 58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 59Cloud Hypervisor Guest 60 API server: /tmp/cloud-hypervisor.sock 61 vCPUs: 1 62 Memory: 512 MB 63 Kernel: None 64 Kernel cmdline: 65 Disk(s): None 66``` 67 68### Endpoints 69 70The Cloud Hypervisor API exposes the following actions through its endpoints: 71 72#### Virtual Machine Manager (VMM) Actions 73 74Action | Endpoint | Request Body | Response Body | Prerequisites 75------------------------------------|-----------------|--------------|----------------------------|--------------------------- 76Check for the REST API availability | `/vmm.ping` | N/A | `/schemas/VmmPingResponse` | N/A 77Shut the VMM down | `/vmm.shutdown` | N/A | N/A | The VMM is running 78 79#### Virtual Machine (VM) Actions 80 81Action | Endpoint | Request Body | Response Body | Prerequisites 82-----------------------------------|---------------------|---------------------------|--------------------------|--------------------------- 83Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet 84Delete the VM | `/vm.delete` | N/A | N/A | N/A 85Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted 86Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted 87Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted 88Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted 89Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused 90Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 91Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted 92Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted 93Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created 94Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted 95Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted 96Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted 97Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted 98Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted 99Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted 100Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted 101Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted 102 103### REST API Examples 104 105For the following set of examples, we assume Cloud Hypervisor is started with 106the REST API available at `/tmp/cloud-hypervisor.sock`: 107 108``` 109$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock 110Cloud Hypervisor Guest 111 API server: /tmp/cloud-hypervisor.sock 112 vCPUs: 1 113 Memory: 512 MB 114 Kernel: None 115 Kernel cmdline: 116 Disk(s): None 117``` 118 119#### Create a Virtual Machine 120 121We want to create a virtual machine with the following characteristics: 122 123* 4 vCPUs 124* 1 GB of RAM 125* 1 virtio based networking interface 126* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at 127 `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu` 128* Using a Ubuntu image as its root filesystem, located at 129 `/opt/clh/images/focal-server-cloudimg-amd64.raw` 130 131```shell 132#!/bin/bash 133 134curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 135 -X PUT 'http://localhost/api/v1/vm.create' \ 136 -H 'Accept: application/json' \ 137 -H 'Content-Type: application/json' \ 138 -d '{ 139 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 140 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 141 "cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 142 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 143 "rng":{"src":"/dev/urandom"}, 144 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 145 }' 146``` 147 148#### Boot a Virtual Machine 149 150Once the VM is created, we can boot it: 151 152```shell 153#!/bin/bash 154 155curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot' 156``` 157 158#### Dump a Virtual Machine Information 159 160We can fetch information about any VM, as soon as it's created: 161 162```shell 163#!/bin/bash 164 165curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 166 -X GET 'http://localhost/api/v1/vm.info' \ 167 -H 'Accept: application/json' 168``` 169 170#### Reboot a Virtual Machine 171 172We can reboot a VM that's already booted: 173 174```shell 175#!/bin/bash 176 177curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot' 178``` 179 180#### Shut a Virtual Machine Down 181 182Once booted, we can shut a VM down from the REST API: 183 184```shell 185#!/bin/bash 186 187curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown' 188``` 189 190### Command Line Interface 191 192The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching 193the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or 194the launched VM once they're up and running. 195 196If you want to inspect the VMM, or control the VM after launching Cloud 197Hypervisor from the CLI, you must use the [REST API](#rest-api). 198 199From the CLI, one can either: 200 2011. Create and boot a complete virtual machine by using the CLI options to build 202 the VM config. Run `cloud-hypervisor --help` for a complete list of CLI 203 options. As soon as the `cloud-hypervisor` binary is launched, the 204 [REST API](#rest-api) is available for controlling and managing the VM. 2051. Start the [REST API](#rest-api) server only, by not passing any VM 206 configuration options. The VM can then be asynchronously created and booted 207 by sending HTTP commands to the [REST API](#rest-api). Check the 208 [REST API examples](#rest-api-examples) section for more details. 209 210### REST API and CLI Architectural Relationship 211 212The REST API and the CLI both rely on a common, [internal API](#internal-api). 213 214The CLI options are parsed by the 215[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into 216[internal API](#internal-api) commands. 217 218The REST API is processed by an HTTP thread using the 219[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 220crate. As with the CLI, the HTTP requests eventually get translated into 221[internal API](#internal-api) commands. 222 223As a summary, the REST API and the CLI are essentially frontends for the 224[internal API](#internal-api): 225 226``` 227 +------------------+ 228 REST API | | 229 +--------->+ micro_http +--------+ 230 | | | | 231 | +------------------+ | 232 | | +------------------------+ 233 | | | | 234+------------+ | | | | 235| | | | | +--------------+ | 236| User +---------+ +------> | Internal API | | 237| | | | | +--------------+ | 238+------------+ | | | | 239 | | | | 240 | | +------------------------+ 241 | +----------+ | VMM 242 | CLI | | | 243 +----------->+ clap +--------------+ 244 | | 245 +----------+ 246 247 248``` 249 250## Internal API 251 252The Cloud Hypervisor internal API, as its name suggests, is used internally 253by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to 254send commands and responses to each others. 255 256It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/), 257and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor 258control loop. 259 260API producers are the HTTP thread handling the [REST API](#rest-api) and the 261main thread that initially parses the [CLI](#command-line-interface). 262 263### Goals and Design 264 265The internal API is designed for controlling, managing and inspecting a Cloud 266Hypervisor VMM and its guest. It is a backend for handling external, user 267visible requests through either the [REST API](#rest-api) or the 268[CLI](#command-line-interface) interfaces. 269 270The API follows a command-response scheme that closely maps the [REST API](#rest-api). 271Any command must be replied to with a response. 272 273Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and 274are received and processed by the VMM control loop. 275 276In order for the VMM control loop to respond to any internal API command, it 277must be able to send a response back to the MPSC sender. For that purpose, all 278internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 279end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel. 280 281The sender of any internal API command is therefore responsible for: 282 2831. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response 284 channel. 2851. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) 286 end of the response channel as part of the internal API command payload. 2871. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html) 288 end of the response channel. 289 290## End to End Example 291 292In order to further understand how the external and internal Cloud Hypervisor 293APIs work together, let's look at a complete VM creation flow, from the 294[REST API](#rest-api) call, to the reply the external user will receive: 295 2961. A user or operator sends an HTTP request to the Cloud Hypervisor 297 [REST API](#rest-api) in order to creates a virtual machine: 298 ``` 299 shell 300 #!/bin/bash 301 302 curl --unix-socket /tmp/cloud-hypervisor.sock -i \ 303 -X PUT 'http://localhost/api/v1/vm.create' \ 304 -H 'Accept: application/json' \ 305 -H 'Content-Type: application/json' \ 306 -d '{ 307 "cpus":{"boot_vcpus": 4, "max_vcpus": 4}, 308 "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"}, 309 "cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"}, 310 "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}], 311 "rng":{"src":"/dev/urandom"}, 312 "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}] 313 }' 314 ``` 3151. The Cloud Hypervisor HTTP thread processes the request and de-serializes the 316 HTTP request JSON body into an internal `VmConfig` structure. 3171. The Cloud Hypervisor HTTP thread creates an 318 [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API 319 server to send its response back. 3201. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a 321 virtual machine. The command's payload is made of the de-serialized 322 `VmConfig` structure and the response channel: 323 ```Rust 324 VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>) 325 ``` 3261. The Cloud Hypervisor HTTP thread sends the internal API command, and waits 327 for the response: 328 ```Rust 329 // Send the VM creation request. 330 api_sender 331 .send(ApiRequest::VmCreate(config, response_sender)) 332 .map_err(ApiError::RequestSend)?; 333 api_evt.write(1).map_err(ApiError::EventFdWrite)?; 334 335 response_receiver.recv().map_err(ApiError::ResponseRecv)??; 336 ``` 3371. The Cloud Hypervisor control loop receives the command, as it listens on the 338 internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel: 339 ```Rust 340 // Read from the API receiver channel 341 let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?; 342 ``` 3431. The Cloud Hypervisor control loop matches the received internal API against 344 the `VmCreate` payload, and extracts both the `VmConfig` structure and the 345 [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the 346 command payload. It stores the `VmConfig` structure and replies back to the 347 sender ((The HTTP thread): 348 ```Rust 349 match api_request { 350 ApiRequest::VmCreate(config, sender) => { 351 // We only store the passed VM config. 352 // The VM will be created when being asked to boot it. 353 let response = if self.vm_config.is_none() { 354 self.vm_config = Some(config); 355 Ok(ApiResponsePayload::Empty) 356 } else { 357 Err(ApiError::VmAlreadyCreated) 358 }; 359 360 sender.send(response).map_err(Error::ApiResponseSend)?; 361 } 362 ``` 3631. The Cloud Hypervisor HTTP thread receives the internal API command response 364 as the return value from its `VmCreate` HTTP handler. Depending on the 365 control loop internal API response, it generates the appropriate HTTP 366 response: 367 ```Rust 368 // Call vm_create() 369 match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config))) 370 .map_err(HttpError::VmCreate) 371 { 372 Ok(_) => Response::new(Version::Http11, StatusCode::NoContent), 373 Err(e) => error_response(e, StatusCode::InternalServerError), 374 } 375 ``` 3761. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the 377 user. This is abstracted by the 378 [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http) 379 crate. 380