xref: /cloud-hypervisor/docs/api.md (revision 8c92d1dbdc7617b88d34687115e0e69bd78b332c)
1- [Cloud Hypervisor API](#cloud-hypervisor-api)
2  * [External API](#external-api)
3    + [REST API](#rest-api)
4      - [Location and availability](#location-and-availability)
5      - [Endpoints](#endpoints)
6		* [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions)
7		* [Virtual Machine (VM) Actions](#virtual-machine-vm-actions)
8      - [REST API Examples](#rest-api-examples)
9        * [Create a Virtual Machine](#create-a-virtual-machine)
10	    * [Boot a Virtual Machine](#boot-a-virtual-machine)
11        * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information)
12        * [Reboot a Virtual Machine](#reboot-a-virtual-machine)
13        * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down)
14    + [Command Line Interface](#command-line-interface)
15    + [REST API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship)
16  * [Internal API](#internal-api)
17    + [Goals and Design](#goals-and-design)
18  * [End to End Example](#end-to-end-example)
19
20# Cloud Hypervisor API
21
22The Cloud Hypervisor API is made of 2 distinct interfaces:
23
241. **The external API**. This is the user facing API. Users and operators can
25   control and manage Cloud Hypervisor through either a REST API or a Command
26   Line Interface (CLI).
271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/)
28   module. This API is used internally by the Cloud Hypervisor threads to
29   communicate between each others.
30
31The goal of this document is to describe the Cloud Hypervisor API as a whole,
32and to outline how the internal and external APIs are architecturally related.
33
34## External API
35
36### REST API
37
38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer)
39API triggers VM and VMM specific actions, and as such it is designed as a
40collection of RPC-style, static methods.
41
42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md)
43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml)
44document for more details about the API payloads and responses.
45
46### Location and availability
47
48The REST API is available as soon as the Cloud Hypervisor binary is started,
49through a local UNIX socket.
50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`.
51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is
52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`.
53
54The REST API default URL can be overridden through the Cloud Hypervisor
55option `--api-socket`:
56
57```
58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
59Cloud Hypervisor Guest
60    API server: /tmp/cloud-hypervisor.sock
61    vCPUs: 1
62    Memory: 512 MB
63    Kernel: None
64    Kernel cmdline:
65    Disk(s): None
66```
67
68### Endpoints
69
70The Cloud Hypervisor API exposes the following actions through its endpoints:
71
72#### Virtual Machine Manager (VMM) Actions
73
74Action                              | Endpoint        | Request Body | Response Body              | Prerequisites
75------------------------------------|-----------------|--------------|----------------------------|---------------------------
76Check for the REST API availability | `/vmm.ping`     | N/A          | `/schemas/VmmPingResponse` | N/A
77Shut the VMM down                   | `/vmm.shutdown` | N/A          | N/A                        | The VMM is running
78
79#### Virtual Machine (VM) Actions
80
81Action                             | Endpoint            | Request Body              | Response Body            | Prerequisites
82-----------------------------------|---------------------|---------------------------|--------------------------|---------------------------
83Create the VM                      | `/vm.create`        | `/schemas/VmConfig`       | N/A                      | The VM is not created yet
84Delete the VM                      | `/vm.delete`        | N/A                       | N/A                      | N/A
85Boot the VM                        | `/vm.boot`          | N/A                       | N/A                      | The VM is created but not booted
86Shut the VM down                   | `/vm.shutdown`      | N/A                       | N/A                      | The VM is booted
87Reboot the VM                      | `/vm.reboot`        | N/A                       | N/A                      | The VM is booted
88Pause the VM                       | `/vm.pause`         | N/A                       | N/A                      | The VM is booted
89Resume the VM                      | `/vm.resume`        | N/A                       | N/A                      | The VM is paused
90Add/remove CPUs to/from the VM     | `/vm.resize`        | `/schemas/VmResize`       | N/A                      | The VM is booted
91Add/remove memory from the VM      | `/vm.resize`        | `/schemas/VmResize`       | N/A                      | The VM is booted
92Add/remove memory from a zone      | `/vm.resize-zone`   | `/schemas/VmResizeZone`   | N/A                      | The VM is booted
93Dump the VM information            | `/vm.info`          | N/A                       | `/schemas/VmInfo`        | The VM is created
94Add VFIO PCI device to the VM      | `/vm.add-device`    | `/schemas/VmAddDevice`    | `/schemas/PciDeviceInfo` | The VM is booted
95Add disk device to the VM          | `/vm.add-disk`      | `/schemas/DiskConfig`     | `/schemas/PciDeviceInfo` | The VM is booted
96Add fs device to the VM            | `/vm.add-fs`        | `/schemas/FsConfig`       | `/schemas/PciDeviceInfo` | The VM is booted
97Add pmem device to the VM          | `/vm.add-pmem`      | `/schemas/PmemConfig`     | `/schemas/PciDeviceInfo` | The VM is booted
98Add network device to the VM       | `/vm.add-net`       | `/schemas/NetConfig`      | `/schemas/PciDeviceInfo` | The VM is booted
99Add vsock device to the VM         | `/vm.add-vsock`     | `/schemas/VsockConfig`    | `/schemas/PciDeviceInfo` | The VM is booted
100Remove device from the VM          | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A                      | The VM is booted
101Dump the VM counters               | `/vm.counters`      | N/A                       | `/schemas/VmCounters`    | The VM is booted
102
103### REST API Examples
104
105For the following set of examples, we assume Cloud Hypervisor is started with
106the REST API available at `/tmp/cloud-hypervisor.sock`:
107
108```
109$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
110Cloud Hypervisor Guest
111    API server: /tmp/cloud-hypervisor.sock
112    vCPUs: 1
113    Memory: 512 MB
114    Kernel: None
115    Kernel cmdline:
116    Disk(s): None
117```
118
119#### Create a Virtual Machine
120
121We want to create a virtual machine with the following characteristics:
122
123* 4 vCPUs
124* 1 GB of RAM
125* 1 virtio based networking interface
126* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at
127  `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu`
128* Using a Ubuntu image as its root filesystem, located at
129  `/opt/clh/images/focal-server-cloudimg-amd64.raw`
130
131```shell
132#!/bin/bash
133
134curl --unix-socket /tmp/cloud-hypervisor.sock -i \
135     -X PUT 'http://localhost/api/v1/vm.create'  \
136     -H 'Accept: application/json'               \
137     -H 'Content-Type: application/json'         \
138     -d '{
139         "cpus":{"boot_vcpus": 4, "max_vcpus": 4},
140         "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"},
141         "cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"},
142         "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}],
143         "rng":{"src":"/dev/urandom"},
144         "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}]
145         }'
146```
147
148#### Boot a Virtual Machine
149
150Once the VM is created, we can boot it:
151
152```shell
153#!/bin/bash
154
155curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot'
156```
157
158#### Dump a Virtual Machine Information
159
160We can fetch information about any VM, as soon as it's created:
161
162```shell
163#!/bin/bash
164
165curl --unix-socket /tmp/cloud-hypervisor.sock -i \
166     -X GET 'http://localhost/api/v1/vm.info' \
167     -H 'Accept: application/json'
168```
169
170#### Reboot a Virtual Machine
171
172We can reboot a VM that's already booted:
173
174```shell
175#!/bin/bash
176
177curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot'
178```
179
180#### Shut a Virtual Machine Down
181
182Once booted, we can shut a VM down from the REST API:
183
184```shell
185#!/bin/bash
186
187curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown'
188```
189
190### Command Line Interface
191
192The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching
193the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or
194the launched VM once they're up and running.
195
196If you want to inspect the VMM, or control the VM after launching Cloud
197Hypervisor from the CLI, you must use the [REST API](#rest-api).
198
199From the CLI, one can either:
200
2011. Create and boot a complete virtual machine by using the CLI options to build
202   the VM config. Run `cloud-hypervisor --help` for a complete list of CLI
203   options. As soon as the `cloud-hypervisor` binary is launched, the
204   [REST API](#rest-api) is available for controlling and managing the VM.
2051. Start the [REST API](#rest-api) server only, by not passing any VM
206   configuration options. The VM can then be asynchronously created and booted
207   by sending HTTP commands to the [REST API](#rest-api). Check the
208   [REST API examples](#rest-api-examples) section for more details.
209
210### REST API and CLI Architectural Relationship
211
212The REST API and the CLI both rely on a common, [internal API](#internal-api).
213
214The CLI options are parsed by the
215[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into
216[internal API](#internal-api) commands.
217
218The REST API is processed by an HTTP thread using the
219[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http)
220crate. As with the CLI, the HTTP requests eventually get translated into
221[internal API](#internal-api) commands.
222
223As a summary, the REST API and the CLI are essentially frontends for the
224[internal API](#internal-api):
225
226```
227                                  +------------------+
228                        REST API  |                  |
229                       +--------->+    micro_http    +--------+
230                       |          |                  |        |
231                       |          +------------------+        |
232                       |                                      |      +------------------------+
233                       |                                      |      |                        |
234+------------+         |                                      |      |                        |
235|            |         |                                      |      | +--------------+       |
236|    User    +---------+                                      +------> | Internal API |       |
237|            |         |                                      |      | +--------------+       |
238+------------+         |                                      |      |                        |
239                       |                                      |      |                        |
240                       |                                      |      +------------------------+
241                       |            +----------+              |                 VMM
242                       |     CLI    |          |              |
243                       +----------->+   clap   +--------------+
244                                    |          |
245                                    +----------+
246
247
248```
249
250## Internal API
251
252The Cloud Hypervisor internal API, as its name suggests, is used internally
253by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to
254send commands and responses to each others.
255
256It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/),
257and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor
258control loop.
259
260API producers are the HTTP thread handling the [REST API](#rest-api) and the
261main thread that initially parses the [CLI](#command-line-interface).
262
263### Goals and Design
264
265The internal API is designed for controlling, managing and inspecting a Cloud
266Hypervisor VMM and its guest. It is a backend for handling external, user
267visible requests through either the [REST API](#rest-api) or the
268[CLI](#command-line-interface) interfaces.
269
270The API follows a command-response scheme that closely maps the [REST API](#rest-api).
271Any command must be replied to with a response.
272
273Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and
274are received and processed by the VMM control loop.
275
276In order for the VMM control loop to respond to any internal API command, it
277must be able to send a response back to the MPSC sender. For that purpose, all
278internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html)
279end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel.
280
281The sender of any internal API command is therefore responsible for:
282
2831. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response
284   channel.
2851. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html)
286   end of the response channel as part of the internal API command payload.
2871. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html)
288   end of the response channel.
289
290## End to End Example
291
292In order to further understand how the external and internal Cloud Hypervisor
293APIs work together, let's look at a complete VM creation flow, from the
294[REST API](#rest-api) call, to the reply the external user will receive:
295
2961. A user or operator sends an HTTP request to the Cloud Hypervisor
297   [REST API](#rest-api) in order to creates a virtual machine:
298   ```
299   shell
300   #!/bin/bash
301
302	curl --unix-socket /tmp/cloud-hypervisor.sock -i \
303		-X PUT 'http://localhost/api/v1/vm.create'  \
304		-H 'Accept: application/json'               \
305		-H 'Content-Type: application/json'         \
306		-d '{
307			"cpus":{"boot_vcpus": 4, "max_vcpus": 4},
308			"kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"},
309			"cmdline":{"args":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"},
310			"disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}],
311			"rng":{"src":"/dev/urandom"},
312			"net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}]
313			}'
314   ```
3151. The Cloud Hypervisor HTTP thread processes the request and de-serializes the
316   HTTP request JSON body into an internal `VmConfig` structure.
3171. The Cloud Hypervisor HTTP thread creates an
318   [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API
319   server to send its response back.
3201. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a
321   virtual machine. The command's payload is made of the de-serialized
322   `VmConfig` structure and the response channel:
323   ```Rust
324   VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>)
325   ```
3261. The Cloud Hypervisor HTTP thread sends the internal API command, and waits
327   for the response:
328   ```Rust
329   // Send the VM creation request.
330    api_sender
331        .send(ApiRequest::VmCreate(config, response_sender))
332        .map_err(ApiError::RequestSend)?;
333    api_evt.write(1).map_err(ApiError::EventFdWrite)?;
334
335    response_receiver.recv().map_err(ApiError::ResponseRecv)??;
336   ```
3371. The Cloud Hypervisor control loop receives the command, as it listens on the
338   internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel:
339   ```Rust
340   // Read from the API receiver channel
341   let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?;
342   ```
3431. The Cloud Hypervisor control loop matches the received internal API against
344   the `VmCreate` payload, and extracts both the `VmConfig` structure and the
345   [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the
346   command payload. It stores the `VmConfig` structure and replies back to the
347   sender ((The HTTP thread):
348   ```Rust
349   match api_request {
350	   ApiRequest::VmCreate(config, sender) => {
351		   // We only store the passed VM config.
352		   // The VM will be created when being asked to boot it.
353		   let response = if self.vm_config.is_none() {
354			   self.vm_config = Some(config);
355			   Ok(ApiResponsePayload::Empty)
356		   } else {
357			   Err(ApiError::VmAlreadyCreated)
358		   };
359
360	       sender.send(response).map_err(Error::ApiResponseSend)?;
361	   }
362   ```
3631. The Cloud Hypervisor HTTP thread receives the internal API command response
364   as the return value from its `VmCreate` HTTP handler. Depending on the
365   control loop internal API response, it generates the appropriate HTTP
366   response:
367   ```Rust
368   // Call vm_create()
369   match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config)))
370	   .map_err(HttpError::VmCreate)
371   {
372	   Ok(_) => Response::new(Version::Http11, StatusCode::NoContent),
373	   Err(e) => error_response(e, StatusCode::InternalServerError),
374   }
375   ```
3761. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the
377   user. This is abstracted by the
378   [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http)
379   crate.
380