xref: /cloud-hypervisor/docs/api.md (revision 1c9df58440b92efbcc400e152554b28d8f27191b)
1- [Cloud Hypervisor API](#cloud-hypervisor-api)
2  - [External API](#external-api)
3    - [REST API](#rest-api)
4      - [REST API Location and availability](#rest-api-location-and-availability)
5      - [REST API Endpoints](#rest-api-endpoints)
6        - [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions)
7        - [Virtual Machine (VM) Actions](#virtual-machine-vm-actions)
8      - [REST API Examples](#rest-api-examples)
9        - [Create a Virtual Machine](#create-a-virtual-machine)
10        - [Boot a Virtual Machine](#boot-a-virtual-machine)
11        - [Dump a Virtual Machine Information](#dump-a-virtual-machine-information)
12        - [Reboot a Virtual Machine](#reboot-a-virtual-machine)
13        - [Shut a Virtual Machine Down](#shut-a-virtual-machine-down)
14    - [D-Bus API](#d-bus-api)
15      - [D-Bus API Location and availability](#d-bus-api-location-and-availability)
16      - [D-Bus API Interface](#d-bus-api-interface)
17    - [Command Line Interface](#command-line-interface)
18    - [REST API, D-Bus API and CLI Architectural Relationship](#rest-api-and-cli-architectural-relationship)
19  - [Internal API](#internal-api)
20    - [Goals and Design](#goals-and-design)
21  - [End to End Example](#end-to-end-example)
22
23# Cloud Hypervisor API
24
25The Cloud Hypervisor API is made of 2 distinct interfaces:
26
271. **The External API** This is the user facing API. Users and operators
28   can control and manage the Cloud Hypervisor through various options
29   including a REST API, a Command Line Interface (CLI) or a D-Bus based API,
30   which is not compiled into Cloud Hypervisor by default.
31
321. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/)
33   module. This API is used internally by the Cloud Hypervisor threads to
34   communicate between each others.
35
36The goal of this document is to describe the Cloud Hypervisor API as a whole,
37and to outline how the internal and external APIs are architecturally related.
38
39## External API
40
41### REST API
42
43The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer)
44API triggers VM and VMM specific actions, and as such it is designed as a
45collection of RPC-style, static methods.
46
47The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md)
48compliant. Please consult the [Cloud Hypervisor OpenAPI Document](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml)
49for more details about the API payloads and responses.
50
51#### REST API Location and availability
52
53The REST API is available as soon as the Cloud Hypervisor binary is started,
54through a local UNIX socket.
55By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`.
56For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is
57123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`.
58
59The REST API default URL can be overridden through the Cloud Hypervisor
60option `--api-socket`:
61
62```
63$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
64Cloud Hypervisor Guest
65    API server: /tmp/cloud-hypervisor.sock
66    vCPUs: 1
67    Memory: 512 MB
68    Kernel: None
69    Kernel cmdline:
70    Disk(s): None
71```
72
73#### REST API Endpoints
74
75The Cloud Hypervisor API exposes the following actions through its endpoints:
76
77##### Virtual Machine Manager (VMM) Actions
78
79| Action                              | Endpoint        | Request Body | Response Body              | Prerequisites      |
80| ----------------------------------- | --------------- | ------------ | -------------------------- | ------------------ |
81| Check for the REST API availability | `/vmm.ping`     | N/A          | `/schemas/VmmPingResponse` | N/A                |
82| Shut the VMM down                   | `/vmm.shutdown` | N/A          | N/A                        | The VMM is running |
83
84##### Virtual Machine (VM) Actions
85
86| Action                             | Endpoint                | Request Body                    | Response Body            | Prerequisites                                          |
87| ---------------------------------- | ----------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------ |
88| Create the VM                      | `/vm.create`            | `/schemas/VmConfig`             | N/A                      | The VM is not created yet                              |
89| Delete the VM                      | `/vm.delete`            | N/A                             | N/A                      | N/A                                                    |
90| Boot the VM                        | `/vm.boot`              | N/A                             | N/A                      | The VM is created but not booted                       |
91| Shut the VM down                   | `/vm.shutdown`          | N/A                             | N/A                      | The VM is booted                                       |
92| Reboot the VM                      | `/vm.reboot`            | N/A                             | N/A                      | The VM is booted                                       |
93| Trigger power button of the VM     | `/vm.power-button`      | N/A                             | N/A                      | The VM is booted                                       |
94| Pause the VM                       | `/vm.pause`             | N/A                             | N/A                      | The VM is booted                                       |
95| Resume the VM                      | `/vm.resume`            | N/A                             | N/A                      | The VM is paused                                       |
96| Task a snapshot of the VM          | `/vm.snapshot`          | `/schemas/VmSnapshotConfig`     | N/A                      | The VM is paused                                       |
97| Perform a coredump of the VM*      | `/vm.coredump`          | `/schemas/VmCoredumpData`       | N/A                      | The VM is paused                                       |
98| Restore the VM from a snapshot     | `/vm.restore`           | `/schemas/RestoreConfig`        | N/A                      | The VM is created but not booted                       |
99| Add/remove CPUs to/from the VM     | `/vm.resize`            | `/schemas/VmResize`             | N/A                      | The VM is booted                                       |
100| Add/remove memory from the VM      | `/vm.resize`            | `/schemas/VmResize`             | N/A                      | The VM is booted                                       |
101| Add/remove memory from a zone      | `/vm.resize-zone`       | `/schemas/VmResizeZone`         | N/A                      | The VM is booted                                       |
102| Dump the VM information            | `/vm.info`              | N/A                             | `/schemas/VmInfo`        | The VM is created                                      |
103| Add VFIO PCI device to the VM      | `/vm.add-device`        | `/schemas/VmAddDevice`          | `/schemas/PciDeviceInfo` | The VM is booted                                       |
104| Add disk device to the VM          | `/vm.add-disk`          | `/schemas/DiskConfig`           | `/schemas/PciDeviceInfo` | The VM is booted                                       |
105| Add fs device to the VM            | `/vm.add-fs`            | `/schemas/FsConfig`             | `/schemas/PciDeviceInfo` | The VM is booted                                       |
106| Add pmem device to the VM          | `/vm.add-pmem`          | `/schemas/PmemConfig`           | `/schemas/PciDeviceInfo` | The VM is booted                                       |
107| Add network device to the VM       | `/vm.add-net`           | `/schemas/NetConfig`            | `/schemas/PciDeviceInfo` | The VM is booted                                       |
108| Add userspace PCI device to the VM | `/vm.add-user-device`   | `/schemas/VmAddUserDevice`      | `/schemas/PciDeviceInfo` | The VM is booted                                       |
109| Add vdpa device to the VM          | `/vm.add-vdpa`          | `/schemas/VdpaConfig`           | `/schemas/PciDeviceInfo` | The VM is booted                                       |
110| Add vsock device to the VM         | `/vm.add-vsock`         | `/schemas/VsockConfig`          | `/schemas/PciDeviceInfo` | The VM is booted                                       |
111| Remove device from the VM          | `/vm.remove-device`     | `/schemas/VmRemoveDevice`       | N/A                      | The VM is booted                                       |
112| Dump the VM counters               | `/vm.counters`          | N/A                             | `/schemas/VmCounters`    | The VM is booted                                       |
113| Prepare to receive a migration     | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A                      | N/A                                                    |
114| Start to send migration to target  | `/vm.send-migration`    | `/schemas/SendMigrationData`    | N/A                      | The VM is booted and (shared mem or hugepages enabled) |
115
116* The `vmcoredump` action is available exclusively for the `x86_64`
117architecture and can be executed only when the `guest_debug` feature is
118enabled. Without this feature, the corresponding [REST API](#rest-api) or
119[D-Bus API](#d-bus-api) endpoints are not available.
120
121#### REST API Examples
122
123For the following set of examples, we assume Cloud Hypervisor is started with
124the REST API available at `/tmp/cloud-hypervisor.sock`:
125
126```
127$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
128Cloud Hypervisor Guest
129    API server: /tmp/cloud-hypervisor.sock
130    vCPUs: 1
131    Memory: 512 MB
132    Kernel: None
133    Kernel cmdline:
134    Disk(s): None
135```
136
137##### Create a Virtual Machine
138
139We want to create a virtual machine with the following characteristics:
140
141* 4 vCPUs
142* 1 GB of RAM
143* 1 virtio based networking interface
144* Direct kernel boot from a custom 5.6.0-rc4 Linux kernel located at
145  `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu`
146* Using a Ubuntu image as its root filesystem, located at
147  `/opt/clh/images/focal-server-cloudimg-amd64.raw`
148
149```shell
150#!/bin/bash
151
152curl --unix-socket /tmp/cloud-hypervisor.sock -i \
153     -X PUT 'http://localhost/api/v1/vm.create'  \
154     -H 'Accept: application/json'               \
155     -H 'Content-Type: application/json'         \
156     -d '{
157         "cpus":{"boot_vcpus": 4, "max_vcpus": 4},
158         "payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"},
159         "disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}],
160         "rng":{"src":"/dev/urandom"},
161         "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}]
162         }'
163```
164
165##### Boot a Virtual Machine
166
167Once the VM is created, we can boot it:
168
169```shell
170#!/bin/bash
171
172curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot'
173```
174
175##### Dump a Virtual Machine Information
176
177We can fetch information about any VM, as soon as it's created:
178
179```shell
180#!/bin/bash
181
182curl --unix-socket /tmp/cloud-hypervisor.sock -i \
183     -X GET 'http://localhost/api/v1/vm.info' \
184     -H 'Accept: application/json'
185```
186
187##### Reboot a Virtual Machine
188
189We can reboot a VM that's already booted:
190
191```shell
192#!/bin/bash
193
194curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot'
195```
196
197##### Shut a Virtual Machine Down
198
199Once booted, we can shut a VM down from the REST API:
200
201```shell
202#!/bin/bash
203
204curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown'
205```
206
207### D-Bus API
208
209Cloud Hypervisor offers a D-Bus API as an alternative to its REST API. As of
210writing this document, the D-Bus API mirrors the functionality of the REST
211API and shares the same set of endpoints, meaning that it supports every call
212that is supported by the REST API and can be a drop-in replacement since it
213also consumes/produces JSON.
214
215#### D-Bus API Location and availability
216
217This feature is not compiled into Cloud Hypervisor by default. Users who
218wish to use the D-Bus API, must explicitly enable it with the `dbus_api`
219feature flag when compiling Cloud Hypervisor.
220
221```sh
222$ ./scripts/dev_cli.sh build --release --libc musl -- --features dbus_api
223```
224
225Once this feature is enabled, it can be configured with the following
226CLI options:
227
228```
229  --dbus-service-name
230                    well known name of the service
231  --dbus-object-path
232                    object path to serve the dbus interface
233  --dbus-system-bus use the system bus instead of a session bus
234```
235
236Example invocation:
237
238```sh
239$ ./cloud-hypervisor --dbus-service-name "org.cloudhypervisor.DBusApi" \
240                     --dbus-object-path "/org/cloudhypervisor/DBusApi"
241```
242
243This will start serving a service with the name `org.cloudhypervisor.DBusApi1`
244which in turn can be used to control and manage Cloud Hypervisor.
245
246#### D-Bus API Interface
247
248Please refer to the [REST API](#rest-api) documentation. As previously
249mentioned, the D-Bus API currently mirrors the behaviour of the REST API.
250
251### Command Line Interface
252
253The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching
254the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or
255the launched VM once they're up and running.
256
257If you want to inspect the VMM, or control the VM after launching Cloud
258Hypervisor from the CLI, you must use either the [REST API](#rest-api)
259or the [D-Bus API](#d-bus-api).
260
261From the CLI, one can:
262
2631. Create and boot a complete virtual machine by using the CLI options to build
264   the VM config. Run `cloud-hypervisor --help` for a complete list of CLI
265   options. As soon as the `cloud-hypervisor` binary is launched, contrary
266   to the [D-Bus API](#d-bus-api), the [REST API](#rest-api) is available
267   for controlling and managing the VM. The [D-Bus API](#d-bus-api) doesn't start
268   automatically and needs to be explicitly configured in order to be run.
2691. Start either the REST API, D-Bus API or both simultaneously without passing
270   any VM configuration options. The VM can then be asynchronously created and
271   booted by calling API methods of choice. It should be noted that one external
272   API does not exclude another; it is possible to have both the REST and D-Bus
273   APIs running simultaneously.
274
275### REST API, D-Bus API and CLI Architectural Relationship
276
277The REST API, D-Bus API and the CLI all rely on a common, [internal API](#internal-api).
278
279The CLI options are parsed by the
280[argh crate](https://docs.rs/argh/latest/argh/) and then translated into
281[internal API](#internal-api) commands.
282
283The REST API is processed by an HTTP thread using the
284[Firecracker's `micro_http`](https://github.com/firecracker-microvm/micro-http)
285crate. As with the CLI, the HTTP requests eventually get translated into
286[internal API](#internal-api) commands.
287
288The D-Bus API is implemented using the [zbus](https://github.com/dbus2/zbus)
289crate and runs in its own thread. Whenever it needs to call the [internal API](#internal-api),
290the [blocking](https://github.com/smol-rs/blocking) crate is used perform the call in zbus' async context.
291
292As a summary, the REST API, the D-Bus API and the CLI are essentially frontends for the
293[internal API](#internal-api):
294
295```
296                                  +------------------+
297                        REST API  |                  |
298                       +--------->+    micro_http    +--------+
299                       |          |                  |        |
300                       |          +------------------+        |
301                       |                                      |      +------------------------+
302                       |                                      |      |                        |
303+------------+         |            +----------+              |      |                        |
304|            |         |  D-Bus API |          |              |      | +--------------+       |
305|    User    +---------+----------->+   zbus   +--------------+------> | Internal API |       |
306|            |         |            |          |              |      | +--------------+       |
307+------------+         |            +----------+              |      |                        |
308                       |                                      |      |                        |
309                       |                                      |      +------------------------+
310                       |            +----------+              |                 VMM
311                       |     CLI    |          |              |
312                       +----------->+   argh   +--------------+
313                                    |          |
314                                    +----------+
315
316
317```
318
319## Internal API
320
321The Cloud Hypervisor internal API, as its name suggests, is used internally
322by the different Cloud Hypervisor threads (VMM, HTTP, D-Bus, control loop,
323etc) to send commands and responses to each others.
324
325It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/),
326and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor
327control loop.
328
329API producers are the HTTP thread handling the [REST API](#rest-api), the
330D-Bus thread handling the [D-Bus API](#d-bus-api) and the main thread that
331initially parses the [CLI](#command-line-interface).
332
333### Goals and Design
334
335The internal API is designed for controlling, managing and inspecting a Cloud
336Hypervisor VMM and its guest. It is a backend for handling external, user
337visible requests through the [REST API](#rest-api), the [D-Bus API](#d-bus-api)
338or the [CLI](#command-line-interface) interfaces.
339
340The API follows a command-response scheme that closely maps the [REST API](#rest-api).
341Any command must be replied to with a response.
342
343Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and
344are received and processed by the VMM control loop.
345
346In order for the VMM control loop to respond to any internal API command, it
347must be able to send a response back to the MPSC sender. For that purpose, all
348internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html)
349end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel.
350
351The sender of any internal API command is therefore responsible for:
352
3531. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response
354   channel.
3551. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html)
356   end of the response channel as part of the internal API command payload.
3571. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html)
358   end of the response channel.
359
360## End to End Example
361
362In order to further understand how the external and internal Cloud Hypervisor
363APIs work together, let's look at a complete VM creation flow, from the
364[REST API](#rest-api) call, to the reply the external user will receive:
365
3661. A user or operator sends an HTTP request to the Cloud Hypervisor
367   [REST API](#rest-api) in order to creates a virtual machine:
368   ```
369   shell
370   #!/bin/bash
371
372	curl --unix-socket /tmp/cloud-hypervisor.sock -i \
373		-X PUT 'http://localhost/api/v1/vm.create'  \
374		-H 'Accept: application/json'               \
375		-H 'Content-Type: application/json'         \
376		-d '{
377			"cpus":{"boot_vcpus": 4, "max_vcpus": 4},
378			"payload":{"kernel":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu", "cmdline":"console=ttyS0 console=hvc0 root=/dev/vda1 rw"},
379			"disks":[{"path":"/opt/clh/images/focal-server-cloudimg-amd64.raw"}],
380			"rng":{"src":"/dev/urandom"},
381			"net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}]
382			}'
383   ```
3841. The Cloud Hypervisor HTTP thread processes the request and de-serializes the
385   HTTP request JSON body into an internal `VmConfig` structure.
3861. The Cloud Hypervisor HTTP thread creates an
387   [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API
388   server to send its response back.
3891. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a
390   virtual machine. The command's payload is made of the de-serialized
391   `VmConfig` structure and the response channel:
392   ```Rust
393   VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>)
394   ```
3951. The Cloud Hypervisor HTTP thread sends the internal API command, and waits
396   for the response:
397   ```Rust
398   // Send the VM creation request.
399    api_sender
400        .send(ApiRequest::VmCreate(config, response_sender))
401        .map_err(ApiError::RequestSend)?;
402    api_evt.write(1).map_err(ApiError::EventFdWrite)?;
403
404    response_receiver.recv().map_err(ApiError::ResponseRecv)??;
405   ```
4061. The Cloud Hypervisor control loop receives the command, as it listens on the
407   internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel:
408   ```Rust
409   // Read from the API receiver channel
410   let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?;
411   ```
4121. The Cloud Hypervisor control loop matches the received internal API against
413   the `VmCreate` payload, and extracts both the `VmConfig` structure and the
414   [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the
415   command payload. It stores the `VmConfig` structure and replies back to the
416   sender ((The HTTP thread):
417   ```Rust
418   match api_request {
419	   ApiRequest::VmCreate(config, sender) => {
420		   // We only store the passed VM config.
421		   // The VM will be created when being asked to boot it.
422		   let response = if self.vm_config.is_none() {
423			   self.vm_config = Some(config);
424			   Ok(ApiResponsePayload::Empty)
425		   } else {
426			   Err(ApiError::VmAlreadyCreated)
427		   };
428
429	       sender.send(response).map_err(Error::ApiResponseSend)?;
430	   }
431   ```
4321. The Cloud Hypervisor HTTP thread receives the internal API command response
433   as the return value from its `VmCreate` HTTP handler. Depending on the
434   control loop internal API response, it generates the appropriate HTTP
435   response:
436   ```Rust
437   // Call vm_create()
438   match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config)))
439	   .map_err(HttpError::VmCreate)
440   {
441	   Ok(_) => Response::new(Version::Http11, StatusCode::NoContent),
442	   Err(e) => error_response(e, StatusCode::InternalServerError),
443   }
444   ```
4451. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the
446   user. This is abstracted by the
447   [micro_http](https://github.com/firecracker-microvm/micro-http)
448   crate.
449