xref: /cloud-hypervisor/docs/api.md (revision 2518b9e3cdd8cc4cfcb9503ed849a34aa7ade9ef)
1- [Cloud Hypervisor API](#cloud-hypervisor-api)
2  * [External API](#external-api)
3    + [REST API](#rest-api)
4      - [Location and availability](#location-and-availability)
5      - [Endpoints](#endpoints)
6		* [Virtual Machine Manager (VMM) Actions](#virtual-machine-manager-vmm-actions)
7		* [Virtual Machine (VM) Actions](#virtual-machine-vm-actions)
8      - [REST API Examples](#rest-api-examples)
9        * [Create a Virtual Machine](#create-a-virtual-machine)
10	    * [Boot a Virtual Machine](#boot-a-virtual-machine)
11        * [Dump a Virtual Machine Information](#dump-a-virtual-machine-information)
12        * [Reboot a Virtual Machine](#reboot-a-virtual-machine)
13        * [Shut a Virtual Machine Down](#shut-a-virtual-machine-down)
14    + [Command Line Interface](#command-line-interface)
15    + [REST API and CLI Architecural Relationship](#rest-api-and-cli-architectural-relationship)
16  * [Internal API](#internal-api)
17    + [Goals and Design](#goals-and-design)
18  * [End to End Example](#end-to-end-example)
19
20# Cloud Hypervisor API
21
22The Cloud Hypervisor API is made of 2 distinct interfaces:
23
241. **The external API**. This is the user facing API. Users and operators can
25   control and manage Cloud Hypervisor through either a REST API or a Command
26   Line Interface (CLI).
271. **The internal API**, based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/)
28   module. This API is used internally by the Cloud Hypervisor threads to
29   communicate between each others.
30
31The goal of this document is to describe the Cloud Hypervisor API as a whole,
32and to outline how the internal and external APIs are architecturally related.
33
34## External API
35
36### REST API
37
38The Cloud Hypervisor [REST](https://en.wikipedia.org/wiki/Representational_state_transfer)
39API triggers VM and VMM specific actions, and as such it is designed as a
40collection of RPC-style, static methods.
41
42The API is [OpenAPI 3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md)
43compliant. Please consult the [Cloud Hypervisor API](https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml)
44document for more details about the API payloads and responses.
45
46### Location and availability
47
48The REST API is available as soon as the Cloud Hypervisor binary is started,
49through a local UNIX socket.
50By default, it is located at `/run/user/{user ID}/cloud-hypervisor.{Cloud Hypervisor PID}`.
51For example, if you launched Cloud Hypervisor as user ID 1000 and its PID is
52123456, the Cloud Hypervisor REST API will be available at `/run/user/1000/cloud-hypervisor.123456`.
53
54The REST API default URL can be overridden through the Cloud Hypervisor
55option `--api-socket`:
56
57```
58$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
59Cloud Hypervisor Guest
60    API server: /tmp/cloud-hypervisor.sock
61    vCPUs: 1
62    Memory: 512 MB
63    Kernel: None
64    Kernel cmdline:
65    Disk(s): None
66```
67
68### Endpoints
69
70The Cloud Hypervisor API exposes the following actions through its endpoints:
71
72#### Virtual Machine Manager (VMM) Actions
73
74Action                              | Endpoint        | Request Body | Response Body              | Prerequisites
75------------------------------------|-----------------|--------------|----------------------------|---------------------------
76Check for the REST API availability | `/vmm.ping`     | N/A          | `/schemas/VmmPingResponse` | N/A
77Shut the VMM down                   | `/vmm.shutdown` | N/A          | N/A                        | The VMM is running
78
79#### Virtual Machine (VM) Actions
80
81Action                             | Endpoint            | Request Body              | Response Body            | Prerequisites
82-----------------------------------|---------------------|---------------------------|--------------------------|---------------------------
83Create the VM                      | `/vm.create`        | `/schemas/VmConfig`       | N/A                      | The VM is not created yet
84Delete the VM                      | `/vm.delete`        | N/A                       | N/A                      | N/A
85Boot the VM                        | `/vm.boot`          | N/A                       | N/A                      | The VM is created but not booted
86Shut the VM down                   | `/vm.shutdown`      | N/A                       | N/A                      | The VM is booted
87Reboot the VM                      | `/vm.reboot`        | N/A                       | N/A                      | The VM is booted
88Pause the VM                       | `/vm.pause`         | N/A                       | N/A                      | The VM is booted
89Resume the VM                      | `/vm.resume`        | N/A                       | N/A                      | The VM is paused
90Add/remove CPUs to/from the VM     | `/vm.resize`        | `/schemas/VmResize`       | N/A                      | The VM is booted
91Remove memory from the VM          | `/vm.resize`        | `/schemas/VmResize`       | N/A                      | The VM is booted
92Dump the VM information            | `/vm.info`          | N/A                       | `/schemas/VmInfo`        | The VM is created
93Add VFIO PCI device to the VM      | `/vm.add-device`    | `/schemas/VmAddDevice`    | `/schemas/PciDeviceInfo` | The VM is booted
94Remove VFIO PCI device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A                      | The VM is booted
95Add disk device to the VM          | `/vm.add-disk`      | `/schemas/DiskConfig`     | `/schemas/PciDeviceInfo` | The VM is booted
96Add fs device to the VM            | `/vm.add-fs`        | `/schemas/FsConfig`       | `/schemas/PciDeviceInfo` | The VM is booted
97Add pmem device to the VM          | `/vm.add-pmem`      | `/schemas/PmemConfig`     | `/schemas/PciDeviceInfo` | The VM is booted
98Add network device to the VM       | `/vm.add-net`       | `/schemas/NetConfig`      | `/schemas/PciDeviceInfo` | The VM is booted
99Add vsock device to the VM         | `/vm.add-vsock`     | `/schemas/VsockConfig`    | `/schemas/PciDeviceInfo` | The VM is booted
100Dump the VM counters               | `/vm.counters`      | N/A                       | `/schemas/VmCounters`    | The VM is booted
101
102### REST API Examples
103
104For the following set of examples, we assume Cloud Hypervisor is started with
105the REST API available at `/tmp/cloud-hypervisor.sock`:
106
107```
108$ ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
109Cloud Hypervisor Guest
110    API server: /tmp/cloud-hypervisor.sock
111    vCPUs: 1
112    Memory: 512 MB
113    Kernel: None
114    Kernel cmdline:
115    Disk(s): None
116```
117
118#### Create a Virtual Machine
119
120We want to create a virtual machine with the following characteristics:
121
122* 4 vCPUs
123* 1 GB of RAM
124* 1 virtio based networking interface
125* Direct kernel boot from a custom 5.5.0 Linux kernel located at
126  `/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu`
127* Using a Clear Linux image as its root filesystem, located at
128  `/opt/clh/images/clear-30080-kvm.img`
129
130```shell
131#!/bin/bash
132
133curl --unix-socket /tmp/cloud-hypervisor.sock -i \
134     -X PUT 'http://localhost/api/v1/vm.create'  \
135     -H 'Accept: application/json'               \
136     -H 'Content-Type: application/json'         \
137     -d '{
138         "cpus":{"boot_vcpus": 4, "max_vcpus": 4},
139         "kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"},
140         "cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"},
141         "disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}],
142         "rng":{"src":"/dev/urandom"},
143         "net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}]
144         }'
145```
146
147#### Boot a Virtual Machine
148
149Once the VM is created, we can boot it:
150
151```shell
152#!/bin/bash
153
154curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.boot'
155```
156
157#### Dump a Virtual Machine Information
158
159We can fetch information about any VM, as soon as it's created:
160
161```shell
162#!/bin/bash
163
164curl --unix-socket /tmp/cloud-hypervisor.sock -i \
165     -X GET 'http://localhost/api/v1/vm.info' \
166     -H 'Accept: application/json'
167```
168
169#### Reboot a Virtual Machine
170
171We can reboot a VM that's already booted:
172
173```shell
174#!/bin/bash
175
176curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.reboot'
177```
178
179#### Shut a Virtual Machine Down
180
181Once booted, we can shut a VM down from the REST API:
182
183```shell
184#!/bin/bash
185
186curl --unix-socket /tmp/cloud-hypervisor.sock -i -X PUT 'http://localhost/api/v1/vm.shutdown'
187```
188
189### Command Line Interface
190
191The Cloud Hypervisor Command Line Interface (CLI) can only be used for launching
192the Cloud Hypervisor binary, i.e. it can not be used for controlling the VMM or
193the launched VM once they're up and running.
194
195If you want to inspect the VMM, or control the VM after launching Cloud
196Hypervisor from the CLI, you must use the [REST API](#rest-api).
197
198From the CLI, one can either:
199
2001. Create and boot a complete virtual machine by using the CLI options to build
201   the VM config. Run `cloud-hypervisor --help` for a complete list of CLI
202   options. As soon as the `cloud-hypervisor` binary is launched, the
203   [REST API](#rest-api) is available for controlling and managing the VM.
2041. Start the [REST API](#rest-api) server only, by not passing any VM
205   configuration options. The VM can then be asynchronously created and booted
206   by sending HTTP commands to the [REST API](#rest-api). Check the
207   [REST API examples](#rest-api-examples) section for more details.
208
209### REST API and CLI Architectural Relationship
210
211The REST API and the CLI both rely on a common, [internal API](#internal-api).
212
213The CLI options are parsed by the
214[clap crate](https://docs.rs/clap/2.33.0/clap/) and then translated into
215[internal API](#internal-api) commands.
216
217The REST API is processed by an HTTP thread using the
218[Firecracker's `micro_http`](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http)
219crate. As with the CLI, the HTTP requests eventually get translated into
220[internal API](#internal-api) commands.
221
222As a summary, the REST API and the CLI are essentially frontends for the
223[internal API](#internal-api):
224
225```
226                                  +------------------+
227                        REST API  |                  |
228                       +--------->+    micro_http    +--------+
229                       |          |                  |        |
230                       |          +------------------+        |
231                       |                                      |      +------------------------+
232                       |                                      |      |                        |
233+------------+         |                                      |      |                        |
234|            |         |                                      |      | +--------------+       |
235|    User    +---------+                                      +------> | Internal API |       |
236|            |         |                                      |      | +--------------+       |
237+------------+         |                                      |      |                        |
238                       |                                      |      |                        |
239                       |                                      |      +------------------------+
240                       |            +----------+              |                 VMM
241                       |     CLI    |          |              |
242                       +----------->+   clap   +--------------+
243                                    |          |
244                                    +----------+
245
246
247```
248
249## Internal API
250
251The Cloud Hypervisor internal API, as its name suggests, is used internally
252by the different Cloud Hypervisor threads (VMM, HTTP, control loop, etc) to
253send commands and responses to each others.
254
255It is based on [rust's Multi-Producer, Single-Consumer (MPSC)](https://doc.rust-lang.org/std/sync/mpsc/),
256and the single consumer (a.k.a. the API receiver) is the Cloud Hypervisor
257control loop.
258
259API producers are the HTTP thread handling the [REST API](#rest-api) and the
260main thread that initially parses the [CLI](#command-line-interface).
261
262### Goals and Design
263
264The internal API is designed for controlling, managing and inspecting a Cloud
265Hypervisor VMM and its guest. It is a backend for handling external, user
266visible requests through either the [REST API](#rest-api) or the
267[CLI](#command-line-interface) interfaces.
268
269The API follows a command-response scheme that closely maps the [REST API](#rest-api).
270Any command must be replied to with a response.
271
272Commands are [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) based messages and
273are received and processed by the VMM control loop.
274
275In order for the VMM control loop to respond to any internal API command, it
276must be able to send a response back to the MPSC sender. For that purpose, all
277internal API command payload carry the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html)
278end of an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel.
279
280The sender of any internal API command is therefore responsible for:
281
2821. Creating an [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) response
283   channel.
2841. Passing the [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html)
285   end of the response channel as part of the internal API command payload.
2861. Waiting for the internal API command's response on the [Receiver](https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html)
287   end of the response channel.
288
289## End to End Example
290
291In order to further understand how the external and internal Cloud Hypervisor
292APIs work together, let's look at a complete VM creation flow, from the
293[REST API](#rest-api) call, to the reply the external user will receive:
294
2951. A user or operator sends an HTTP request to the Cloud Hypervisor
296   [REST API](#rest-api) in order to creates a virtual machine:
297   ```
298   shell
299   #!/bin/bash
300
301	curl --unix-socket /tmp/cloud-hypervisor.sock -i \
302		-X PUT 'http://localhost/api/v1/vm.create'  \
303		-H 'Accept: application/json'               \
304		-H 'Content-Type: application/json'         \
305		-d '{
306			"cpus":{"boot_vcpus": 4, "max_vcpus": 4},
307			"kernel":{"path":"/opt/clh/kernel/vmlinux-virtio-fs-virtio-iommu"},
308			"cmdline":{"args":"console=hvc0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3"},
309			"disks":[{"path":"/opt/clh/images/clear-30080-kvm.img"}],
310			"rng":{"src":"/dev/urandom"},
311			"net":[{"ip":"192.168.10.10", "mask":"255.255.255.0", "mac":"12:34:56:78:90:01"}]
312			}'
313   ```
3141. The Cloud Hypervisor HTTP thread processes the request and de-serializes the
315   HTTP request JSON body into an internal `VmConfig` structure.
3161. The Cloud Hypervisor HTTP thread creates an
317   [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel for the internal API
318   server to send its response back.
3191. The Cloud Hypervisor HTTP thread prepares an internal API command for creating a
320   virtual machine. The command's payload is made of the de-serialized
321   `VmConfig` structure and the response channel:
322   ```Rust
323   VmCreate(Arc<Mutex<VmConfig>>, Sender<ApiResponse>)
324   ```
3251. The Cloud Hypervisor HTTP thread sends the internal API command, and waits
326   for the response:
327   ```Rust
328   // Send the VM creation request.
329    api_sender
330        .send(ApiRequest::VmCreate(config, response_sender))
331        .map_err(ApiError::RequestSend)?;
332    api_evt.write(1).map_err(ApiError::EventFdWrite)?;
333
334    response_receiver.recv().map_err(ApiError::ResponseRecv)??;
335   ```
3361. The Cloud Hypervisor control loop receives the command, as it listens on the
337   internal API [MPSC](https://doc.rust-lang.org/std/sync/mpsc/) channel:
338   ```Rust
339   // Read from the API receiver channel
340   let api_request = api_receiver.recv().map_err(Error::ApiRequestRecv)?;
341   ```
3421. The Cloud Hypervisor control loop matches the received internal API against
343   the `VmCreate` payload, and extracts both the `VmConfig` structure and the
344   [Sender](https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html) from the
345   command payload. It stores the `VmConfig` structure and replies back to the
346   sender ((The HTTP thread):
347   ```Rust
348   match api_request {
349	   ApiRequest::VmCreate(config, sender) => {
350		   // We only store the passed VM config.
351		   // The VM will be created when being asked to boot it.
352		   let response = if self.vm_config.is_none() {
353			   self.vm_config = Some(config);
354			   Ok(ApiResponsePayload::Empty)
355		   } else {
356			   Err(ApiError::VmAlreadyCreated)
357		   };
358
359	       sender.send(response).map_err(Error::ApiResponseSend)?;
360	   }
361   ```
3621. The Cloud Hypervisor HTTP thread receives the internal API command response
363   as the return value from its `VmCreate` HTTP handler. Depending on the
364   control loop internal API response, it generates the appropriate HTTP
365   response:
366   ```Rust
367   // Call vm_create()
368   match vm_create(api_notifier, api_sender, Arc::new(Mutex::new(vm_config)))
369	   .map_err(HttpError::VmCreate)
370   {
371	   Ok(_) => Response::new(Version::Http11, StatusCode::NoContent),
372	   Err(e) => error_response(e, StatusCode::InternalServerError),
373   }
374   ```
3751. The Cloud Hypervisor HTTP thread sends the formed HTTP response back to the
376   user. This is abstracted by the
377   [micro_http](https://github.com/firecracker-microvm/firecracker/tree/master/src/micro_http)
378   crate.
379