xref: /cloud-hypervisor/release-notes.md (revision b440cb7d2330770cd415b63544a371d4caa2db3a)
1- [v25.0](#v250)
2    - [`ch-remote` Improvements](#ch-remote-improvements)
3    - [VM "Coredump" Support](#vm-coredump-support)
4    - [Notable Bug Fixes](#notable-bug-fixes)
5    - [Removals](#removals)
6    - [Contributors](#contributors)
7- [v24.0](#v240)
8    - [Bypass Mode for `virtio-iommu`](#bypass-mode-for-virtio-iommu)
9    - [Ensure Identifiers Uniqueness](#ensure-identifiers-uniqueness)
10    - [Sparse Mmap support](#sparse-mmap-support)
11    - [Expose Platform Serial Number](#expose-platform-serial-number)
12    - [Notable Bug Fixes](#notable-bug-fixes-1)
13    - [Notable Improvements](#notable-improvements)
14    - [Deprecations](#deprecations)
15    - [New on the Website](#new-on-the-website)
16    - [Contributors](#contributors-1)
17- [v23.1](#v231)
18- [v23.0](#v230)
19    - [vDPA Support](#vdpa-support)
20    - [Updated OS Support list](#updated-os-support-list)
21    - [`AArch64` Memory Map Improvements](#aarch64-memory-map-improvements)
22    - [`AMX` Support](#amx-support)
23    - [Notable Bug Fixes](#notable-bug-fixes-2)
24    - [Deprecations](#deprecations-1)
25    - [Contributors](#contributors-2)
26- [v22.1](#v221)
27- [v22.0](#v220)
28    - [GDB Debug Stub Support](#gdb-debug-stub-support)
29    - [`virtio-iommu` Backed Segments](#virtio-iommu-backed-segments)
30    - [Before Boot Configuration Changes](#before-boot-configuration-changes)
31    - [`virtio-balloon` Free Page Reporting](#virtio-balloon-free-page-reporting)
32    - [Support for Direct Kernel Booting with TDX](#support-for-direct-kernel-booting-with-tdx)
33    - [PMU Support for AArch64](#pmu-support-for-aarch64)
34    - [Documentation Under CC-BY-4.0 License](#documentation-under-cc-by-40-license)
35    - [Deprecation of "Classic" `virtiofsd`](#deprecation-of-classic-virtiofsd)
36    - [Notable Bug Fixes](#notable-bug-fixes-3)
37    - [Contributors](#contributors-3)
38- [v21.0](#v210)
39    - [Efficient Local Live Migration (for Live Upgrade)](#efficient-local-live-migration-for-live-upgrade)
40    - [Recommended Kernel is Now 5.15](#recommended-kernel-is-now-515)
41    - [Notable Bug fixes](#notable-bug-fixes-4)
42    - [Contributors](#contributors-4)
43- [v20.2](#v202)
44- [v20.1](#v201)
45- [v20.0](#v200)
46    - [Multiple PCI segments support](#multiple-pci-segments-support)
47    - [CPU pinning](#cpu-pinning)
48    - [Improved VFIO support](#improved-vfio-support)
49    - [Safer code](#safer-code)
50    - [Extended documentation](#extended-documentation)
51    - [Notable bug fixes](#notable-bug-fixes-5)
52    - [Contributors](#contributors-5)
53- [v19.0](#v190)
54    - [Improved PTY handling for serial and `virtio-console`](#improved-pty-handling-for-serial-and-virtio-console)
55    - [PCI boot time optimisations](#pci-boot-time-optimisations)
56    - [Improved TDX support](#improved-tdx-support)
57    - [Live migration enhancements](#live-migration-enhancements)
58    - [`virtio-mem` support with `vfio-user`](#virtio-mem-support-with-vfio-user)
59    - [AArch64 for `virtio-iommu`](#aarch64-for-virtio-iommu)
60    - [Notable bug fixes](#notable-bug-fixes-6)
61    - [Contributors](#contributors-6)
62- [v18.0](#v180)
63    - [Experimental User Device (`vfio-user`) support](#experimental-user-device-vfio-user-support)
64    - [Migration support for `vhost-user` devices](#migration-support-for-vhost-user-devices)
65    - [VHDX disk image support](#vhdx-disk-image-support)
66    - [Device pass through on MSHV hypervisor](#device-pass-through-on-mshv-hypervisor)
67    - [AArch64 for support `virtio-mem`](#aarch64-for-support-virtio-mem)
68    - [Live migration on MSHV hypervisor](#live-migration-on-mshv-hypervisor)
69    - [AArch64 CPU topology support](#aarch64-cpu-topology-support)
70    - [Power button support on AArch64](#power-button-support-on-aarch64)
71    - [Notable bug fixes](#notable-bug-fixes-7)
72    - [Contributors](#contributors-7)
73- [v17.0](#v170)
74    - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi)
75    - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend)
76    - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices)
77    - [Improved SGX support](#improved-sgx-support)
78    - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices)
79    - [Notable bug fixes](#notable-bug-fixes-8)
80    - [Contributors](#contributors-8)
81- [v16.0](#v160)
82    - [Improved live migration support](#improved-live-migration-support)
83    - [Improved `vhost-user` support](#improved-vhost-user-support)
84    - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support)
85    - [Notable bug fixes](#notable-bug-fixes-9)
86    - [Removed functionality](#removed-functionality)
87    - [Contributors](#contributors-9)
88- [v15.0](#v150)
89    - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees)
90    - [Network device rate limiting](#network-device-rate-limiting)
91    - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload)
92    - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter)
93    - [Bug fixes](#bug-fixes)
94    - [Deprecations](#deprecations-2)
95    - [Contributors](#contributors-10)
96- [v0.14.1](#v0141)
97- [v0.14.0](#v0140)
98    - [Structured event monitoring](#structured-event-monitoring)
99    - [MSHV improvements](#mshv-improvements)
100    - [Improved aarch64 platform](#improved-aarch64-platform)
101    - [Updated hotplug documentation](#updated-hotplug-documentation)
102    - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console)
103    - [Block device rate limiting](#block-device-rate-limiting)
104    - [Deprecations](#deprecations-3)
105    - [Contributors](#contributors-11)
106- [v0.13.0](#v0130)
107    - [Wider VFIO device support](#wider-vfio-device-support)
108    - [Improved huge page support](#improved-huge-page-support)
109    - [MACvTAP support](#macvtap-support)
110    - [VHD disk image support](#vhd-disk-image-support)
111    - [Improved Virtio device threading](#improved-virtio-device-threading)
112    - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button)
113    - [Contributors](#contributors-12)
114- [v0.12.0](#v0120)
115    - [ARM64 enhancements](#arm64-enhancements)
116    - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning)
117    - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend)
118    - [Enhanced "info" API](#enhanced-info-api)
119    - [Contributors](#contributors-13)
120- [v0.11.0](#v0110)
121    - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block)
122    - [Windows Guest Support](#windows-guest-support)
123    - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation)
124    - [`virtio-mmio` Removal](#virtio-mmio-removal)
125    - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64)
126    - [Improved Linux Boot Time](#improved-linux-boot-time)
127    - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling)
128    - [Default Log Level Changed](#default-log-level-changed)
129    - [New `--balloon` Parameter Added](#new---balloon-parameter-added)
130    - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support)
131    - [Notable Bug Fixes](#notable-bug-fixes-10)
132    - [Contributors](#contributors-14)
133- [v0.10.0](#v0100)
134    - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors)
135    - [Memory Zones](#memory-zones)
136    - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements)
137    - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control)
138    - [Notable Bug Fixes](#notable-bug-fixes-11)
139    - [Contributors](#contributors-15)
140- [v0.9.0](#v090)
141    - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support)
142    - [Block and Network Device Statistics](#block-and-network-device-statistics)
143    - [HTTP API Responses](#http-api-responses)
144    - [CPU Topology](#cpu-topology)
145    - [Release Build Optimization](#release-build-optimization)
146    - [Hypervisor Abstraction](#hypervisor-abstraction)
147    - [Snapshot/Restore Improvements](#snapshotrestore-improvements)
148    - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support)
149    - [Enhancements to ARM64 Support](#enhancements-to-arm64-support)
150    - [Intel SGX Support](#intel-sgx-support)
151    - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1)
152    - [Notable Bug Fixes](#notable-bug-fixes-12)
153    - [Contributors](#contributors-16)
154- [v0.8.0](#v080)
155    - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support)
156    - [Experimental ARM64 Support](#experimental-arm64-support)
157    - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests)
158    - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices)
159    - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements)
160    - [Notable Bug Fixes](#notable-bug-fixes-13)
161    - [Command Line and API Changes](#command-line-and-api-changes)
162    - [Contributors](#contributors-17)
163- [v0.7.0](#v070)
164    - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug)
165    - [Alternative `libc` Support](#alternative-libc-support)
166    - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends)
167    - [Initial RamFS Support](#initial-ramfs-support)
168    - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem)
169    - [`Seccomp` Sandboxing](#seccomp-sandboxing)
170    - [Updated Distribution Support](#updated-distribution-support)
171    - [Command Line and API Changes](#command-line-and-api-changes-1)
172    - [Contributors](#contributors-18)
173- [v0.6.0](#v060)
174    - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug)
175    - [Shared Filesystem Improvements](#shared-filesystem-improvements)
176    - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading)
177    - [Command Line Interface](#command-line-interface)
178    - [PVH Boot](#pvh-boot)
179    - [Contributors](#contributors-19)
180- [v0.5.1](#v051)
181- [v0.5.0](#v050)
182    - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing)
183    - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization)
184    - [New Interrupt Management Framework](#new-interrupt-management-framework)
185    - [Development Tools](#development-tools)
186    - [Kata Containers Integration](#kata-containers-integration)
187    - [Contributors](#contributors-20)
188- [v0.4.0](#v040)
189    - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition)
190    - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation)
191    - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends)
192    - [Guest pause and resume](#guest-pause-and-resume)
193    - [Userspace IOAPIC by default](#userspace-ioapic-by-default)
194    - [PCI BAR reprogramming](#pci-bar-reprogramming)
195    - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization)
196    - [Contributors](#contributors-21)
197- [v0.3.0](#v030)
198    - [Block device offloading](#block-device-offloading)
199    - [Network device backend](#network-device-backend)
200    - [Virtual sockets](#virtual-sockets)
201    - [HTTP based API](#http-based-api)
202    - [Memory mapped virtio transport](#memory-mapped-virtio-transport)
203    - [Paravirtualized IOMMU](#paravirtualized-iommu)
204    - [Ubuntu 19.10](#ubuntu-1910)
205    - [Large memory guests](#large-memory-guests)
206- [v0.2.0](#v020)
207    - [Network device offloading](#network-device-offloading)
208    - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi)
209    - [Debug I/O port](#debug-io-port)
210    - [Improved direct device assignment](#improved-direct-device-assignment)
211    - [Improved shared filesystem](#improved-shared-filesystem)
212    - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci)
213- [v0.1.0](#v010)
214    - [Shared filesystem](#shared-filesystem)
215    - [Initial direct device assignment support](#initial-direct-device-assignment-support)
216    - [Userspace IOAPIC](#userspace-ioapic)
217    - [Virtual persistent memory](#virtual-persistent-memory)
218    - [Linux kernel bzImage](#linux-kernel-bzimage)
219    - [Console over virtio](#console-over-virtio)
220    - [Unit testing](#unit-testing)
221    - [Integration tests parallelization](#integration-tests-parallelization)
222
223# v25.0
224
225This release has been tracked through the [v25.0
226project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/29).
227
228### `ch-remote` Improvements
229
230The `ch-remote` command has gained support for creating the VM from a JSON
231config and support for booting and deleting the VM from the VMM.
232
233### VM "Coredump" Support
234
235Under the `guest_debug` feature flag it is now possible to extract the memory
236of the guest for use in debugging with e.g. the `crash` utility. (#4012)
237
238### Notable Bug Fixes
239
240* Always restore console mode on exit (#4249, #4248)
241* Restore vCPUs in numerical order which fixes aarch64 snapshot/restore (#4244)
242* Don't try and configure `IFF_RUNNING` on TAP devices (#4279)
243* Propagate configured queue size through to vhost-user backend (#4286)
244* Always Program vCPU CPUID before running the vCPU to fix running on Linux
245  5.16 (#4156)
246* Enable ACPI MADT "Online Capable" flag for hotpluggable vCPUs to fix newer
247  Linux guest
248
249### Removals
250
251The following functionality has been removed:
252
253* The `mergeable` option from the `virtio-pmem` support has been removed
254  (#3968)
255* The `dax` option from the `virtio-fs` support has been removed (#3889)
256
257### Contributors
258
259Many thanks to everyone who has contributed to our release:
260
261* Dylan Bargatze <dbargatz@users.noreply.github.com>
262* Jinank Jain <jinankjain@microsoft.com>
263* Michael Zhao <michael.zhao@arm.com>
264* Rob Bradford <robert.bradford@intel.com>
265* Sebastien Boeuf <sebastien.boeuf@intel.com>
266* Wei Liu <liuwe@microsoft.com>
267* Yi Wang <wang.yi59@zte.com.cn>
268
269# v24.0
270
271This release has been tracked through the [v24.0
272project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/28).
273
274### Bypass Mode for `virtio-iommu`
275
276`virtio-iommu` specification describes how a device can be attached by default
277to a bypass domain. This feature is particularly helpful for booting a VM with
278guest software which doesn't support `virtio-iommu` but still need to access
279the device. Now that Cloud Hypervisor supports this feature, it can boot a VM
280with Rust Hypervisor Firmware or OVMF even if the `virtio-block` device exposing
281the disk image is placed behind a virtual IOMMU.
282
283### Ensure Identifiers Uniqueness
284
285Multiple checks have been added to the code to prevent devices with identical
286identifiers from being created, and therefore avoid unexpected behaviors at boot
287or whenever a device was hot plugged into the VM.
288
289### Sparse Mmap support
290
291Sparse mmap support has been added to both VFIO and vfio-user devices. This
292allows the device regions that are not fully mappable to be partially mapped.
293And the more a device region can be mapped into the guest address space, the
294fewer VM exits will be generated when this device is accessed. This directly
295impacts the performance related to this device.
296
297### Expose Platform Serial Number
298
299A new `serial_number` option has been added to `--platform`, allowing a user to
300set a specific serial number for the platform. This number is exposed to the
301guest through the SMBIOS.
302
303### Notable Bug Fixes
304
305* Fix loading RAW firmware (#4072)
306* Reject compressed QCOW images (#4055)
307* Reject virtio-mem resize if device is not activated (#4003)
308* Fix potential mmap leaks from VFIO/vfio-user MMIO regions (#4069)
309* Fix algorithm finding HOB memory resources (#3983)
310
311### Notable Improvements
312
313* Refactor interrupt handling (#4083)
314* Load kernel asynchronously (#4022)
315* Only create ACPI memory manager DSDT when resizable (#4013)
316
317### Deprecations
318
319Deprecated features will be removed in a subsequent release and users should
320plan to use alternatives
321
322* The `mergeable` option from the `virtio-pmem` support has been deprecated
323  (#3968)
324* The `dax` option from the `virtio-fs` support has been deprecated (#3889)
325
326### New on the Website
327
328A new blog post [Achieving Bare Metal Performance Within a Virtual
329Machine](https://www.cloudhypervisor.org/blog/achieving-bare-metal-performance-within-a-virtual-machine)
330has been added to the Cloud Hypervisor website.
331
332### Contributors
333
334Many thanks to everyone who has contributed to our release:
335
336* Anatol Belski <anbelski@linux.microsoft.com>
337* Bo Chen <chen.bo@intel.com>
338* Fabiano Fidêncio <fabiano.fidencio@intel.com>
339* LiHui <andrewli@kubesphere.io>
340* Maksym Pavlenko <pavlenko.maksym@gmail.com>
341* Rob Bradford <robert.bradford@intel.com>
342* Sebastien Boeuf <sebastien.boeuf@intel.com>
343* Steven Dake <steven.dake@gmail.com>
344* Vincent Batts <vbatts@hashbangbash.com>
345* Wei Liu <liuwe@microsoft.com>
346
347# v23.1
348
349This is a bug fix release. The following issues have been addressed:
350
351* Add some missing seccomp rules
352* Remove `virtio-fs` filesystem entries from config on removal
353* Do not delete API socket on API server start (#4026)
354* Reject `virtio-mem` resize if the guest doesn't activate the device
355* Fix OpenAPI naming of I/O throttling knobs
356
357# v23.0
358
359This release has been tracked through the [v23.0
360project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/27).
361
362### vDPA Support
363
364A vDPA device has a datapath that complies with the virtio specification but
365with a vendor specific control path. The addition of `--vdpa` and the REST API
366equivalent allows the use of these devices with Cloud Hypervisor.
367
368### Updated OS Support list
369
370The list of officially supported and tested OS versions has been updated to
371include Ubuntu "jammy" 22.04 and EOLed versions removed.
372
373### `AArch64` Memory Map Improvements
374
375The memory map when running on `AArch64` has been improved for the handling of
376the UEFI region which means that the booted guest OS now has full access to its
377allocated RAM. (#3938)
378
379### `AMX` Support
380
381Under a compile time gate of `amx` it is possible compile in support for the
382`AMX` instruction set extension for guest use. This also requires runtime
383enabling with `--cpu features=amx`.
384
385### Notable Bug Fixes
386
387* Generate error when incorrect HTTP method used for some API endpoints (#3887)
388* CMOS based reset is now available to support rebooting on "jammy" (#3914)
389* ACPI tables are not produced for memory hotplug when running with
390  `virtio-mem` (#3883)
391* `virtio-iommu` backed PCI segments are now comprehensively placed behind the
392  vIOMMU (#3870)
393* Seccomp rules have been extended for `virtio-fs` to support direct access
394  (#3848)
395
396### Deprecations
397
398Deprecated features will be removed in a subsequent release and users should
399plan to use alternatives
400
401* The `mergeable` option from the `virtio-pmem` support has been deprecated
402  (#3968)
403* The `dax` option from the `virtio-fs` support has been deprecated (#3889)
404
405### Contributors
406
407Many thanks to everyone who has contributed to our release:
408
409* Bo Chen <chen.bo@intel.com>
410* Fabiano Fidêncio <fabiano.fidencio@intel.com>
411* Henry Wang <Henry.Wang@arm.com>
412* Jianyong Wu <jianyong.wu@arm.com>
413* LiHui <andrewli@kubesphere.io>
414* Michael Zhao <michael.zhao@arm.com>
415* Rob Bradford <robert.bradford@intel.com>
416* Sebastien Boeuf <sebastien.boeuf@intel.com>
417* Wei Liu <liuwe@microsoft.com>
418* William Douglas <william.douglas@intel.com>
419* Yi Wang <wang.yi59@zte.com.cn>
420
421# v22.1
422
423This is a bug fix release. The following issues have been addressed:
424
425* VFIO ioctl reordering to fix MSI on AMD platforms (#3827)
426* Fix `virtio-net` control queue (#3829)
427
428# v22.0
429
430This release has been tracked through the [v22.0
431project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/25).
432
433### GDB Debug Stub Support
434
435Cloud Hypervisor can now be used as debug target with GDB. This is controlled
436by the `gdb` compile time feature and details of how to use it can be found in
437the [gdb
438documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/gdb.md).
439
440### `virtio-iommu` Backed Segments
441
442In order to facilitate hotplug devices that require being behind an IOMMU (e.g.
443QAT) there is a new option `--platform iommu_segments=<list_of_segments>` that
444will place all the specified segments behind the IOMMU.
445
446### Before Boot Configuration Changes
447
448It is now possible to change the VM configuration (e.g. add or remove devices,
449resize) before the VM is booted.
450
451### `virtio-balloon` Free Page Reporting
452
453If `--balloon free_page_reporting=on` is used then the guest can report pages
454that is it not using to the VMM. The VMM will then notify the host OS that
455those pages are no longer in use and can be freed. This can result in improved
456memory density.
457
458### Support for Direct Kernel Booting with TDX
459
460Through the use of `TD-Shim` lightweight firmware it is now possible to
461directly boot into the kernel with TDX. The [TDX
462documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/intel_tdx.md#tdshim)
463has been updated for this usage.
464
465### PMU Support for AArch64
466
467A PMU is now available on AArch64 for guest performance profiling. This will be
468exposed automatically if available from the host.
469
470### Documentation Under CC-BY-4.0 License
471
472The documentation is now licensed under the "Creative Commons Attribution 4.0
473International" license which is aligned with the project charter under the
474Linux Foundation.
475
476### Deprecation of "Classic" `virtiofsd`
477
478The use of the Rust based [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd)
479is now recommended and we are no longer testing against the C based "classic"
480version.
481
482### Notable Bug Fixes
483
484* Can now be used on kernels without `AF_INET` support (#3785)
485* `virtio-balloon` size is now validated against guest RAM size (#3689)
486* Ensure that I/O related KVM VM Exits are correctly handled (#3677)
487* Multiple TAP file descriptors can be used for `virtio-net` device hotplug (#3607)
488* Minor API improvements and fixes (#3756, #3766, #3647, #3578)
489* Fix sporadic seccomp violation from glibc memory freeing (#3610, #3609)
490* Fix Windows 11 on AArch64 due to wider MSI-X register accesses (#3714, #3720)
491* Ensure `vhost-user` features are correct across migration (#3737)
492* Improved vCPU topology on AArch64 (#3735, #3733)
493
494### Contributors
495
496Many thanks to everyone who has contributed to our release:
497
498* Akira Moroo <retrage01@gmail.com>
499* Barret Rhoden <brho@google.com>
500* Bo Chen <chen.bo@intel.com>
501* Fabiano Fidêncio <fabiano.fidencio@intel.com>
502* Feng Ye <yefeng@smartx.com>
503* Henry Wang <Henry.Wang@arm.com>
504* Jianyong Wu <jianyong.wu@arm.com>
505* lizhaoxin1 <Lxiaoyouling@163.com>
506* Michael Zhao <michael.zhao@arm.com>
507* Rob Bradford <robert.bradford@intel.com>
508* Sebastien Boeuf <sebastien.boeuf@intel.com>
509* Wei Liu <liuwe@microsoft.com>
510
511# v21.0
512
513This release has been tracked through the [v21.0
514project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/24).
515
516### Efficient Local Live Migration (for Live Upgrade)
517
518In order to support fast live upgrade of the VMM an optimised path has been
519added in which the memory for the VM is not compared from source to
520destination. This is activated by passing `--local` to the `ch-remote
521send-migration` command. This means that the live upgrade can complete in the
522order of 50ms vs 3s. (#3566)
523
524### Recommended Kernel is Now 5.15
525
526Due to an issue in the `virtio-net` code in 5.14 the recommended Linux kernel
527is now 5.15. (#3530)
528
529### Notable Bug fixes
530
531* Multiple fixes were made to the OpenAPI YAML file to match the implementaion (#3555,#3562)
532* Avoid live migration deadlock when triggered during the kernel boot (#3585)
533* Support live migration within firmware (#3586)
534* Validate the `virtio-net` desciptor chain (#3548)
535* `direct=on` (`O_DIRECT`) can now be used with a guest that makes unaligned accesses (e.g. firmware) (#3587)
536
537### Contributors
538
539Many thanks to everyone who has contributed to our release:
540
541* Anatol Belski <anbelski@linux.microsoft.com>
542* Barret Rhoden <brho@google.com>
543* Bo Chen <chen.bo@intel.com>
544* Fabiano Fidêncio <fabiano.fidencio@intel.com>
545* Henry Wang <Henry.Wang@arm.com>
546* Liang Zhou <zhoul110@chinatelecom.cn>
547* Michael Zhao <michael.zhao@arm.com>
548* Muhammad Falak R Wani <falakreyaz@gmail.com>
549* Muminul Islam <muislam@microsoft.com>
550* Rob Bradford <robert.bradford@intel.com>
551* Sebastien Boeuf <sebastien.boeuf@intel.com>
552* Wei Liu <liuwe@microsoft.com>
553* Ziye Yang <ziye.yang@intel.com>
554
555# v20.2
556
557This is a bug fix release. The following issues have been addressed:
558
559* Don't error out when setting up the SIGWINCH handler (for console resize)
560  when this fails due to older kernel (#3456)
561* Seccomp rules were refined to remove syscalls that are now unused
562* Fix reboot on older host kernels when SIGWINCH handler was not initialised
563  (#3496)
564* Fix virtio-vsock blocking issue (#3497)
565
566# v20.1
567
568This is a bug fix release. The following issues have been addressed:
569
570* Networking performance regression with `virtio-net` (#3450)
571* Limit file descriptors sent in `vfio-user` support (#3401)
572* Fully advertise PCI MMIO config regions in ACPI tables (#3432)
573* Set the TSS and KVM identity maps so they don't overlap with firmware RAM
574* Correctly update the `DeviceTree` on restore
575
576# v20.0
577
578This release has been tracked through the [v20.0
579project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/23).
580
581### Multiple PCI segments support
582
583Cloud Hypervisor is no longer limited to 31 PCI devices. For both `x86_64` and
584`aarch64` architectures, it is now possible to create up to 16 PCI segments,
585increasing the total amount of supported PCI devices to 496.
586
587### CPU pinning
588
589For each vCPU, the user can define a limited set of host CPUs on which it is
590allowed to run. This can be useful when assigning a 1:1 mapping between host and
591guest resources, or when running a VM on a specific NUMA node.
592
593### Improved VFIO support
594
595Based on VFIO region capabilities, all regions can be memory mapped, limiting
596the amount of triggered VM exits, and therefore increasing the performance of
597the passthrough device.
598
599### Safer code
600
601Several sections containing unsafe Rust code have been replaced with safe
602alternatives, and multiple comments have been added to clarify why the remaining
603unsafe sections are safe to use.
604
605### Extended documentation
606
607The documentation related to VFIO has been updated while some new documents have
608been introduced to cover the usage of `--cpus` parameter as well as how to run
609Cloud Hypervisor on Intel TDX.
610
611### Notable bug fixes
612
613* Naturally align PCI BARs on relocation (#3244)
614* Fix panic in SIGWINCH listener thread when no seccomp filter set (#3338)
615* Use the tty raw mode implementation from libc (#3344)
616* Fix the emulation of register D for CMOS/RTC device (#3393)
617
618### Contributors
619
620Many thanks to everyone who has contributed to our release:
621
622* Alyssa Ross <hi@alyssa.is>
623* Bo Chen <chen.bo@intel.com>
624* Fabiano Fidêncio <fabiano.fidencio@intel.com>
625* Michael Zhao <michael.zhao@arm.com>
626* Muminul Islam <muislam@microsoft.com>
627* Rob Bradford <robert.bradford@intel.com>
628* Sebastien Boeuf <sebastien.boeuf@intel.com>
629* Wei Liu <liuwe@microsoft.com>
630* Willen Yang <willenyang@gmail.com>
631* William Douglas <william.douglas@intel.com>
632* Ziye Yang <ziye.yang@intel.com>
633
634# v19.0
635
636This release has been tracked through the [v19.0
637project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/22).
638
639### Improved PTY handling for serial and `virtio-console`
640
641The PTY support for serial has been enhanced with improved buffering when the
642the PTY is not yet connected to. Using `virtio-console` with PTY now results in
643the console being resized if the PTY window is also resized.
644
645### PCI boot time optimisations
646
647Multiple optimisations have been made to the PCI handling resulting in
648significant improvements in the boot time of the guest.
649
650### Improved TDX support
651
652When using the latest TDVF firmware the ACPI tables created by the VMM are now
653exposed via the firmware to the guest.
654
655### Live migration enhancements
656
657Live migration support has been enhanced to support migration with `virtio-mem`
658based memory hotplug and the `virtio-balloon` device now supports live
659migration.
660
661### `virtio-mem` support with `vfio-user`
662
663The use of `vfio-user` userspaces devices can now be used in conjunction with
664`virtio-mem` based memory hotplug and unplug.
665
666### AArch64 for `virtio-iommu`
667
668A paravirtualised IOMMU can now be used on the AArch64 platform.
669
670### Notable bug fixes
671
672* ACPI hotplugged memory is correctly restored after a live migration or
673  snapshot/restore (#3165)
674* Multiple devices from the same IOMMU group can be passed through via VFIO
675  (#3078 #3113)
676* Live migration with large blocks of memory was buggy due to an in issue in
677  the underlying crate (#3157)
678
679### Contributors
680
681Many thanks to everyone who has contributed to our release:
682
683* Alyssa Ross <hi@alyssa.is>
684* Bo Chen <chen.bo@intel.com>
685* Henry Wang <Henry.Wang@arm.com>
686* Hui Zhu <teawater@antfin.com>
687* Jianyong Wu <jianyong.wu@arm.com>
688* Li Yu <liyu.yukiteru@bytedance.com>
689* Michael Zhao <michael.zhao@arm.com>
690* Muminul Islam <muislam@microsoft.com>
691* Rob Bradford <robert.bradford@intel.com>
692* Sebastien Boeuf <sebastien.boeuf@intel.com>
693* Wei Liu <liuwe@microsoft.com>
694* William Douglas <william.douglas@intel.com>
695* Yu Li <liyu.yukiteru@bytedance.com>
696
697# v18.0
698
699This release has been tracked through the [v18.0
700project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/21).
701
702### Experimental User Device (`vfio-user`) support
703
704Experimental support for running PCI devices in userspace via `vfio-user`
705has been included. This allows the use of the SPDK NVMe `vfio-user` controller
706with Cloud Hypervisor. This is enabled by `--user-device` on the command line.
707
708### Migration support for `vhost-user` devices
709
710Devices exposed into the VM via `vhost-user` can now be migrated using the live
711migration support. This requires support from the backend however the commonly
712used DPDK `vhost-user` backend does support this.
713
714### VHDX disk image support
715
716Images using the VHDX disk image format can now be used with Cloud Hypervisor.
717
718### Device pass through on MSHV hypervisor
719
720When running on the MSHV hypervisor it is possible to pass through devices from
721the host through to the guest (e.g with `--device`)
722
723### AArch64 for support `virtio-mem`
724
725The reference Linux kernel we recommend for using with Cloud Hypervisor now supports `virtio-mem` on AArch64.
726
727### Live migration on MSHV hypervisor
728
729Live migration is now supported when running on the MSHV hypervisor including
730efficient tracking of dirty pages.
731
732### AArch64 CPU topology support
733
734The CPU topology (as configured through `--cpu topology=`) can now be
735configured on AArch64 platforms and is conveyed through either ACPI or device
736tree.
737
738### Power button support on AArch64
739
740Use of the ACPI power button (e.g `ch-remote --api-socket=<API socket> power-button`)
741is now supported when running on AArch64.
742
743### Notable bug fixes
744
745* Using two PTY outputs e.g. `--serial pty --console pty` now works correctly (#3012)
746* TTY input is now always sent to the correct destination (#3005)
747* The boot is no longer blocked when using a unattached PTY on the serial console (#3004)
748* Live migration is now supported on AArch64 (#3049)
749* Ensure signal handlers are run on the correct thread (#3069)
750
751### Contributors
752
753Many thanks to everyone who has contributed to our release:
754
755* Alyssa Ross <hi@alyssa.is>
756* Anatol Belski <anbelski@linux.microsoft.com>
757* Arafatms <arafatms@outlook.com>
758* Bo Chen <chen.bo@intel.com>
759* Fazla Mehrab <akm.fazla.mehrab@vt.edu>
760* Henry Wang <Henry.Wang@arm.com>
761* Jianyong Wu <jianyong.wu@arm.com>
762* Jiaqi Gao <jiaqi.gao@intel.com>
763* Markus Theil <markus.theil@tu-ilmenau.de>
764* Muminul Islam <muislam@microsoft.com>
765* Rob Bradford <robert.bradford@intel.com>
766* Sebastien Boeuf <sebastien.boeuf@intel.com>
767* Wei Liu <liuwe@microsoft.com>
768* Yu Li <liyu.yukiteru@bytedance.com>
769
770# v17.0
771
772This release has been tracked through the [v17.0
773project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/20).
774
775### ARM64 NUMA support using ACPI
776
777The support for ACPI on ARM64 has been enhanced to include support for
778specifying a NUMA configuration using the existing control options.
779
780### `Seccomp` support for MSHV backend
781
782The `seccomp` rules have now been extended to support running against the MSHV
783hypervisor backend.
784
785### Hotplug of `macvtap` devices
786
787Hotplug of `macvtap` devices is now supported with the file descriptor for the
788network device if opened by the user and passed to the VMM. The `ch-remote`
789tool supports this functionality when adding a network device.
790
791### Improved SGX support
792
793The SGX support has been updated to match the latest Linux kernel support and
794now supports SGX provisioning and associating EPC sections to NUMA nodes.
795
796### Inflight tracking for `vhost-user` devices
797
798Support for handling inflight tracking of I/O requests has been added to the
799`vhost-user` devices allowing recovery after device reconnection.
800
801### Notable bug fixes
802
803* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821).
804* The VMM side of `vhost-user` devices no longer advertise the
805  `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM
806(#2833).
807* On ARM64 VMs can be created with more than 16 vCPUs (#2763).
808
809### Contributors
810
811Many thanks to everyone who has contributed to our release:
812
813* Anatol Belski <anbelski@linux.microsoft.com>
814* Arafatms <arafatms@outlook.com>
815* Bo Chen <chen.bo@intel.com>
816* Fei Li <lifei.shirley@bytedance.com>
817* Henry Wang <Henry.Wang@arm.com>
818* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
819* Jianyong Wu <jianyong.wu@arm.com>
820* Li Hangjing <lihangjing@bytedance.com>
821* Michael Zhao <michael.zhao@arm.com>
822* Muminul Islam <muislam@microsoft.com>
823* Rob Bradford <robert.bradford@intel.com>
824* Sebastien Boeuf <sebastien.boeuf@intel.com>
825* Wei Liu <liuwe@microsoft.com>
826* Yukiteru <wfly1998@sina.com>
827
828# v16.0
829
830This release has been tracked through the [v16.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/19).
831
832### Improved live migration support
833
834The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details.
835
836### Improved `vhost-user` support
837
838When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch.
839
840### ARM64 ACPI and UEFI support
841
842Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot.
843
844### Notable bug fixes
845
846* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578).
847* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563).
848* The KVM clock is now correctly handled during live migration / snapshot & restore.
849
850### Removed functionality
851
852The following formerly deprecated features have been removed:
853
854* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage`
855  binaries has been deprecated. When using direct boot users should configure
856  their kernel with `CONFIG_PVH=y`.
857
858### Contributors
859
860Many thanks to everyone who has contributed to our release including some new faces.
861
862* Anatol Belski <anbelski@linux.microsoft.com>
863* Bo Chen <chen.bo@intel.com>
864* Dayu Liu <liu.dayu@zte.com.cn>
865* Henry Wang <Henry.Wang@arm.com>
866* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
867* Jianyong Wu <jianyong.wu@arm.com>
868* Michael Zhao <michael.zhao@arm.com>
869* Mikko Ylinen <mikko.ylinen@intel.com>
870* Muminul Islam <muislam@microsoft.com>
871* Ren Lei <ren.lei4@zte.com.cn>
872* Rob Bradford <robert.bradford@intel.com>
873* Sebastien Boeuf <sebastien.boeuf@intel.com>
874* Wei Liu <liuwe@microsoft.com>
875* Yi Wang <wang.yi59@zte.com.cn>
876
877# v15.0
878
879This release has been tracked through the [v15.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/18).
880
881Highlights for `cloud-hypervisor` version v15.0 include:
882
883### Version numbering and stability guarantees
884
885This release is the first in a new version numbering scheme to represent that
886we believe Cloud Hypervisor is maturing and entering a period of stability.
887With this new release we are beginning our new stability guarantees:
888
889* The API (including command line options) will not be removed or changed in a
890  breaking way without a minimum of 2 releases notice. Where possible warnings
891  will be given about the use of deprecated functionality and the deprecations
892  will be documented in the release notes.
893* Point releases will be made between individual releases where there are
894  substantial bug fixes or security issues that need to be fixed.
895
896Currently the following items are **not** guaranteed across updates:
897
898* Snapshot/restore is not supported across different versions
899* Live migration is not supported across different versions
900* The following features are considered experimental and may change
901  substantially between releases: TDX, SGX.
902
903### Network device rate limiting
904
905Building on our existing support for rate limiting block activity the network
906device also now supports rate limiting. Full details of the controls are in the
907[IO throttling documentation.](docs/io_throttling.md)
908
909### Support for runtime control of `virtio-net` guest offload
910
911The guest is now able to change the offload settings for the `virtio-net`
912device. As well as providing a useful control this mitigates an issue in the
913Linux kernel where the guest will attempt to reprogram the offload settings
914even if they are not advertised as configurable (#2528).
915
916### `--api-socket` supports file descriptor parameter
917
918The `--api-socket` can now take an `fd=` parameter to specify an existing file
919descriptor to use. This is particularly beneficial for frameworks that need to
920programmatically control Cloud Hypervisor.
921
922### Bug fixes
923
924* A workaround has been put in place to mitigate a Linux kernel issues that
925  results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277).
926* PCI BARs are now correctly aligned removing the need for the guest to
927  reprogram them (#1797,#1798)
928* Handle TAP interface not being writable within virtio-net (due to the buffer
929  exhaustion on the host) (#2517)
930* The recommended Linux kernel is now v5.12.0 as it contains a fix that
931  prevents snapshot & restore working (#2535)
932
933### Deprecations
934
935Deprecated features will be removed in a subsequent release and users should plan to use alternatives
936
937* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage`
938  binaries has been deprecated. When using direct boot users should configure
939  their kernel with `CONFIG_PVH=y`. Will be removed in v16.0.
940
941### Contributors
942
943Many thanks to everyone who has contributed to our release including some new faces.
944
945* Alyssa Ross <hi@alyssa.is>
946* Anatol Belski <anbelski@linux.microsoft.com>
947* Bo Chen <chen.bo@intel.com>
948* Gaelan Steele <gbs@canishe.com>
949* Jianyong Wu <jianyong.wu@arm.com>
950* Michael Zhao <michael.zhao@arm.com>
951* Muminul Islam <muislam@microsoft.com>
952* Rob Bradford <robert.bradford@intel.com>
953* Sebastien Boeuf <sebastien.boeuf@intel.com>
954* Wei Liu <liuwe@microsoft.com>
955* William Douglas <william.douglas@intel.com>
956
957# v0.14.1
958
959Bug fix release branched off the v0.14.0 release. The following bugs were fixed
960in this release:
961
962* CPU hotplug on Windows failed due to misreported CPU state information and
963  the lack of HyperV CPUID bit enabled (#2437, #2449, #2436)
964* A seccomp rule was missing that was triggered on CPU unplug (#2455)
965* A bounds check in VIRTIO queue validation was erroneously generating
966  DescriptorChainTooShort errors in certain circumstances (#2450, #2424)
967
968# v0.14.0
969
970This release has been tracked through the [0.14.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/17).
971
972Highlights for `cloud-hypervisor` version 0.14.0 include:
973
974### Structured event monitoring
975
976A new option was added to the VMM `--event-monitor` which reports structured
977events (JSON) over a file or file descriptor at key events in the lifecycle of
978the VM. The list of events is limited at the moment but will be further
979extended over subsequent releases. The events exposed form part of the Cloud
980Hypervisor API surface.
981
982### MSHV improvements
983
984Basic support has been added for running Windows guests atop the MSHV
985hypervisor as an alternative to KVM and further improvements have been made to
986the MSHV support.
987
988### Improved aarch64 platform
989
990The aarch64 platform has been enhanced with more devices exposed to the running
991VM including an enhanced serial UART.
992
993### Updated hotplug documentation
994
995The documentation for the hotplug support has been updated to reflect the use
996of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as
997well as documenting hotplug of paravirtualised and VFIO devices.
998
999### PTY control for serial and `virtio-console`
1000
1001The `--serial` and `--console` parameters can now direct the console to a PTY
1002allowing programmatic control of the console from another process through the
1003PTY subsystem.
1004
1005### Block device rate limiting
1006
1007The block device performance can now be constrained as part of the VM
1008configuration allowing rate limiting. Full details of the controls are in the
1009[IO throttling documentation.](docs/io_throttling.md)
1010
1011
1012### Deprecations
1013
1014Deprecated features will be removed in a subsequent release and users should plan to use alternatives
1015
1016* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage`
1017  binaries has been deprecated. When using direct boot users should configure
1018  their kernel with `CONFIG_PVH=y`.
1019
1020
1021### Contributors
1022
1023Many thanks to everyone who has contributed to our 0.14.0 release including
1024some new faces.
1025
1026Bo Chen <chen.bo@intel.com>
1027Henry Wang <Henry.Wang@arm.com>
1028Iggy Jackson <iggy@theiggy.com>
1029Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
1030Michael Zhao <michael.zhao@arm.com>
1031Muminul Islam <muislam@microsoft.com>
1032Penny Zheng <Penny.Zheng@arm.com>
1033Rob Bradford <robert.bradford@intel.com>
1034Sebastien Boeuf <sebastien.boeuf@intel.com>
1035Vineeth Pillai <viremana@linux.microsoft.com>
1036Wei Liu <liuwe@microsoft.com>
1037William Douglas <william.r.douglas@gmail.com>
1038Zide Chen <zide.chen@intel.com>
1039
1040# v0.13.0
1041
1042This release has been tracked through the [0.13.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/16).
1043
1044Highlights for `cloud-hypervisor` version 0.13.0 include:
1045
1046### Wider VFIO device support
1047
1048It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI
1049devices that do not support MSI or MSI-X and instead rely on INTx interrupts.
1050Most notably this widens the support to most NVIDIA cards with the proprietary
1051drivers.
1052
1053### Improved huge page support
1054
1055Through the addition of `hugepage_size` on `--memory` it is now possible to
1056specify the desired size of the huge pages used when allocating the guest
1057memory. The user is required to ensure they have sufficient pages of the
1058desired size in their pool.
1059
1060### MACvTAP support
1061
1062It is now possible to provide file descriptors using the `fd` parameter to
1063`--net` which point at TAP devices that have already been opened by the user.
1064This aids integration with `libvirt` but also permits the use of MACvTAP
1065support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md).
1066
1067### VHD disk image support
1068
1069It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw
1070disk image with Cloud Hypervisor.
1071
1072### Improved Virtio device threading
1073
1074Device threads are now derived from the main VMM thread which allows more
1075restrictive seccomp filters to be applied to them. The threads also have a
1076predictable name derived from the device id.
1077
1078### Clean shutdown support via synthetic power button
1079
1080It is now possible to request that the guest VM shut itself down by triggering
1081a synthetic ACPI power button press from the VMM. If the guest is listening for
1082such an event (e.g. using systemd) then it will process the event and cleanly
1083shut down. This functionality is exposed through the HTTP API and can be
1084triggered via `ch-remote --api-socket=<API socket> power-button`.
1085
1086### Contributors
1087
1088Many thanks to everyone who has contributed to our 0.13.0 release including
1089some new faces.
1090
1091* Bo Chen <chen.bo@intel.com>
1092* Mikko Ylinen <mikko.ylinen@intel.com>
1093* Muminul Islam <muislam@microsoft.com>
1094* Rob Bradford <robert.bradford@intel.com>
1095* Samuel Ortiz <sameo@linux.intel.com>
1096* Sebastien Boeuf <sebastien.boeuf@intel.com>
1097* Vineeth Pillai <viremana@linux.microsoft.com>
1098* Wei Liu <liuwe@microsoft.com>
1099* William Douglas <william.r.douglas@gmail.com>
1100* Xie Yongji <xieyongji@bytedance.com>
1101
1102# v0.12.0
1103
1104This release has been tracked through the [0.12.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/15).
1105
1106Highlights for `cloud-hypervisor` version 0.12.0 include:
1107
1108### ARM64 enhancements
1109
1110The use of `--watchdog` is now fully supported as is the ability to reboot the
1111VM from within the guest when running Cloud Hypervisor on an ARM64 system.
1112
1113### Removal of `vhost-user-net` and `vhost-user-block` self spawning
1114
1115In order to use `vhost-user-net` or `vhost-user-block` backends the user is now
1116responsible for starting the backend and providing the socket for the VMM to
1117use. This functionality was deprecated in the last release and how now been
1118removed.
1119
1120### Migration of `vhost-user-fs` backend
1121
1122The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is
1123instead hosted in [it's own
1124repository](https://gitlab.com/virtio-fs/virtiofsd-rs)
1125
1126### Enhanced "info" API
1127
1128The `vm.info` HTTP API endpoint has been extended to include the details of the
1129devices used by the VM including any VFIO devices used.
1130
1131### Contributors
1132
1133Many thanks to everyone who has contributed to our 0.12.0 release:
1134
1135* Anatol Belski <anbelski@linux.microsoft.com>
1136* Julio Montes <julio.montes@intel.com>
1137* Michael Zhao <michael.zhao@arm.com>
1138* Muminul Islam <muislam@microsoft.com>
1139* Rob Bradford <robert.bradford@intel.com>
1140* Samuel Ortiz <sameo@linux.intel.com>
1141* Wei Liu <liuwe@microsoft.com>
1142
1143# v0.11.0
1144
1145This release has been tracked through the [0.11.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/14).
1146
1147Highlights for `cloud-hypervisor` version 0.11.0 include:
1148
1149### `io_uring` support by default for `virtio-block`
1150
1151Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will
1152be used for a significantly higher performance block device.
1153
1154### Windows Guest Support
1155
1156This is the first release where we officially support Windows running as a
1157guest. Full details of how to setup the image and run Cloud Hypervisor with a
1158Windows guest can be found in the dedicated [Windows
1159documentation](docs/windows.md).
1160
1161### `vhost-user` "Self Spawning" Deprecation
1162
1163Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now
1164deprecated. Users of this functionality will receive a warning and should make
1165adjustments. The functionality will be removed in the next release.
1166
1167### `virtio-mmio` Removal
1168
1169Support for using the `virtio-mmio` transport, rather than using PCI, has been
1170removed. This has been to simplify the code and significantly
1171reduce the testing burden of the project.
1172
1173### Snapshot/Restore support for ARM64
1174
1175When running on the ARM64 architecture snapshot and restore has now been
1176implemented.
1177
1178### Improved Linux Boot Time
1179
1180The time to boot the Linux kernel has been significantly improved by the
1181identifying some areas of delays around PCI bus probing, IOAPIC programming and
1182MPTABLE issues. Full details can be seen in #1728.
1183
1184### `SIGTERM/SIGINT` Interrupt Signal Handling
1185
1186When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will
1187trigger the VMM process to cleanly deallocate resources before exiting. The
1188guest VM will not be cleanly shutdown but the VMM process will clean up its
1189resources.
1190
1191### Default Log Level Changed
1192
1193The default logging level was changed to include warnings which should make it
1194easier to see potential issues. New [logging
1195documentation](docs/logging) was also added.
1196
1197### New `--balloon` Parameter Added
1198
1199Control of the setup of `virtio-balloon` has been moved from `--memory` to its
1200own dedicated parameter. This makes it easier to add more balloon specific
1201controls without overloading `--memory`.
1202
1203### Experimental `virtio-watchdog` Support
1204
1205Support for using a new `virtio-watchdog` has been added which can be used to
1206have the VMM reboot the guest if the guest userspace fails to ping the
1207watchdog. This is enabled with `--watchdog` and requires kernel support.
1208
1209### Notable Bug Fixes
1210
1211* MTRR bit was missing from CPUID advertised to guest
1212* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170)
1213* CPU identification string is now exposed to the guest
1214* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as
1215  read only so avoids excessive VM exits (#1795)
1216* PCI device hotplug after an unplug was fixed (#1802)
1217* When using the ACPI method to resize the guest memory the full reserved size
1218  can be used (#1803)
1219* Snapshot and restore followed by a second snapshot and restore now works
1220  correctly
1221* Snapshot and restore of VMs with more than 2GiB in one region now work
1222  correctly
1223
1224### Contributors
1225
1226Many thanks to everyone who has contributed to our 0.11.0 release including some new faces.
1227
1228* Anatol Belski <anbelski@linux.microsoft.com>
1229* Bo Chen <chen.bo@intel.com>
1230* Daniel Verkamp <dverkamp@chromium.org>
1231* Henry Wang <Henry.Wang@arm.com>
1232* Hui Zhu <teawater@antfin.com>
1233* Jiangbo Wu <jiangbo.wu@intel.com>
1234* Josh Soref <jsoref@users.noreply.github.com>
1235* Julio Montes <julio.montes@intel.com>
1236* Michael Zhao <michael.zhao@arm.com>
1237* Muminul Islam <muislam@microsoft.com>
1238* pierwill <19642016+pierwill@users.noreply.github.com>
1239* Praveen Paladugu <prapal@microsoft.com>
1240* Rob Bradford <robert.bradford@intel.com>
1241* Sebastien Boeuf <sebastien.boeuf@intel.com>
1242* Wei Liu <liuwe@microsoft.com>
1243
1244
1245# v0.10.0
1246
1247This release has been tracked through the [0.10.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/13).
1248
1249Highlights for `cloud-hypervisor` version 0.10.0 include:
1250
1251### `virtio-block` Support for Multiple Descriptors
1252
1253Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers.
1254
1255### Memory Zones
1256
1257Support has been added for fine grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md).
1258
1259### `Seccomp` Sandbox Improvements
1260
1261All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`.
1262
1263### Preliminary KVM HyperV Emulation Control
1264
1265A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices.
1266
1267### Notable Bug Fixes
1268
1269- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596)
1270- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564)
1271- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593)
1272- `--version` shows the version for released binaries (#1669)
1273- Errors generated by worker threads for `virtio` devices are now printed out (#1551)
1274
1275### Contributors
1276
1277Many thanks to everyone who has contributed to our 0.10.0 release including some new faces.
1278
1279* Alyssa Ross <hi@alyssa.is>
1280* Amey Narkhede <ameynarkhede02@gmail.com>
1281* Anatol Belski <ab@php.net>
1282* Bo Chen <chen.bo@intel.com>
1283* Hui Zhu <teawater@antfin.com>
1284* Michael Zhao <michael.zhao@arm.com>
1285* Muminul Islam <muislam@microsoft.com>
1286* Rob Bradford <robert.bradford@intel.com>
1287* Samuel Ortiz <sameo@linux.intel.com>
1288* Sebastien Boeuf <sebastien.boeuf@intel.com>
1289* Wei Liu <liuwe@microsoft.com>
1290
1291# v0.9.0
1292
1293This release has been tracked through the [0.9.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/12).
1294
1295Highlights for `cloud-hypervisor` version 0.9.0 include:
1296
1297### `io_uring` Based Block Device Support
1298
1299If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement.
1300
1301### Block and Network Device Statistics
1302
1303Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM.
1304
1305### HTTP API Responses
1306
1307The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF.
1308
1309### CPU Topology
1310
1311A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core.
1312
1313### Release Build Optimization
1314
1315Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size.
1316
1317### Hypervisor Abstraction
1318
1319A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`.
1320
1321### Snapshot/Restore Improvements
1322
1323Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs.
1324
1325### Virtio Memory Ballooning Support
1326
1327A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest.
1328
1329### Enhancements to ARM64 Support
1330
1331The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting.
1332
1333### Intel SGX Support
1334
1335The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md).
1336
1337### `Seccomp` Sandbox Improvements
1338
1339The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development.
1340
1341### Notable Bug Fixes
1342
1343* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes.
1344* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338)
1345* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543).
1346* `virtio-mmio` based devices are now more widely tested (#275).
1347* Multiple issues have been fixed with virtio device configuration (#1217)
1348* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521)
1349
1350### Contributors
1351
1352Many thanks to everyone who has contributed to our 0.9.0 release including some new faces.
1353
1354* Anatol Belski <ab@php.net>
1355* Bo Chen <chen.bo@intel.com>
1356* Dr. David Alan Gilbert <dgilbert@redhat.com>
1357* Henry Wang <Henry.Wang@arm.com>
1358* Howard Zhang <howard.zhang@arm.com>
1359* Hui Zhu <teawater@antfin.com>
1360* Jianyong Wu <jianyong.wu@arm.com>
1361* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
1362* LiYa'nan <oliverliyn@gmail.com>
1363* Michael Zhao <michael.zhao@arm.com>
1364* Muminul Islam <muislam@microsoft.com>
1365* Praveen Paladugu <prapal@microsoft.com>
1366* Ricardo Koller <ricarkol@gmail.com>
1367* Rob Bradford <robert.bradford@intel.com>
1368* Samuel Ortiz <sameo@linux.intel.com>
1369* Sebastien Boeuf <sebastien.boeuf@intel.com>
1370* Stefano Garzarella <sgarzare@redhat.com>
1371* Wei Liu <liuwe@microsoft.com>
1372
1373
1374# v0.8.0
1375
1376This release has been tracked through the [0.8.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/10).
1377
1378Highlights for `cloud-hypervisor` version 0.8.0 include:
1379
1380### Experimental Snapshot and Restore Support
1381
1382This release includes the first version of the snapshot and restore feature.
1383This allows a VM to be paused and then subsequently snapshotted. At a later
1384point that snapshot may be restored into a new running VM identical to the
1385original VM at the point it was paused.
1386
1387This feature can be used for offline migration from one VM host to another, to
1388allow the upgrading or rebooting of the host machine transparently to the guest
1389or for templating the VM. This is an experimental feature and cannot be used on
1390a VM using passthrough (VFIO) devices. Issues with SMP have also been observed
1391(#1176).
1392
1393### Experimental ARM64 Support
1394
1395Included in this release is experimental support for running on ARM64.
1396Currently only `virtio-mmio` devices and a serial port are supported. Full
1397details can be found in the [ARM64 documentation](docs/arm64.md).
1398
1399### Support for Using 5-level Paging in Guests
1400
1401If the host supports it the guest is now enabled for 5-level paging (aka LA57).
1402This works when booting the Linux kernel with a vmlinux, bzImage or firmware
1403based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not
1404work due to current limitations in the PVH boot process.
1405
1406### Virtio Device Interrupt Suppression for Network Devices
1407
1408With `virtio-net` and `vhost-user-net` devices the guest can suppress
1409interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This
1410can lead to an improvement in performance by reducing the number of interrupts
1411the guest must service.
1412
1413### `vhost_user_fs` Improvements
1414
1415The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`.
1416
1417
1418### Notable Bug Fixes
1419
1420* VMs that have not yet been booted can now be deleted (#1110).
1421* By creating the `tap` device ahead of creating the VM it is not required to
1422  run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273).
1423* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to
1424  the specification and synchronizes to the underlying filesystem as required
1425  based on guest feature negotiation. This avoids potential data loss (#399,
1426  #1216).
1427* When booting with a large number of vCPUs then the ACPI table would be
1428  overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the
1429  `MPTABLE` will no longer be generated (#1132).
1430* Shutting down VMs that have been paused is now supported (#816).
1431* Created socket files are deleted on shutdown (#1083).
1432* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds
1433  (#751).
1434
1435### Command Line and API Changes
1436
1437This is non exhaustive list of HTTP API and command line changes:
1438
1439* All user visible socket parameters are now consistently called `socket`
1440  rather than `sock` in some cases.
1441* The `ch-remote` tool now shows any error message generated by the VMM
1442* The `wce` parameter has been removed from `--disk` as the feature is always
1443  offered for negotiation.
1444* `--net` has gained a `host_mac` option that allows the setting of the MAC
1445  address for the `tap` device on the host.
1446
1447### Contributors
1448
1449Many thanks to everyone who has contributed to our 0.8.0 release including some new faces.
1450
1451* Anatol Belski <ab@php.net>
1452* Arron Wang <arron.wang@intel.com>
1453* Bo Chen <chen.bo@intel.com>
1454* Dr. David Alan Gilbert <dgilbert@redhat.com>
1455* Henry Wang <Henry.Wang@arm.com>
1456* Hui Zhu <teawater@antfin.com>
1457* LiYa'nan <oliverliyn@gmail.com>
1458* Michael Zhao <michael.zhao@arm.com>
1459* Rob Bradford <robert.bradford@intel.com>
1460* Samuel Ortiz <sameo@linux.intel.com>
1461* Sebastien Boeuf <sebastien.boeuf@intel.com>
1462* Sergio Lopez <slp@redhat.com>
1463
1464# v0.7.0
1465
1466This release has been tracked through the [0.7.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7).
1467
1468Highlights for `cloud-hypervisor` version 0.7.0 include:
1469
1470### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug
1471
1472Further to our effort to support modifying a running guest we now support
1473hotplug and unplug of the following virtio backed devices: block, network,
1474pmem, virtio-fs and vsock. This functionality is available on the (default) PCI
1475based transport and is exposed through the HTTP API. The `ch-remote` utility
1476provides a CLI for adding or removing these device types after the VM has
1477booted. User can use the `id` parameter on the devices to choose names for
1478devices to ease their removal.
1479
1480### Alternative `libc` Support
1481
1482Cloud Hypervisor can now be compiled with the `musl` C library and this release
1483contains a static binary compiled using that toolchain.
1484
1485### Multithreaded Multi Queued `vhost-user` Backends
1486
1487The `vhost-user` backends for network and block support that are shipped by
1488Cloud Hypervisor have been enhanced to support multiple threads and queues to
1489improve throughput. These backends are used automatically if `vhost_user=true`
1490is passed when the devices are created.
1491
1492### Initial RamFS Support
1493
1494By passing the `--initramfs` command line option the user can specify a file to
1495be loaded into the guest memory to be used as the kernel initial filesystem.
1496This is usually used to allow the loading of drivers needed to be able to
1497access the real root filesystem but it can also be used standalone for a very
1498minimal image.
1499
1500### Alternative Memory Hotplug: `virtio-mem`
1501
1502As well as supporting ACPI based hotplug Cloud Hypervisor now supports using
1503the `virtio-mem` hotplug alternative. This can be controlled by the
1504`hotplug_method` parameter on the `--memory` command line option. It currently
1505requires kernel patches to be able to support it.
1506
1507### `Seccomp` Sandboxing
1508
1509Cloud Hypervisor now has support for restricting the system calls that the
1510process can use via the `seccomp` security API. This on by default and is
1511controlled by the `--seccomp` command line option.
1512
1513### Updated Distribution Support
1514
1515With the release of Ubuntu 20.04 we have added that to the list of supported
1516distributions and is part of our regular testing programme.
1517
1518### Command Line and API Changes
1519
1520This is non exhaustive list of HTTP API and command line changes
1521
1522* New `id` fields added for devices to allow them to be named to ease removal.
1523  If no name is specified the VMM chooses one.
1524* Use `--memory`'s `shared` and `hugepages` controls for determining backing
1525  memory instead of providing a path.
1526* The `--vsock` parameter only takes one device as the Linux kernel only
1527  supports a single Vsock device. The REST API has removed the vector for this
1528  option and replaced it with a single optional field.
1529* There is enhanced validation of the command line and API provided
1530  configurations to ensure that the provided options are compatible e.g. that
1531  shared memory is in use if any attempt is made to used a `vhost-user` backed
1532  device.
1533* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and
1534  `add-vsock` subcommands. For removal `remove-device` is used. The REST API
1535  has appropriate new HTTP endpoints too.
1536* Specifying a `size` with `--pmem` is no longer required and instead the size
1537  will be obtained from the file. A `discard_writes` option has also been added
1538  to provide the equivalent of a read-only file.
1539* The parameters to `--block-backend` have been changed to more closely align
1540  with those used by `--disk`.
1541
1542### Contributors
1543
1544Many thanks to everyone who has contributed to our 0.7.0 release including some new faces.
1545
1546* Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
1547* Bo Chen <chen.bo@intel.com>
1548* Cathy Zhang <cathy.zhang@intel.com>
1549* Damjan Georgievski <gdamjan@gmail.com>
1550* Dean Sheather <dean@coder.com>
1551* Eryu Guan <eguan@linux.alibaba.com>
1552* Hui Zhu <teawater@antfin.com>
1553* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
1554* Martin Xu <martin.xu@intel.com>
1555* Muminul Islam <muislam@microsoft.com>
1556* Rob Bradford <robert.bradford@intel.com>
1557* Samuel Ortiz <sameo@linux.intel.com>
1558* Sebastien Boeuf <sebastien.boeuf@intel.com>
1559* Sergio Lopez <slp@redhat.com>
1560* Yang Zhong <yang.zhong@intel.com>
1561* Yi Sun <yi.y.sun@linux.intel.com>
1562
1563# v0.6.0
1564
1565This release has been tracked through the [0.6.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7).
1566
1567Highlights for `cloud-hypervisor` version 0.6.0 include:
1568
1569### Directly Assigned Devices Hotplug
1570
1571We continued our efforts around supporting dynamically changing the guest
1572resources. After adding support for CPU and memory hotplug, Cloud Hypervisor
1573now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`)
1574devices into an already running guest. This closes the features gap for
1575providing a complete Kata Containers workloads support with Cloud Hypervisor.
1576
1577### Shared Filesystem Improvements
1578
1579We enhanced our shared filesystem support through many `virtio-fs` improvements.
1580By adding support for DAX, parallel processing of multiple requests, `FS_IO`,
1581`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we
1582improved our filesystem sharing performance, but also made it more stable and
1583compatible with other `virtio-fs` implementations.
1584
1585### Block and Networking IO Self Offloading
1586
1587When choosing to offload the paravirtualized block and networking I/O to an
1588external process (through the `vhost-user` protocol), Cloud Hypervisor now
1589automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends
1590into their own, separate processes.
1591This provides a seamless paravirtualized I/O user experience for those who want
1592to run their guest I/O into separate executions contexts.
1593
1594### Command Line Interface
1595
1596More and more Cloud Hypervisor services are exposed through the
1597[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only
1598accessible via relatively cumbersome HTTP calls. In order to abstract
1599those calls into a more user friendly tool, we created a Cloud Hypervisor
1600Command Line Interface (CLI) called `ch-remote`.  The `ch-remote` binary
1601is created with each build and available e.g. at
1602`cloud-hypervisor/target/debug/ch-remote` when doing a debug build.
1603
1604Please check `ch-remote --help` for a complete description of all available
1605commands.
1606
1607### PVH Boot
1608
1609In addition to the traditional Linux boot protocol, Cloud Hypervisor now
1610supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html).
1611
1612### Contributors
1613
1614With the 0.6.0 release, we are welcoming a few new contributors. Many thanks
1615to them and to everyone that contributed to this release:
1616
1617* Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
1618* Arron Wang <arron.wang@intel.com>
1619* Bin Liu <liubin0329@gmail.com>
1620* Bo Chen <chen.bo@intel.com>
1621* Cathy Zhang <cathy.zhang@intel.com>
1622* Eryu Guan <eguan@linux.alibaba.com>
1623* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
1624* Liu Bo <bo.liu@linux.alibaba.com>
1625* Qiu Wenbo <qiuwenbo@phytium.com.cn>
1626* Rob Bradford <robert.bradford@intel.com>
1627* Samuel Ortiz <sameo@linux.intel.com>
1628* Sebastien Boeuf <sebastien.boeuf@intel.com>
1629* Sergio Lopez <slp@redhat.com>
1630
1631# v0.5.1
1632
1633This is a bugfix release branched off v0.5.0. It contains the following fixes:
1634
1635* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez
1636* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf
1637* Fixed error reporting for resize command - Samuel Ortiz
1638* Double reboot workaround (#783) - Rob Bradford
1639* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford
1640
1641# v0.5.0
1642
1643This release has been tracked through the [0.5.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/6).
1644
1645Highlights for `cloud-hypervisor` version 0.5.0 include:
1646
1647### Virtual Machine Dynamic Resizing
1648
1649With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and
1650memory hot plug as well. This allows to dynamically resize Cloud Hypervisor
1651guests which is needed for e.g. Kubernetes related use cases.
1652The memory hot plug implementation is based on the same framework as the CPU hot
1653plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest.
1654
1655Next on our VM resizing roadmap is the PCI devices hotplug feature.
1656
1657### Multi-Queue, Multi-Threaded Paravirtualization
1658
1659We enhanced our virtio networking and block support by having both devices use
1660multiple I/O queues handled by multiple threads. This improves our default
1661paravirtualized networking and block devices throughput.
1662
1663### New Interrupt Management Framework
1664
1665We improved our interrupt management implementation by introducing an Interrupt
1666Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device)
1667crates discussions. This move made the code significantly cleaner, and allowed
1668us to remove several KVM related dependencies from crates like the PCI and
1669virtio ones.
1670
1671### Development Tools
1672
1673In order to provide a better developer experience, we worked on improving our
1674build, development and testing tools.
1675Somehow similar to the excellent
1676[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool),
1677we now provide a [dev_cli script](scripts/dev_cli.sh).
1678
1679With this new tool, our users and contributors will be able to build and test
1680Cloud Hypervisor through a containerized environment.
1681
1682### Kata Containers Integration
1683
1684We spent some significant time and efforts debugging and fixing our integration
1685with the [Kata Containers](https://github.com/kata-containers) project. Cloud
1686Hypervisor is now a fully supported Kata Containers hypervisor, and is
1687integrated into the project's CI.
1688
1689### Contributors
1690
1691Many thanks to everyone that contributed to the 0.5.0 release:
1692
1693* Bo Chen <chen.bo@intel.com>
1694* Cathy Zhang <cathy.zhang@intel.com>
1695* Qiu Wenbo <qiuwenbo@phytium.com.cn>
1696* Rob Bradford <robert.bradford@intel.com>
1697* Samuel Ortiz <sameo@linux.intel.com>
1698* Sebastien Boeuf <sebastien.boeuf@intel.com>
1699* Sergio Lopez <slp@redhat.com>
1700* Yang Zhong <yang.zhong@intel.com>
1701
1702# v0.4.0
1703
1704This release has been tracked through the [0.4.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/4).
1705
1706Highlights for `cloud-hypervisor` version 0.4.0 include:
1707
1708### Dynamic virtual CPUs addition
1709
1710As a way to vertically scale Cloud Hypervisor guests, we now support dynamically
1711adding virtual CPUs to the guests, a mechanism also known as CPU hot plug.
1712Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs
1713to an already running guest and the high level operations for that process are
1714documented [here](docs/hotplug.md)
1715
1716During the next release cycles we are planning to extend Cloud Hypervisor
1717hot plug framework to other resources, namely PCI devices and memory.
1718
1719### Programmatic firmware tables generation
1720
1721As part of the CPU hot plug feature enablement, and as a requirement for hot
1722plugging other resources like devices or RAM, we added support for
1723programmatically generating the needed ACPI tables. Through a dedicated
1724`acpi-tables` crate, we now have a flexible and clean way of generating those
1725tables based on the VMM device model and topology.
1726
1727### Filesystem and block devices vhost-user backends
1728
1729Our objective of running all Cloud Hypervisor paravirtualized I/O to a
1730vhost-user based framework is getting closer as we've added Rust based
1731implementations for vhost-user-blk and virtiofs backends. Together with the
1732vhost-user-net backend that came with the 0.3.0 release, this will form the
1733default Cloud Hypervisor I/O architecture.
1734
1735### Guest pause and resume
1736
1737As an initial requirement for enabling live migration, we added support for
1738pausing and resuming any VMM components. As an intermediate step towards live
1739migration, the upcoming guest snapshotting feature will be based on the pause
1740and resume capabilities.
1741
1742### Userspace IOAPIC by default
1743
1744As a way to simplify our device manager implementation, but also in order to
1745stay away from privileged rings as often as possible, any device that relies on
1746pin based interrupts will be using the userspace IOAPIC implementation by
1747default.
1748
1749### PCI BAR reprogramming
1750
1751In order to allow for a more flexible device model, and also support guests
1752that would want to move PCI devices, we added support for PCI devices BAR
1753reprogramming.
1754
1755### New `cloud-hypervisor` organization
1756
1757As we wanted to be more flexible on how we manage the Cloud Hypervisor project,
1758we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor).
1759Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor)
1760project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux)
1761and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware)
1762repositories. We may also use it to host any rust-vmm that we'd need to
1763temporarily fork.
1764Thanks to GitHub's seamless repository redirections, the move is completely
1765transparent to all Cloud Hypervisor contributors, users and followers.
1766
1767### Contributors
1768
1769Many thanks to everyone that contributed to the 0.4.0 release:
1770
1771* Cathy Zhang <cathy.zhang@intel.com>
1772* Emin Ghuliev <drmint80@gmail.com>
1773* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
1774* Qiu Wenbo <qiuwenbo@phytium.com.cn>
1775* Rob Bradford <robert.bradford@intel.com>
1776* Samuel Ortiz <sameo@linux.intel.com>
1777* Sebastien Boeuf <sebastien.boeuf@intel.com>
1778* Sergio Lopez <slp@redhat.com>
1779* Wu Zongyong <wuzongyong@linux.alibaba.com>
1780
1781# v0.3.0
1782
1783This release has been tracked through the [0.3.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/3).
1784
1785Highlights for `cloud-hypervisor` version 0.3.0 include:
1786
1787### Block device offloading
1788
1789We continue to work on offloading paravirtualized I/O to external processes,
1790and we added support for
1791[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends.
1792This enables `cloud-hypervisor` users to plug a `vhost-user` based block device
1793like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage
1794backend.
1795
1796### Network device backend
1797
1798The previous release provided support for
1799[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we
1800also provide a TAP based vhost-user-net backend, implemented in Rust. Together
1801with the vhost-user-net device implementation, this will eventually become the
1802Cloud Hypervisor default paravirtualized networking architecture.
1803
1804### Virtual sockets
1805
1806In order to more efficiently and securely communicate between host and guest,
1807we added an hybrid implementation of the
1808[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address
1809family over virtio. Credits go to the
1810[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md)
1811project as our implementation is a copy of theirs.
1812
1813### HTTP based API
1814
1815In anticipation of the need to support asynchronous operations to Cloud
1816Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP
1817based API to the VMM. The API will be more extensively documented during the
1818next release cycle.
1819
1820### Memory mapped virtio transport
1821
1822In order to support potential PCI-free use cases, we added support for the
1823[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002)
1824transport layer. This will allow us to support simple, minimal guest
1825configurations that do not require a PCI bus emulation.
1826
1827### Paravirtualized IOMMU
1828
1829As we want to improve our nested guests support, we added support for exposing
1830a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows
1831for a safer nested virtio and directly assigned devices support.
1832
1833To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor
1834users to be able to specify if devices had to be handled through this virtual
1835IOMMU or not. In particular, the `--disk` option now expects disk paths to be
1836prefixed with a `path=` string, and supports an optional `iommu=[on|off]`
1837setting.
1838
1839### Ubuntu 19.10
1840
1841With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware),
1842we can now support the latest
1843[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images.
1844
1845### Large memory guests
1846
1847After simplifying and changing our guest address space handling, we can now
1848support guests with large amount of memory (more than 64GB).
1849
1850# v0.2.0
1851
1852This release has been tracked through the [0.2.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/2).
1853
1854Highlights for `cloud-hypervisor` version 0.2.0 include:
1855
1856### Network device offloading
1857
1858As part of our general effort to offload paravirtualized I/O to external
1859processes, we added support for
1860[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This
1861enables `cloud-hypervisor` users to plug a `vhost-user` based networking device
1862(e.g. [DPDK](https://dpdk.org)) into the VMM as their virtio network backend.
1863
1864### Minimal hardware-reduced ACPI
1865
1866In order to properly implement and guest reset and shutdown, we implemented
1867a minimal version of the hardware-reduced ACPI specification. Together with
1868a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to
1869cleanly reboot and shutdown.
1870
1871The ACPI implementation is a `cloud-hypervisor` build time option that is
1872enabled by default.
1873
1874### Debug I/O port
1875
1876Based on the Firecracker idea of using a dedicated I/O port to measure guest
1877boot times, we added support for logging guest events through the
1878[0x80](https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html)
1879PC debug port. This allows, among other things, for granular guest boot time
1880measurements. See our [debug port documentation](docs/debug-port.md) for more
1881details.
1882
1883### Improved direct device assignment
1884
1885We fixed a major performance issue with our initial VFIO implementation: When
1886enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads
1887were (in many cases) not cached. After correctly tagging the guest memory from
1888`cloud-hypervisor` we're now able to reach the expected performance from
1889directly assigned devices.
1890
1891### Improved shared filesystem
1892
1893We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt)
1894support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system.
1895This provides better shared filesystem IO performance with a smaller guest
1896memory footprint.
1897
1898### Ubuntu bionic based CI
1899
1900Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware)
1901improvements, we are now able to boot Ubuntu bionic images. We added those to
1902our CI pipeline.
1903
1904# v0.1.0
1905
1906This release has been tracked through the [0.1.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/1).
1907
1908Highlights for `cloud-hypervisor` version 0.1.0 include:
1909
1910### Shared filesystem
1911
1912We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file
1913system, allowing for an efficient and reliable way of sharing a filesystem
1914between the host and the `cloud-hypervisor` guest.
1915
1916See our [filesystem sharing](docs/fs.md) documentation for more details on how
1917to use virtio-fs with `cloud-hypervisor`.
1918
1919### Initial direct device assignment support
1920
1921VFIO (Virtual Function I/O) is a kernel framework that exposes direct device
1922access to userspace. `cloud-hypervisor` uses VFIO to directly assign host
1923physical devices into its guest.
1924
1925See our [VFIO](docs/vfio.md) documentation for more detail on how to directly
1926assign host devices to `cloud-hypervisor` guests.
1927
1928### Userspace IOAPIC
1929
1930`cloud-hypervisor` supports a so-called split IRQ chip implementation by
1931implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC).
1932By moving part of the IRQ chip implementation from kernel space to user space,
1933the IRQ chip emulation does not always run in a fully privileged mode.
1934
1935### Virtual persistent memory
1936
1937The `virtio-pmem` implementation emulates a virtual persistent memory device
1938that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device
1939allows to bypass the guest page cache and improve the guest memory footprint.
1940
1941### Linux kernel bzImage
1942
1943The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from
1944`bzImage` kernel images, which is usually the format that Linux distributions
1945use to ship their kernels. For example, this allows for booting from the host
1946distribution kernel image.
1947
1948### Console over virtio
1949
1950`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although
1951using this device as a guest console can potentially cut some early boot
1952messages, it can reduce the guest boot time and provides a complete console
1953implementation.
1954
1955The `virtio-console` device is enabled by default for the guest console.
1956Switching back to the legacy serial port is done by selecting
1957`--serial tty --console off` from the command line.
1958
1959### Unit testing
1960
1961We now run all unit tests from all our crates directly from our CI.
1962
1963### Integration tests parallelization
1964
1965The CI cycle run time has been significantly reduced by refactoring our
1966integration tests; allowing them to all be run in parallel.
1967