1- [v25.0](#v250) 2 - [`ch-remote` Improvements](#ch-remote-improvements) 3 - [VM "Coredump" Support](#vm-coredump-support) 4 - [Notable Bug Fixes](#notable-bug-fixes) 5 - [Removals](#removals) 6 - [Contributors](#contributors) 7- [v24.0](#v240) 8 - [Bypass Mode for `virtio-iommu`](#bypass-mode-for-virtio-iommu) 9 - [Ensure Identifiers Uniqueness](#ensure-identifiers-uniqueness) 10 - [Sparse Mmap support](#sparse-mmap-support) 11 - [Expose Platform Serial Number](#expose-platform-serial-number) 12 - [Notable Bug Fixes](#notable-bug-fixes-1) 13 - [Notable Improvements](#notable-improvements) 14 - [Deprecations](#deprecations) 15 - [New on the Website](#new-on-the-website) 16 - [Contributors](#contributors-1) 17- [v23.1](#v231) 18- [v23.0](#v230) 19 - [vDPA Support](#vdpa-support) 20 - [Updated OS Support list](#updated-os-support-list) 21 - [`AArch64` Memory Map Improvements](#aarch64-memory-map-improvements) 22 - [`AMX` Support](#amx-support) 23 - [Notable Bug Fixes](#notable-bug-fixes-2) 24 - [Deprecations](#deprecations-1) 25 - [Contributors](#contributors-2) 26- [v22.1](#v221) 27- [v22.0](#v220) 28 - [GDB Debug Stub Support](#gdb-debug-stub-support) 29 - [`virtio-iommu` Backed Segments](#virtio-iommu-backed-segments) 30 - [Before Boot Configuration Changes](#before-boot-configuration-changes) 31 - [`virtio-balloon` Free Page Reporting](#virtio-balloon-free-page-reporting) 32 - [Support for Direct Kernel Booting with TDX](#support-for-direct-kernel-booting-with-tdx) 33 - [PMU Support for AArch64](#pmu-support-for-aarch64) 34 - [Documentation Under CC-BY-4.0 License](#documentation-under-cc-by-40-license) 35 - [Deprecation of "Classic" `virtiofsd`](#deprecation-of-classic-virtiofsd) 36 - [Notable Bug Fixes](#notable-bug-fixes-3) 37 - [Contributors](#contributors-3) 38- [v21.0](#v210) 39 - [Efficient Local Live Migration (for Live Upgrade)](#efficient-local-live-migration-for-live-upgrade) 40 - [Recommended Kernel is Now 5.15](#recommended-kernel-is-now-515) 41 - [Notable Bug fixes](#notable-bug-fixes-4) 42 - [Contributors](#contributors-4) 43- [v20.2](#v202) 44- [v20.1](#v201) 45- [v20.0](#v200) 46 - [Multiple PCI segments support](#multiple-pci-segments-support) 47 - [CPU pinning](#cpu-pinning) 48 - [Improved VFIO support](#improved-vfio-support) 49 - [Safer code](#safer-code) 50 - [Extended documentation](#extended-documentation) 51 - [Notable bug fixes](#notable-bug-fixes-5) 52 - [Contributors](#contributors-5) 53- [v19.0](#v190) 54 - [Improved PTY handling for serial and `virtio-console`](#improved-pty-handling-for-serial-and-virtio-console) 55 - [PCI boot time optimisations](#pci-boot-time-optimisations) 56 - [Improved TDX support](#improved-tdx-support) 57 - [Live migration enhancements](#live-migration-enhancements) 58 - [`virtio-mem` support with `vfio-user`](#virtio-mem-support-with-vfio-user) 59 - [AArch64 for `virtio-iommu`](#aarch64-for-virtio-iommu) 60 - [Notable bug fixes](#notable-bug-fixes-6) 61 - [Contributors](#contributors-6) 62- [v18.0](#v180) 63 - [Experimental User Device (`vfio-user`) support](#experimental-user-device-vfio-user-support) 64 - [Migration support for `vhost-user` devices](#migration-support-for-vhost-user-devices) 65 - [VHDX disk image support](#vhdx-disk-image-support) 66 - [Device pass through on MSHV hypervisor](#device-pass-through-on-mshv-hypervisor) 67 - [AArch64 for support `virtio-mem`](#aarch64-for-support-virtio-mem) 68 - [Live migration on MSHV hypervisor](#live-migration-on-mshv-hypervisor) 69 - [AArch64 CPU topology support](#aarch64-cpu-topology-support) 70 - [Power button support on AArch64](#power-button-support-on-aarch64) 71 - [Notable bug fixes](#notable-bug-fixes-7) 72 - [Contributors](#contributors-7) 73- [v17.0](#v170) 74 - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi) 75 - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend) 76 - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices) 77 - [Improved SGX support](#improved-sgx-support) 78 - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices) 79 - [Notable bug fixes](#notable-bug-fixes-8) 80 - [Contributors](#contributors-8) 81- [v16.0](#v160) 82 - [Improved live migration support](#improved-live-migration-support) 83 - [Improved `vhost-user` support](#improved-vhost-user-support) 84 - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support) 85 - [Notable bug fixes](#notable-bug-fixes-9) 86 - [Removed functionality](#removed-functionality) 87 - [Contributors](#contributors-9) 88- [v15.0](#v150) 89 - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees) 90 - [Network device rate limiting](#network-device-rate-limiting) 91 - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload) 92 - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter) 93 - [Bug fixes](#bug-fixes) 94 - [Deprecations](#deprecations-2) 95 - [Contributors](#contributors-10) 96- [v0.14.1](#v0141) 97- [v0.14.0](#v0140) 98 - [Structured event monitoring](#structured-event-monitoring) 99 - [MSHV improvements](#mshv-improvements) 100 - [Improved aarch64 platform](#improved-aarch64-platform) 101 - [Updated hotplug documentation](#updated-hotplug-documentation) 102 - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console) 103 - [Block device rate limiting](#block-device-rate-limiting) 104 - [Deprecations](#deprecations-3) 105 - [Contributors](#contributors-11) 106- [v0.13.0](#v0130) 107 - [Wider VFIO device support](#wider-vfio-device-support) 108 - [Improved huge page support](#improved-huge-page-support) 109 - [MACvTAP support](#macvtap-support) 110 - [VHD disk image support](#vhd-disk-image-support) 111 - [Improved Virtio device threading](#improved-virtio-device-threading) 112 - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button) 113 - [Contributors](#contributors-12) 114- [v0.12.0](#v0120) 115 - [ARM64 enhancements](#arm64-enhancements) 116 - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning) 117 - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend) 118 - [Enhanced "info" API](#enhanced-info-api) 119 - [Contributors](#contributors-13) 120- [v0.11.0](#v0110) 121 - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block) 122 - [Windows Guest Support](#windows-guest-support) 123 - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation) 124 - [`virtio-mmio` Removal](#virtio-mmio-removal) 125 - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64) 126 - [Improved Linux Boot Time](#improved-linux-boot-time) 127 - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling) 128 - [Default Log Level Changed](#default-log-level-changed) 129 - [New `--balloon` Parameter Added](#new---balloon-parameter-added) 130 - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support) 131 - [Notable Bug Fixes](#notable-bug-fixes-10) 132 - [Contributors](#contributors-14) 133- [v0.10.0](#v0100) 134 - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors) 135 - [Memory Zones](#memory-zones) 136 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements) 137 - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control) 138 - [Notable Bug Fixes](#notable-bug-fixes-11) 139 - [Contributors](#contributors-15) 140- [v0.9.0](#v090) 141 - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support) 142 - [Block and Network Device Statistics](#block-and-network-device-statistics) 143 - [HTTP API Responses](#http-api-responses) 144 - [CPU Topology](#cpu-topology) 145 - [Release Build Optimization](#release-build-optimization) 146 - [Hypervisor Abstraction](#hypervisor-abstraction) 147 - [Snapshot/Restore Improvements](#snapshotrestore-improvements) 148 - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support) 149 - [Enhancements to ARM64 Support](#enhancements-to-arm64-support) 150 - [Intel SGX Support](#intel-sgx-support) 151 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1) 152 - [Notable Bug Fixes](#notable-bug-fixes-12) 153 - [Contributors](#contributors-16) 154- [v0.8.0](#v080) 155 - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support) 156 - [Experimental ARM64 Support](#experimental-arm64-support) 157 - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests) 158 - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices) 159 - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements) 160 - [Notable Bug Fixes](#notable-bug-fixes-13) 161 - [Command Line and API Changes](#command-line-and-api-changes) 162 - [Contributors](#contributors-17) 163- [v0.7.0](#v070) 164 - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug) 165 - [Alternative `libc` Support](#alternative-libc-support) 166 - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends) 167 - [Initial RamFS Support](#initial-ramfs-support) 168 - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem) 169 - [`Seccomp` Sandboxing](#seccomp-sandboxing) 170 - [Updated Distribution Support](#updated-distribution-support) 171 - [Command Line and API Changes](#command-line-and-api-changes-1) 172 - [Contributors](#contributors-18) 173- [v0.6.0](#v060) 174 - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug) 175 - [Shared Filesystem Improvements](#shared-filesystem-improvements) 176 - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading) 177 - [Command Line Interface](#command-line-interface) 178 - [PVH Boot](#pvh-boot) 179 - [Contributors](#contributors-19) 180- [v0.5.1](#v051) 181- [v0.5.0](#v050) 182 - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing) 183 - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization) 184 - [New Interrupt Management Framework](#new-interrupt-management-framework) 185 - [Development Tools](#development-tools) 186 - [Kata Containers Integration](#kata-containers-integration) 187 - [Contributors](#contributors-20) 188- [v0.4.0](#v040) 189 - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition) 190 - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation) 191 - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends) 192 - [Guest pause and resume](#guest-pause-and-resume) 193 - [Userspace IOAPIC by default](#userspace-ioapic-by-default) 194 - [PCI BAR reprogramming](#pci-bar-reprogramming) 195 - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization) 196 - [Contributors](#contributors-21) 197- [v0.3.0](#v030) 198 - [Block device offloading](#block-device-offloading) 199 - [Network device backend](#network-device-backend) 200 - [Virtual sockets](#virtual-sockets) 201 - [HTTP based API](#http-based-api) 202 - [Memory mapped virtio transport](#memory-mapped-virtio-transport) 203 - [Paravirtualized IOMMU](#paravirtualized-iommu) 204 - [Ubuntu 19.10](#ubuntu-1910) 205 - [Large memory guests](#large-memory-guests) 206- [v0.2.0](#v020) 207 - [Network device offloading](#network-device-offloading) 208 - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi) 209 - [Debug I/O port](#debug-io-port) 210 - [Improved direct device assignment](#improved-direct-device-assignment) 211 - [Improved shared filesystem](#improved-shared-filesystem) 212 - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci) 213- [v0.1.0](#v010) 214 - [Shared filesystem](#shared-filesystem) 215 - [Initial direct device assignment support](#initial-direct-device-assignment-support) 216 - [Userspace IOAPIC](#userspace-ioapic) 217 - [Virtual persistent memory](#virtual-persistent-memory) 218 - [Linux kernel bzImage](#linux-kernel-bzimage) 219 - [Console over virtio](#console-over-virtio) 220 - [Unit testing](#unit-testing) 221 - [Integration tests parallelization](#integration-tests-parallelization) 222 223# v25.0 224 225This release has been tracked through the [v25.0 226project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/29). 227 228### `ch-remote` Improvements 229 230The `ch-remote` command has gained support for creating the VM from a JSON 231config and support for booting and deleting the VM from the VMM. 232 233### VM "Coredump" Support 234 235Under the `guest_debug` feature flag it is now possible to extract the memory 236of the guest for use in debugging with e.g. the `crash` utility. (#4012) 237 238### Notable Bug Fixes 239 240* Always restore console mode on exit (#4249, #4248) 241* Restore vCPUs in numerical order which fixes aarch64 snapshot/restore (#4244) 242* Don't try and configure `IFF_RUNNING` on TAP devices (#4279) 243* Propagate configured queue size through to vhost-user backend (#4286) 244* Always Program vCPU CPUID before running the vCPU to fix running on Linux 245 5.16 (#4156) 246* Enable ACPI MADT "Online Capable" flag for hotpluggable vCPUs to fix newer 247 Linux guest 248 249### Removals 250 251The following functionality has been removed: 252 253* The `mergeable` option from the `virtio-pmem` support has been removed 254 (#3968) 255* The `dax` option from the `virtio-fs` support has been removed (#3889) 256 257### Contributors 258 259Many thanks to everyone who has contributed to our release: 260 261* Dylan Bargatze <dbargatz@users.noreply.github.com> 262* Jinank Jain <jinankjain@microsoft.com> 263* Michael Zhao <michael.zhao@arm.com> 264* Rob Bradford <robert.bradford@intel.com> 265* Sebastien Boeuf <sebastien.boeuf@intel.com> 266* Wei Liu <liuwe@microsoft.com> 267* Yi Wang <wang.yi59@zte.com.cn> 268 269# v24.0 270 271This release has been tracked through the [v24.0 272project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/28). 273 274### Bypass Mode for `virtio-iommu` 275 276`virtio-iommu` specification describes how a device can be attached by default 277to a bypass domain. This feature is particularly helpful for booting a VM with 278guest software which doesn't support `virtio-iommu` but still need to access 279the device. Now that Cloud Hypervisor supports this feature, it can boot a VM 280with Rust Hypervisor Firmware or OVMF even if the `virtio-block` device exposing 281the disk image is placed behind a virtual IOMMU. 282 283### Ensure Identifiers Uniqueness 284 285Multiple checks have been added to the code to prevent devices with identical 286identifiers from being created, and therefore avoid unexpected behaviors at boot 287or whenever a device was hot plugged into the VM. 288 289### Sparse Mmap support 290 291Sparse mmap support has been added to both VFIO and vfio-user devices. This 292allows the device regions that are not fully mappable to be partially mapped. 293And the more a device region can be mapped into the guest address space, the 294fewer VM exits will be generated when this device is accessed. This directly 295impacts the performance related to this device. 296 297### Expose Platform Serial Number 298 299A new `serial_number` option has been added to `--platform`, allowing a user to 300set a specific serial number for the platform. This number is exposed to the 301guest through the SMBIOS. 302 303### Notable Bug Fixes 304 305* Fix loading RAW firmware (#4072) 306* Reject compressed QCOW images (#4055) 307* Reject virtio-mem resize if device is not activated (#4003) 308* Fix potential mmap leaks from VFIO/vfio-user MMIO regions (#4069) 309* Fix algorithm finding HOB memory resources (#3983) 310 311### Notable Improvements 312 313* Refactor interrupt handling (#4083) 314* Load kernel asynchronously (#4022) 315* Only create ACPI memory manager DSDT when resizable (#4013) 316 317### Deprecations 318 319Deprecated features will be removed in a subsequent release and users should 320plan to use alternatives 321 322* The `mergeable` option from the `virtio-pmem` support has been deprecated 323 (#3968) 324* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 325 326### New on the Website 327 328A new blog post [Achieving Bare Metal Performance Within a Virtual 329Machine](https://www.cloudhypervisor.org/blog/achieving-bare-metal-performance-within-a-virtual-machine) 330has been added to the Cloud Hypervisor website. 331 332### Contributors 333 334Many thanks to everyone who has contributed to our release: 335 336* Anatol Belski <anbelski@linux.microsoft.com> 337* Bo Chen <chen.bo@intel.com> 338* Fabiano Fidêncio <fabiano.fidencio@intel.com> 339* LiHui <andrewli@kubesphere.io> 340* Maksym Pavlenko <pavlenko.maksym@gmail.com> 341* Rob Bradford <robert.bradford@intel.com> 342* Sebastien Boeuf <sebastien.boeuf@intel.com> 343* Steven Dake <steven.dake@gmail.com> 344* Vincent Batts <vbatts@hashbangbash.com> 345* Wei Liu <liuwe@microsoft.com> 346 347# v23.1 348 349This is a bug fix release. The following issues have been addressed: 350 351* Add some missing seccomp rules 352* Remove `virtio-fs` filesystem entries from config on removal 353* Do not delete API socket on API server start (#4026) 354* Reject `virtio-mem` resize if the guest doesn't activate the device 355* Fix OpenAPI naming of I/O throttling knobs 356 357# v23.0 358 359This release has been tracked through the [v23.0 360project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/27). 361 362### vDPA Support 363 364A vDPA device has a datapath that complies with the virtio specification but 365with a vendor specific control path. The addition of `--vdpa` and the REST API 366equivalent allows the use of these devices with Cloud Hypervisor. 367 368### Updated OS Support list 369 370The list of officially supported and tested OS versions has been updated to 371include Ubuntu "jammy" 22.04 and EOLed versions removed. 372 373### `AArch64` Memory Map Improvements 374 375The memory map when running on `AArch64` has been improved for the handling of 376the UEFI region which means that the booted guest OS now has full access to its 377allocated RAM. (#3938) 378 379### `AMX` Support 380 381Under a compile time gate of `amx` it is possible compile in support for the 382`AMX` instruction set extension for guest use. This also requires runtime 383enabling with `--cpu features=amx`. 384 385### Notable Bug Fixes 386 387* Generate error when incorrect HTTP method used for some API endpoints (#3887) 388* CMOS based reset is now available to support rebooting on "jammy" (#3914) 389* ACPI tables are not produced for memory hotplug when running with 390 `virtio-mem` (#3883) 391* `virtio-iommu` backed PCI segments are now comprehensively placed behind the 392 vIOMMU (#3870) 393* Seccomp rules have been extended for `virtio-fs` to support direct access 394 (#3848) 395 396### Deprecations 397 398Deprecated features will be removed in a subsequent release and users should 399plan to use alternatives 400 401* The `mergeable` option from the `virtio-pmem` support has been deprecated 402 (#3968) 403* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 404 405### Contributors 406 407Many thanks to everyone who has contributed to our release: 408 409* Bo Chen <chen.bo@intel.com> 410* Fabiano Fidêncio <fabiano.fidencio@intel.com> 411* Henry Wang <Henry.Wang@arm.com> 412* Jianyong Wu <jianyong.wu@arm.com> 413* LiHui <andrewli@kubesphere.io> 414* Michael Zhao <michael.zhao@arm.com> 415* Rob Bradford <robert.bradford@intel.com> 416* Sebastien Boeuf <sebastien.boeuf@intel.com> 417* Wei Liu <liuwe@microsoft.com> 418* William Douglas <william.douglas@intel.com> 419* Yi Wang <wang.yi59@zte.com.cn> 420 421# v22.1 422 423This is a bug fix release. The following issues have been addressed: 424 425* VFIO ioctl reordering to fix MSI on AMD platforms (#3827) 426* Fix `virtio-net` control queue (#3829) 427 428# v22.0 429 430This release has been tracked through the [v22.0 431project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/25). 432 433### GDB Debug Stub Support 434 435Cloud Hypervisor can now be used as debug target with GDB. This is controlled 436by the `gdb` compile time feature and details of how to use it can be found in 437the [gdb 438documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/gdb.md). 439 440### `virtio-iommu` Backed Segments 441 442In order to facilitate hotplug devices that require being behind an IOMMU (e.g. 443QAT) there is a new option `--platform iommu_segments=<list_of_segments>` that 444will place all the specified segments behind the IOMMU. 445 446### Before Boot Configuration Changes 447 448It is now possible to change the VM configuration (e.g. add or remove devices, 449resize) before the VM is booted. 450 451### `virtio-balloon` Free Page Reporting 452 453If `--balloon free_page_reporting=on` is used then the guest can report pages 454that is it not using to the VMM. The VMM will then notify the host OS that 455those pages are no longer in use and can be freed. This can result in improved 456memory density. 457 458### Support for Direct Kernel Booting with TDX 459 460Through the use of `TD-Shim` lightweight firmware it is now possible to 461directly boot into the kernel with TDX. The [TDX 462documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/intel_tdx.md#tdshim) 463has been updated for this usage. 464 465### PMU Support for AArch64 466 467A PMU is now available on AArch64 for guest performance profiling. This will be 468exposed automatically if available from the host. 469 470### Documentation Under CC-BY-4.0 License 471 472The documentation is now licensed under the "Creative Commons Attribution 4.0 473International" license which is aligned with the project charter under the 474Linux Foundation. 475 476### Deprecation of "Classic" `virtiofsd` 477 478The use of the Rust based [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd) 479is now recommended and we are no longer testing against the C based "classic" 480version. 481 482### Notable Bug Fixes 483 484* Can now be used on kernels without `AF_INET` support (#3785) 485* `virtio-balloon` size is now validated against guest RAM size (#3689) 486* Ensure that I/O related KVM VM Exits are correctly handled (#3677) 487* Multiple TAP file descriptors can be used for `virtio-net` device hotplug (#3607) 488* Minor API improvements and fixes (#3756, #3766, #3647, #3578) 489* Fix sporadic seccomp violation from glibc memory freeing (#3610, #3609) 490* Fix Windows 11 on AArch64 due to wider MSI-X register accesses (#3714, #3720) 491* Ensure `vhost-user` features are correct across migration (#3737) 492* Improved vCPU topology on AArch64 (#3735, #3733) 493 494### Contributors 495 496Many thanks to everyone who has contributed to our release: 497 498* Akira Moroo <retrage01@gmail.com> 499* Barret Rhoden <brho@google.com> 500* Bo Chen <chen.bo@intel.com> 501* Fabiano Fidêncio <fabiano.fidencio@intel.com> 502* Feng Ye <yefeng@smartx.com> 503* Henry Wang <Henry.Wang@arm.com> 504* Jianyong Wu <jianyong.wu@arm.com> 505* lizhaoxin1 <Lxiaoyouling@163.com> 506* Michael Zhao <michael.zhao@arm.com> 507* Rob Bradford <robert.bradford@intel.com> 508* Sebastien Boeuf <sebastien.boeuf@intel.com> 509* Wei Liu <liuwe@microsoft.com> 510 511# v21.0 512 513This release has been tracked through the [v21.0 514project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/24). 515 516### Efficient Local Live Migration (for Live Upgrade) 517 518In order to support fast live upgrade of the VMM an optimised path has been 519added in which the memory for the VM is not compared from source to 520destination. This is activated by passing `--local` to the `ch-remote 521send-migration` command. This means that the live upgrade can complete in the 522order of 50ms vs 3s. (#3566) 523 524### Recommended Kernel is Now 5.15 525 526Due to an issue in the `virtio-net` code in 5.14 the recommended Linux kernel 527is now 5.15. (#3530) 528 529### Notable Bug fixes 530 531* Multiple fixes were made to the OpenAPI YAML file to match the implementaion (#3555,#3562) 532* Avoid live migration deadlock when triggered during the kernel boot (#3585) 533* Support live migration within firmware (#3586) 534* Validate the `virtio-net` desciptor chain (#3548) 535* `direct=on` (`O_DIRECT`) can now be used with a guest that makes unaligned accesses (e.g. firmware) (#3587) 536 537### Contributors 538 539Many thanks to everyone who has contributed to our release: 540 541* Anatol Belski <anbelski@linux.microsoft.com> 542* Barret Rhoden <brho@google.com> 543* Bo Chen <chen.bo@intel.com> 544* Fabiano Fidêncio <fabiano.fidencio@intel.com> 545* Henry Wang <Henry.Wang@arm.com> 546* Liang Zhou <zhoul110@chinatelecom.cn> 547* Michael Zhao <michael.zhao@arm.com> 548* Muhammad Falak R Wani <falakreyaz@gmail.com> 549* Muminul Islam <muislam@microsoft.com> 550* Rob Bradford <robert.bradford@intel.com> 551* Sebastien Boeuf <sebastien.boeuf@intel.com> 552* Wei Liu <liuwe@microsoft.com> 553* Ziye Yang <ziye.yang@intel.com> 554 555# v20.2 556 557This is a bug fix release. The following issues have been addressed: 558 559* Don't error out when setting up the SIGWINCH handler (for console resize) 560 when this fails due to older kernel (#3456) 561* Seccomp rules were refined to remove syscalls that are now unused 562* Fix reboot on older host kernels when SIGWINCH handler was not initialised 563 (#3496) 564* Fix virtio-vsock blocking issue (#3497) 565 566# v20.1 567 568This is a bug fix release. The following issues have been addressed: 569 570* Networking performance regression with `virtio-net` (#3450) 571* Limit file descriptors sent in `vfio-user` support (#3401) 572* Fully advertise PCI MMIO config regions in ACPI tables (#3432) 573* Set the TSS and KVM identity maps so they don't overlap with firmware RAM 574* Correctly update the `DeviceTree` on restore 575 576# v20.0 577 578This release has been tracked through the [v20.0 579project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/23). 580 581### Multiple PCI segments support 582 583Cloud Hypervisor is no longer limited to 31 PCI devices. For both `x86_64` and 584`aarch64` architectures, it is now possible to create up to 16 PCI segments, 585increasing the total amount of supported PCI devices to 496. 586 587### CPU pinning 588 589For each vCPU, the user can define a limited set of host CPUs on which it is 590allowed to run. This can be useful when assigning a 1:1 mapping between host and 591guest resources, or when running a VM on a specific NUMA node. 592 593### Improved VFIO support 594 595Based on VFIO region capabilities, all regions can be memory mapped, limiting 596the amount of triggered VM exits, and therefore increasing the performance of 597the passthrough device. 598 599### Safer code 600 601Several sections containing unsafe Rust code have been replaced with safe 602alternatives, and multiple comments have been added to clarify why the remaining 603unsafe sections are safe to use. 604 605### Extended documentation 606 607The documentation related to VFIO has been updated while some new documents have 608been introduced to cover the usage of `--cpus` parameter as well as how to run 609Cloud Hypervisor on Intel TDX. 610 611### Notable bug fixes 612 613* Naturally align PCI BARs on relocation (#3244) 614* Fix panic in SIGWINCH listener thread when no seccomp filter set (#3338) 615* Use the tty raw mode implementation from libc (#3344) 616* Fix the emulation of register D for CMOS/RTC device (#3393) 617 618### Contributors 619 620Many thanks to everyone who has contributed to our release: 621 622* Alyssa Ross <hi@alyssa.is> 623* Bo Chen <chen.bo@intel.com> 624* Fabiano Fidêncio <fabiano.fidencio@intel.com> 625* Michael Zhao <michael.zhao@arm.com> 626* Muminul Islam <muislam@microsoft.com> 627* Rob Bradford <robert.bradford@intel.com> 628* Sebastien Boeuf <sebastien.boeuf@intel.com> 629* Wei Liu <liuwe@microsoft.com> 630* Willen Yang <willenyang@gmail.com> 631* William Douglas <william.douglas@intel.com> 632* Ziye Yang <ziye.yang@intel.com> 633 634# v19.0 635 636This release has been tracked through the [v19.0 637project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/22). 638 639### Improved PTY handling for serial and `virtio-console` 640 641The PTY support for serial has been enhanced with improved buffering when the 642the PTY is not yet connected to. Using `virtio-console` with PTY now results in 643the console being resized if the PTY window is also resized. 644 645### PCI boot time optimisations 646 647Multiple optimisations have been made to the PCI handling resulting in 648significant improvements in the boot time of the guest. 649 650### Improved TDX support 651 652When using the latest TDVF firmware the ACPI tables created by the VMM are now 653exposed via the firmware to the guest. 654 655### Live migration enhancements 656 657Live migration support has been enhanced to support migration with `virtio-mem` 658based memory hotplug and the `virtio-balloon` device now supports live 659migration. 660 661### `virtio-mem` support with `vfio-user` 662 663The use of `vfio-user` userspaces devices can now be used in conjunction with 664`virtio-mem` based memory hotplug and unplug. 665 666### AArch64 for `virtio-iommu` 667 668A paravirtualised IOMMU can now be used on the AArch64 platform. 669 670### Notable bug fixes 671 672* ACPI hotplugged memory is correctly restored after a live migration or 673 snapshot/restore (#3165) 674* Multiple devices from the same IOMMU group can be passed through via VFIO 675 (#3078 #3113) 676* Live migration with large blocks of memory was buggy due to an in issue in 677 the underlying crate (#3157) 678 679### Contributors 680 681Many thanks to everyone who has contributed to our release: 682 683* Alyssa Ross <hi@alyssa.is> 684* Bo Chen <chen.bo@intel.com> 685* Henry Wang <Henry.Wang@arm.com> 686* Hui Zhu <teawater@antfin.com> 687* Jianyong Wu <jianyong.wu@arm.com> 688* Li Yu <liyu.yukiteru@bytedance.com> 689* Michael Zhao <michael.zhao@arm.com> 690* Muminul Islam <muislam@microsoft.com> 691* Rob Bradford <robert.bradford@intel.com> 692* Sebastien Boeuf <sebastien.boeuf@intel.com> 693* Wei Liu <liuwe@microsoft.com> 694* William Douglas <william.douglas@intel.com> 695* Yu Li <liyu.yukiteru@bytedance.com> 696 697# v18.0 698 699This release has been tracked through the [v18.0 700project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/21). 701 702### Experimental User Device (`vfio-user`) support 703 704Experimental support for running PCI devices in userspace via `vfio-user` 705has been included. This allows the use of the SPDK NVMe `vfio-user` controller 706with Cloud Hypervisor. This is enabled by `--user-device` on the command line. 707 708### Migration support for `vhost-user` devices 709 710Devices exposed into the VM via `vhost-user` can now be migrated using the live 711migration support. This requires support from the backend however the commonly 712used DPDK `vhost-user` backend does support this. 713 714### VHDX disk image support 715 716Images using the VHDX disk image format can now be used with Cloud Hypervisor. 717 718### Device pass through on MSHV hypervisor 719 720When running on the MSHV hypervisor it is possible to pass through devices from 721the host through to the guest (e.g with `--device`) 722 723### AArch64 for support `virtio-mem` 724 725The reference Linux kernel we recommend for using with Cloud Hypervisor now supports `virtio-mem` on AArch64. 726 727### Live migration on MSHV hypervisor 728 729Live migration is now supported when running on the MSHV hypervisor including 730efficient tracking of dirty pages. 731 732### AArch64 CPU topology support 733 734The CPU topology (as configured through `--cpu topology=`) can now be 735configured on AArch64 platforms and is conveyed through either ACPI or device 736tree. 737 738### Power button support on AArch64 739 740Use of the ACPI power button (e.g `ch-remote --api-socket=<API socket> power-button`) 741is now supported when running on AArch64. 742 743### Notable bug fixes 744 745* Using two PTY outputs e.g. `--serial pty --console pty` now works correctly (#3012) 746* TTY input is now always sent to the correct destination (#3005) 747* The boot is no longer blocked when using a unattached PTY on the serial console (#3004) 748* Live migration is now supported on AArch64 (#3049) 749* Ensure signal handlers are run on the correct thread (#3069) 750 751### Contributors 752 753Many thanks to everyone who has contributed to our release: 754 755* Alyssa Ross <hi@alyssa.is> 756* Anatol Belski <anbelski@linux.microsoft.com> 757* Arafatms <arafatms@outlook.com> 758* Bo Chen <chen.bo@intel.com> 759* Fazla Mehrab <akm.fazla.mehrab@vt.edu> 760* Henry Wang <Henry.Wang@arm.com> 761* Jianyong Wu <jianyong.wu@arm.com> 762* Jiaqi Gao <jiaqi.gao@intel.com> 763* Markus Theil <markus.theil@tu-ilmenau.de> 764* Muminul Islam <muislam@microsoft.com> 765* Rob Bradford <robert.bradford@intel.com> 766* Sebastien Boeuf <sebastien.boeuf@intel.com> 767* Wei Liu <liuwe@microsoft.com> 768* Yu Li <liyu.yukiteru@bytedance.com> 769 770# v17.0 771 772This release has been tracked through the [v17.0 773project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/20). 774 775### ARM64 NUMA support using ACPI 776 777The support for ACPI on ARM64 has been enhanced to include support for 778specifying a NUMA configuration using the existing control options. 779 780### `Seccomp` support for MSHV backend 781 782The `seccomp` rules have now been extended to support running against the MSHV 783hypervisor backend. 784 785### Hotplug of `macvtap` devices 786 787Hotplug of `macvtap` devices is now supported with the file descriptor for the 788network device if opened by the user and passed to the VMM. The `ch-remote` 789tool supports this functionality when adding a network device. 790 791### Improved SGX support 792 793The SGX support has been updated to match the latest Linux kernel support and 794now supports SGX provisioning and associating EPC sections to NUMA nodes. 795 796### Inflight tracking for `vhost-user` devices 797 798Support for handling inflight tracking of I/O requests has been added to the 799`vhost-user` devices allowing recovery after device reconnection. 800 801### Notable bug fixes 802 803* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821). 804* The VMM side of `vhost-user` devices no longer advertise the 805 `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM 806(#2833). 807* On ARM64 VMs can be created with more than 16 vCPUs (#2763). 808 809### Contributors 810 811Many thanks to everyone who has contributed to our release: 812 813* Anatol Belski <anbelski@linux.microsoft.com> 814* Arafatms <arafatms@outlook.com> 815* Bo Chen <chen.bo@intel.com> 816* Fei Li <lifei.shirley@bytedance.com> 817* Henry Wang <Henry.Wang@arm.com> 818* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 819* Jianyong Wu <jianyong.wu@arm.com> 820* Li Hangjing <lihangjing@bytedance.com> 821* Michael Zhao <michael.zhao@arm.com> 822* Muminul Islam <muislam@microsoft.com> 823* Rob Bradford <robert.bradford@intel.com> 824* Sebastien Boeuf <sebastien.boeuf@intel.com> 825* Wei Liu <liuwe@microsoft.com> 826* Yukiteru <wfly1998@sina.com> 827 828# v16.0 829 830This release has been tracked through the [v16.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/19). 831 832### Improved live migration support 833 834The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details. 835 836### Improved `vhost-user` support 837 838When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch. 839 840### ARM64 ACPI and UEFI support 841 842Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot. 843 844### Notable bug fixes 845 846* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578). 847* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563). 848* The KVM clock is now correctly handled during live migration / snapshot & restore. 849 850### Removed functionality 851 852The following formerly deprecated features have been removed: 853 854* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 855 binaries has been deprecated. When using direct boot users should configure 856 their kernel with `CONFIG_PVH=y`. 857 858### Contributors 859 860Many thanks to everyone who has contributed to our release including some new faces. 861 862* Anatol Belski <anbelski@linux.microsoft.com> 863* Bo Chen <chen.bo@intel.com> 864* Dayu Liu <liu.dayu@zte.com.cn> 865* Henry Wang <Henry.Wang@arm.com> 866* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 867* Jianyong Wu <jianyong.wu@arm.com> 868* Michael Zhao <michael.zhao@arm.com> 869* Mikko Ylinen <mikko.ylinen@intel.com> 870* Muminul Islam <muislam@microsoft.com> 871* Ren Lei <ren.lei4@zte.com.cn> 872* Rob Bradford <robert.bradford@intel.com> 873* Sebastien Boeuf <sebastien.boeuf@intel.com> 874* Wei Liu <liuwe@microsoft.com> 875* Yi Wang <wang.yi59@zte.com.cn> 876 877# v15.0 878 879This release has been tracked through the [v15.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/18). 880 881Highlights for `cloud-hypervisor` version v15.0 include: 882 883### Version numbering and stability guarantees 884 885This release is the first in a new version numbering scheme to represent that 886we believe Cloud Hypervisor is maturing and entering a period of stability. 887With this new release we are beginning our new stability guarantees: 888 889* The API (including command line options) will not be removed or changed in a 890 breaking way without a minimum of 2 releases notice. Where possible warnings 891 will be given about the use of deprecated functionality and the deprecations 892 will be documented in the release notes. 893* Point releases will be made between individual releases where there are 894 substantial bug fixes or security issues that need to be fixed. 895 896Currently the following items are **not** guaranteed across updates: 897 898* Snapshot/restore is not supported across different versions 899* Live migration is not supported across different versions 900* The following features are considered experimental and may change 901 substantially between releases: TDX, SGX. 902 903### Network device rate limiting 904 905Building on our existing support for rate limiting block activity the network 906device also now supports rate limiting. Full details of the controls are in the 907[IO throttling documentation.](docs/io_throttling.md) 908 909### Support for runtime control of `virtio-net` guest offload 910 911The guest is now able to change the offload settings for the `virtio-net` 912device. As well as providing a useful control this mitigates an issue in the 913Linux kernel where the guest will attempt to reprogram the offload settings 914even if they are not advertised as configurable (#2528). 915 916### `--api-socket` supports file descriptor parameter 917 918The `--api-socket` can now take an `fd=` parameter to specify an existing file 919descriptor to use. This is particularly beneficial for frameworks that need to 920programmatically control Cloud Hypervisor. 921 922### Bug fixes 923 924* A workaround has been put in place to mitigate a Linux kernel issues that 925 results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277). 926* PCI BARs are now correctly aligned removing the need for the guest to 927 reprogram them (#1797,#1798) 928* Handle TAP interface not being writable within virtio-net (due to the buffer 929 exhaustion on the host) (#2517) 930* The recommended Linux kernel is now v5.12.0 as it contains a fix that 931 prevents snapshot & restore working (#2535) 932 933### Deprecations 934 935Deprecated features will be removed in a subsequent release and users should plan to use alternatives 936 937* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 938 binaries has been deprecated. When using direct boot users should configure 939 their kernel with `CONFIG_PVH=y`. Will be removed in v16.0. 940 941### Contributors 942 943Many thanks to everyone who has contributed to our release including some new faces. 944 945* Alyssa Ross <hi@alyssa.is> 946* Anatol Belski <anbelski@linux.microsoft.com> 947* Bo Chen <chen.bo@intel.com> 948* Gaelan Steele <gbs@canishe.com> 949* Jianyong Wu <jianyong.wu@arm.com> 950* Michael Zhao <michael.zhao@arm.com> 951* Muminul Islam <muislam@microsoft.com> 952* Rob Bradford <robert.bradford@intel.com> 953* Sebastien Boeuf <sebastien.boeuf@intel.com> 954* Wei Liu <liuwe@microsoft.com> 955* William Douglas <william.douglas@intel.com> 956 957# v0.14.1 958 959Bug fix release branched off the v0.14.0 release. The following bugs were fixed 960in this release: 961 962* CPU hotplug on Windows failed due to misreported CPU state information and 963 the lack of HyperV CPUID bit enabled (#2437, #2449, #2436) 964* A seccomp rule was missing that was triggered on CPU unplug (#2455) 965* A bounds check in VIRTIO queue validation was erroneously generating 966 DescriptorChainTooShort errors in certain circumstances (#2450, #2424) 967 968# v0.14.0 969 970This release has been tracked through the [0.14.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/17). 971 972Highlights for `cloud-hypervisor` version 0.14.0 include: 973 974### Structured event monitoring 975 976A new option was added to the VMM `--event-monitor` which reports structured 977events (JSON) over a file or file descriptor at key events in the lifecycle of 978the VM. The list of events is limited at the moment but will be further 979extended over subsequent releases. The events exposed form part of the Cloud 980Hypervisor API surface. 981 982### MSHV improvements 983 984Basic support has been added for running Windows guests atop the MSHV 985hypervisor as an alternative to KVM and further improvements have been made to 986the MSHV support. 987 988### Improved aarch64 platform 989 990The aarch64 platform has been enhanced with more devices exposed to the running 991VM including an enhanced serial UART. 992 993### Updated hotplug documentation 994 995The documentation for the hotplug support has been updated to reflect the use 996of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as 997well as documenting hotplug of paravirtualised and VFIO devices. 998 999### PTY control for serial and `virtio-console` 1000 1001The `--serial` and `--console` parameters can now direct the console to a PTY 1002allowing programmatic control of the console from another process through the 1003PTY subsystem. 1004 1005### Block device rate limiting 1006 1007The block device performance can now be constrained as part of the VM 1008configuration allowing rate limiting. Full details of the controls are in the 1009[IO throttling documentation.](docs/io_throttling.md) 1010 1011 1012### Deprecations 1013 1014Deprecated features will be removed in a subsequent release and users should plan to use alternatives 1015 1016* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 1017 binaries has been deprecated. When using direct boot users should configure 1018 their kernel with `CONFIG_PVH=y`. 1019 1020 1021### Contributors 1022 1023Many thanks to everyone who has contributed to our 0.14.0 release including 1024some new faces. 1025 1026Bo Chen <chen.bo@intel.com> 1027Henry Wang <Henry.Wang@arm.com> 1028Iggy Jackson <iggy@theiggy.com> 1029Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 1030Michael Zhao <michael.zhao@arm.com> 1031Muminul Islam <muislam@microsoft.com> 1032Penny Zheng <Penny.Zheng@arm.com> 1033Rob Bradford <robert.bradford@intel.com> 1034Sebastien Boeuf <sebastien.boeuf@intel.com> 1035Vineeth Pillai <viremana@linux.microsoft.com> 1036Wei Liu <liuwe@microsoft.com> 1037William Douglas <william.r.douglas@gmail.com> 1038Zide Chen <zide.chen@intel.com> 1039 1040# v0.13.0 1041 1042This release has been tracked through the [0.13.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/16). 1043 1044Highlights for `cloud-hypervisor` version 0.13.0 include: 1045 1046### Wider VFIO device support 1047 1048It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI 1049devices that do not support MSI or MSI-X and instead rely on INTx interrupts. 1050Most notably this widens the support to most NVIDIA cards with the proprietary 1051drivers. 1052 1053### Improved huge page support 1054 1055Through the addition of `hugepage_size` on `--memory` it is now possible to 1056specify the desired size of the huge pages used when allocating the guest 1057memory. The user is required to ensure they have sufficient pages of the 1058desired size in their pool. 1059 1060### MACvTAP support 1061 1062It is now possible to provide file descriptors using the `fd` parameter to 1063`--net` which point at TAP devices that have already been opened by the user. 1064This aids integration with `libvirt` but also permits the use of MACvTAP 1065support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md). 1066 1067### VHD disk image support 1068 1069It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw 1070disk image with Cloud Hypervisor. 1071 1072### Improved Virtio device threading 1073 1074Device threads are now derived from the main VMM thread which allows more 1075restrictive seccomp filters to be applied to them. The threads also have a 1076predictable name derived from the device id. 1077 1078### Clean shutdown support via synthetic power button 1079 1080It is now possible to request that the guest VM shut itself down by triggering 1081a synthetic ACPI power button press from the VMM. If the guest is listening for 1082such an event (e.g. using systemd) then it will process the event and cleanly 1083shut down. This functionality is exposed through the HTTP API and can be 1084triggered via `ch-remote --api-socket=<API socket> power-button`. 1085 1086### Contributors 1087 1088Many thanks to everyone who has contributed to our 0.13.0 release including 1089some new faces. 1090 1091* Bo Chen <chen.bo@intel.com> 1092* Mikko Ylinen <mikko.ylinen@intel.com> 1093* Muminul Islam <muislam@microsoft.com> 1094* Rob Bradford <robert.bradford@intel.com> 1095* Samuel Ortiz <sameo@linux.intel.com> 1096* Sebastien Boeuf <sebastien.boeuf@intel.com> 1097* Vineeth Pillai <viremana@linux.microsoft.com> 1098* Wei Liu <liuwe@microsoft.com> 1099* William Douglas <william.r.douglas@gmail.com> 1100* Xie Yongji <xieyongji@bytedance.com> 1101 1102# v0.12.0 1103 1104This release has been tracked through the [0.12.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/15). 1105 1106Highlights for `cloud-hypervisor` version 0.12.0 include: 1107 1108### ARM64 enhancements 1109 1110The use of `--watchdog` is now fully supported as is the ability to reboot the 1111VM from within the guest when running Cloud Hypervisor on an ARM64 system. 1112 1113### Removal of `vhost-user-net` and `vhost-user-block` self spawning 1114 1115In order to use `vhost-user-net` or `vhost-user-block` backends the user is now 1116responsible for starting the backend and providing the socket for the VMM to 1117use. This functionality was deprecated in the last release and how now been 1118removed. 1119 1120### Migration of `vhost-user-fs` backend 1121 1122The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is 1123instead hosted in [it's own 1124repository](https://gitlab.com/virtio-fs/virtiofsd-rs) 1125 1126### Enhanced "info" API 1127 1128The `vm.info` HTTP API endpoint has been extended to include the details of the 1129devices used by the VM including any VFIO devices used. 1130 1131### Contributors 1132 1133Many thanks to everyone who has contributed to our 0.12.0 release: 1134 1135* Anatol Belski <anbelski@linux.microsoft.com> 1136* Julio Montes <julio.montes@intel.com> 1137* Michael Zhao <michael.zhao@arm.com> 1138* Muminul Islam <muislam@microsoft.com> 1139* Rob Bradford <robert.bradford@intel.com> 1140* Samuel Ortiz <sameo@linux.intel.com> 1141* Wei Liu <liuwe@microsoft.com> 1142 1143# v0.11.0 1144 1145This release has been tracked through the [0.11.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/14). 1146 1147Highlights for `cloud-hypervisor` version 0.11.0 include: 1148 1149### `io_uring` support by default for `virtio-block` 1150 1151Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will 1152be used for a significantly higher performance block device. 1153 1154### Windows Guest Support 1155 1156This is the first release where we officially support Windows running as a 1157guest. Full details of how to setup the image and run Cloud Hypervisor with a 1158Windows guest can be found in the dedicated [Windows 1159documentation](docs/windows.md). 1160 1161### `vhost-user` "Self Spawning" Deprecation 1162 1163Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now 1164deprecated. Users of this functionality will receive a warning and should make 1165adjustments. The functionality will be removed in the next release. 1166 1167### `virtio-mmio` Removal 1168 1169Support for using the `virtio-mmio` transport, rather than using PCI, has been 1170removed. This has been to simplify the code and significantly 1171reduce the testing burden of the project. 1172 1173### Snapshot/Restore support for ARM64 1174 1175When running on the ARM64 architecture snapshot and restore has now been 1176implemented. 1177 1178### Improved Linux Boot Time 1179 1180The time to boot the Linux kernel has been significantly improved by the 1181identifying some areas of delays around PCI bus probing, IOAPIC programming and 1182MPTABLE issues. Full details can be seen in #1728. 1183 1184### `SIGTERM/SIGINT` Interrupt Signal Handling 1185 1186When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will 1187trigger the VMM process to cleanly deallocate resources before exiting. The 1188guest VM will not be cleanly shutdown but the VMM process will clean up its 1189resources. 1190 1191### Default Log Level Changed 1192 1193The default logging level was changed to include warnings which should make it 1194easier to see potential issues. New [logging 1195documentation](docs/logging) was also added. 1196 1197### New `--balloon` Parameter Added 1198 1199Control of the setup of `virtio-balloon` has been moved from `--memory` to its 1200own dedicated parameter. This makes it easier to add more balloon specific 1201controls without overloading `--memory`. 1202 1203### Experimental `virtio-watchdog` Support 1204 1205Support for using a new `virtio-watchdog` has been added which can be used to 1206have the VMM reboot the guest if the guest userspace fails to ping the 1207watchdog. This is enabled with `--watchdog` and requires kernel support. 1208 1209### Notable Bug Fixes 1210 1211* MTRR bit was missing from CPUID advertised to guest 1212* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170) 1213* CPU identification string is now exposed to the guest 1214* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as 1215 read only so avoids excessive VM exits (#1795) 1216* PCI device hotplug after an unplug was fixed (#1802) 1217* When using the ACPI method to resize the guest memory the full reserved size 1218 can be used (#1803) 1219* Snapshot and restore followed by a second snapshot and restore now works 1220 correctly 1221* Snapshot and restore of VMs with more than 2GiB in one region now work 1222 correctly 1223 1224### Contributors 1225 1226Many thanks to everyone who has contributed to our 0.11.0 release including some new faces. 1227 1228* Anatol Belski <anbelski@linux.microsoft.com> 1229* Bo Chen <chen.bo@intel.com> 1230* Daniel Verkamp <dverkamp@chromium.org> 1231* Henry Wang <Henry.Wang@arm.com> 1232* Hui Zhu <teawater@antfin.com> 1233* Jiangbo Wu <jiangbo.wu@intel.com> 1234* Josh Soref <jsoref@users.noreply.github.com> 1235* Julio Montes <julio.montes@intel.com> 1236* Michael Zhao <michael.zhao@arm.com> 1237* Muminul Islam <muislam@microsoft.com> 1238* pierwill <19642016+pierwill@users.noreply.github.com> 1239* Praveen Paladugu <prapal@microsoft.com> 1240* Rob Bradford <robert.bradford@intel.com> 1241* Sebastien Boeuf <sebastien.boeuf@intel.com> 1242* Wei Liu <liuwe@microsoft.com> 1243 1244 1245# v0.10.0 1246 1247This release has been tracked through the [0.10.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/13). 1248 1249Highlights for `cloud-hypervisor` version 0.10.0 include: 1250 1251### `virtio-block` Support for Multiple Descriptors 1252 1253Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers. 1254 1255### Memory Zones 1256 1257Support has been added for fine grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md). 1258 1259### `Seccomp` Sandbox Improvements 1260 1261All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`. 1262 1263### Preliminary KVM HyperV Emulation Control 1264 1265A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices. 1266 1267### Notable Bug Fixes 1268 1269- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596) 1270- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564) 1271- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593) 1272- `--version` shows the version for released binaries (#1669) 1273- Errors generated by worker threads for `virtio` devices are now printed out (#1551) 1274 1275### Contributors 1276 1277Many thanks to everyone who has contributed to our 0.10.0 release including some new faces. 1278 1279* Alyssa Ross <hi@alyssa.is> 1280* Amey Narkhede <ameynarkhede02@gmail.com> 1281* Anatol Belski <ab@php.net> 1282* Bo Chen <chen.bo@intel.com> 1283* Hui Zhu <teawater@antfin.com> 1284* Michael Zhao <michael.zhao@arm.com> 1285* Muminul Islam <muislam@microsoft.com> 1286* Rob Bradford <robert.bradford@intel.com> 1287* Samuel Ortiz <sameo@linux.intel.com> 1288* Sebastien Boeuf <sebastien.boeuf@intel.com> 1289* Wei Liu <liuwe@microsoft.com> 1290 1291# v0.9.0 1292 1293This release has been tracked through the [0.9.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/12). 1294 1295Highlights for `cloud-hypervisor` version 0.9.0 include: 1296 1297### `io_uring` Based Block Device Support 1298 1299If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement. 1300 1301### Block and Network Device Statistics 1302 1303Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM. 1304 1305### HTTP API Responses 1306 1307The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF. 1308 1309### CPU Topology 1310 1311A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core. 1312 1313### Release Build Optimization 1314 1315Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size. 1316 1317### Hypervisor Abstraction 1318 1319A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`. 1320 1321### Snapshot/Restore Improvements 1322 1323Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs. 1324 1325### Virtio Memory Ballooning Support 1326 1327A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest. 1328 1329### Enhancements to ARM64 Support 1330 1331The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting. 1332 1333### Intel SGX Support 1334 1335The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md). 1336 1337### `Seccomp` Sandbox Improvements 1338 1339The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development. 1340 1341### Notable Bug Fixes 1342 1343* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes. 1344* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338) 1345* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543). 1346* `virtio-mmio` based devices are now more widely tested (#275). 1347* Multiple issues have been fixed with virtio device configuration (#1217) 1348* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521) 1349 1350### Contributors 1351 1352Many thanks to everyone who has contributed to our 0.9.0 release including some new faces. 1353 1354* Anatol Belski <ab@php.net> 1355* Bo Chen <chen.bo@intel.com> 1356* Dr. David Alan Gilbert <dgilbert@redhat.com> 1357* Henry Wang <Henry.Wang@arm.com> 1358* Howard Zhang <howard.zhang@arm.com> 1359* Hui Zhu <teawater@antfin.com> 1360* Jianyong Wu <jianyong.wu@arm.com> 1361* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1362* LiYa'nan <oliverliyn@gmail.com> 1363* Michael Zhao <michael.zhao@arm.com> 1364* Muminul Islam <muislam@microsoft.com> 1365* Praveen Paladugu <prapal@microsoft.com> 1366* Ricardo Koller <ricarkol@gmail.com> 1367* Rob Bradford <robert.bradford@intel.com> 1368* Samuel Ortiz <sameo@linux.intel.com> 1369* Sebastien Boeuf <sebastien.boeuf@intel.com> 1370* Stefano Garzarella <sgarzare@redhat.com> 1371* Wei Liu <liuwe@microsoft.com> 1372 1373 1374# v0.8.0 1375 1376This release has been tracked through the [0.8.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/10). 1377 1378Highlights for `cloud-hypervisor` version 0.8.0 include: 1379 1380### Experimental Snapshot and Restore Support 1381 1382This release includes the first version of the snapshot and restore feature. 1383This allows a VM to be paused and then subsequently snapshotted. At a later 1384point that snapshot may be restored into a new running VM identical to the 1385original VM at the point it was paused. 1386 1387This feature can be used for offline migration from one VM host to another, to 1388allow the upgrading or rebooting of the host machine transparently to the guest 1389or for templating the VM. This is an experimental feature and cannot be used on 1390a VM using passthrough (VFIO) devices. Issues with SMP have also been observed 1391(#1176). 1392 1393### Experimental ARM64 Support 1394 1395Included in this release is experimental support for running on ARM64. 1396Currently only `virtio-mmio` devices and a serial port are supported. Full 1397details can be found in the [ARM64 documentation](docs/arm64.md). 1398 1399### Support for Using 5-level Paging in Guests 1400 1401If the host supports it the guest is now enabled for 5-level paging (aka LA57). 1402This works when booting the Linux kernel with a vmlinux, bzImage or firmware 1403based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not 1404work due to current limitations in the PVH boot process. 1405 1406### Virtio Device Interrupt Suppression for Network Devices 1407 1408With `virtio-net` and `vhost-user-net` devices the guest can suppress 1409interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This 1410can lead to an improvement in performance by reducing the number of interrupts 1411the guest must service. 1412 1413### `vhost_user_fs` Improvements 1414 1415The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`. 1416 1417 1418### Notable Bug Fixes 1419 1420* VMs that have not yet been booted can now be deleted (#1110). 1421* By creating the `tap` device ahead of creating the VM it is not required to 1422 run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273). 1423* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to 1424 the specification and synchronizes to the underlying filesystem as required 1425 based on guest feature negotiation. This avoids potential data loss (#399, 1426 #1216). 1427* When booting with a large number of vCPUs then the ACPI table would be 1428 overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the 1429 `MPTABLE` will no longer be generated (#1132). 1430* Shutting down VMs that have been paused is now supported (#816). 1431* Created socket files are deleted on shutdown (#1083). 1432* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds 1433 (#751). 1434 1435### Command Line and API Changes 1436 1437This is non exhaustive list of HTTP API and command line changes: 1438 1439* All user visible socket parameters are now consistently called `socket` 1440 rather than `sock` in some cases. 1441* The `ch-remote` tool now shows any error message generated by the VMM 1442* The `wce` parameter has been removed from `--disk` as the feature is always 1443 offered for negotiation. 1444* `--net` has gained a `host_mac` option that allows the setting of the MAC 1445 address for the `tap` device on the host. 1446 1447### Contributors 1448 1449Many thanks to everyone who has contributed to our 0.8.0 release including some new faces. 1450 1451* Anatol Belski <ab@php.net> 1452* Arron Wang <arron.wang@intel.com> 1453* Bo Chen <chen.bo@intel.com> 1454* Dr. David Alan Gilbert <dgilbert@redhat.com> 1455* Henry Wang <Henry.Wang@arm.com> 1456* Hui Zhu <teawater@antfin.com> 1457* LiYa'nan <oliverliyn@gmail.com> 1458* Michael Zhao <michael.zhao@arm.com> 1459* Rob Bradford <robert.bradford@intel.com> 1460* Samuel Ortiz <sameo@linux.intel.com> 1461* Sebastien Boeuf <sebastien.boeuf@intel.com> 1462* Sergio Lopez <slp@redhat.com> 1463 1464# v0.7.0 1465 1466This release has been tracked through the [0.7.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 1467 1468Highlights for `cloud-hypervisor` version 0.7.0 include: 1469 1470### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug 1471 1472Further to our effort to support modifying a running guest we now support 1473hotplug and unplug of the following virtio backed devices: block, network, 1474pmem, virtio-fs and vsock. This functionality is available on the (default) PCI 1475based transport and is exposed through the HTTP API. The `ch-remote` utility 1476provides a CLI for adding or removing these device types after the VM has 1477booted. User can use the `id` parameter on the devices to choose names for 1478devices to ease their removal. 1479 1480### Alternative `libc` Support 1481 1482Cloud Hypervisor can now be compiled with the `musl` C library and this release 1483contains a static binary compiled using that toolchain. 1484 1485### Multithreaded Multi Queued `vhost-user` Backends 1486 1487The `vhost-user` backends for network and block support that are shipped by 1488Cloud Hypervisor have been enhanced to support multiple threads and queues to 1489improve throughput. These backends are used automatically if `vhost_user=true` 1490is passed when the devices are created. 1491 1492### Initial RamFS Support 1493 1494By passing the `--initramfs` command line option the user can specify a file to 1495be loaded into the guest memory to be used as the kernel initial filesystem. 1496This is usually used to allow the loading of drivers needed to be able to 1497access the real root filesystem but it can also be used standalone for a very 1498minimal image. 1499 1500### Alternative Memory Hotplug: `virtio-mem` 1501 1502As well as supporting ACPI based hotplug Cloud Hypervisor now supports using 1503the `virtio-mem` hotplug alternative. This can be controlled by the 1504`hotplug_method` parameter on the `--memory` command line option. It currently 1505requires kernel patches to be able to support it. 1506 1507### `Seccomp` Sandboxing 1508 1509Cloud Hypervisor now has support for restricting the system calls that the 1510process can use via the `seccomp` security API. This on by default and is 1511controlled by the `--seccomp` command line option. 1512 1513### Updated Distribution Support 1514 1515With the release of Ubuntu 20.04 we have added that to the list of supported 1516distributions and is part of our regular testing programme. 1517 1518### Command Line and API Changes 1519 1520This is non exhaustive list of HTTP API and command line changes 1521 1522* New `id` fields added for devices to allow them to be named to ease removal. 1523 If no name is specified the VMM chooses one. 1524* Use `--memory`'s `shared` and `hugepages` controls for determining backing 1525 memory instead of providing a path. 1526* The `--vsock` parameter only takes one device as the Linux kernel only 1527 supports a single Vsock device. The REST API has removed the vector for this 1528 option and replaced it with a single optional field. 1529* There is enhanced validation of the command line and API provided 1530 configurations to ensure that the provided options are compatible e.g. that 1531 shared memory is in use if any attempt is made to used a `vhost-user` backed 1532 device. 1533* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and 1534 `add-vsock` subcommands. For removal `remove-device` is used. The REST API 1535 has appropriate new HTTP endpoints too. 1536* Specifying a `size` with `--pmem` is no longer required and instead the size 1537 will be obtained from the file. A `discard_writes` option has also been added 1538 to provide the equivalent of a read-only file. 1539* The parameters to `--block-backend` have been changed to more closely align 1540 with those used by `--disk`. 1541 1542### Contributors 1543 1544Many thanks to everyone who has contributed to our 0.7.0 release including some new faces. 1545 1546* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 1547* Bo Chen <chen.bo@intel.com> 1548* Cathy Zhang <cathy.zhang@intel.com> 1549* Damjan Georgievski <gdamjan@gmail.com> 1550* Dean Sheather <dean@coder.com> 1551* Eryu Guan <eguan@linux.alibaba.com> 1552* Hui Zhu <teawater@antfin.com> 1553* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1554* Martin Xu <martin.xu@intel.com> 1555* Muminul Islam <muislam@microsoft.com> 1556* Rob Bradford <robert.bradford@intel.com> 1557* Samuel Ortiz <sameo@linux.intel.com> 1558* Sebastien Boeuf <sebastien.boeuf@intel.com> 1559* Sergio Lopez <slp@redhat.com> 1560* Yang Zhong <yang.zhong@intel.com> 1561* Yi Sun <yi.y.sun@linux.intel.com> 1562 1563# v0.6.0 1564 1565This release has been tracked through the [0.6.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 1566 1567Highlights for `cloud-hypervisor` version 0.6.0 include: 1568 1569### Directly Assigned Devices Hotplug 1570 1571We continued our efforts around supporting dynamically changing the guest 1572resources. After adding support for CPU and memory hotplug, Cloud Hypervisor 1573now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`) 1574devices into an already running guest. This closes the features gap for 1575providing a complete Kata Containers workloads support with Cloud Hypervisor. 1576 1577### Shared Filesystem Improvements 1578 1579We enhanced our shared filesystem support through many `virtio-fs` improvements. 1580By adding support for DAX, parallel processing of multiple requests, `FS_IO`, 1581`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we 1582improved our filesystem sharing performance, but also made it more stable and 1583compatible with other `virtio-fs` implementations. 1584 1585### Block and Networking IO Self Offloading 1586 1587When choosing to offload the paravirtualized block and networking I/O to an 1588external process (through the `vhost-user` protocol), Cloud Hypervisor now 1589automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends 1590into their own, separate processes. 1591This provides a seamless paravirtualized I/O user experience for those who want 1592to run their guest I/O into separate executions contexts. 1593 1594### Command Line Interface 1595 1596More and more Cloud Hypervisor services are exposed through the 1597[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only 1598accessible via relatively cumbersome HTTP calls. In order to abstract 1599those calls into a more user friendly tool, we created a Cloud Hypervisor 1600Command Line Interface (CLI) called `ch-remote`. The `ch-remote` binary 1601is created with each build and available e.g. at 1602`cloud-hypervisor/target/debug/ch-remote` when doing a debug build. 1603 1604Please check `ch-remote --help` for a complete description of all available 1605commands. 1606 1607### PVH Boot 1608 1609In addition to the traditional Linux boot protocol, Cloud Hypervisor now 1610supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html). 1611 1612### Contributors 1613 1614With the 0.6.0 release, we are welcoming a few new contributors. Many thanks 1615to them and to everyone that contributed to this release: 1616 1617* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 1618* Arron Wang <arron.wang@intel.com> 1619* Bin Liu <liubin0329@gmail.com> 1620* Bo Chen <chen.bo@intel.com> 1621* Cathy Zhang <cathy.zhang@intel.com> 1622* Eryu Guan <eguan@linux.alibaba.com> 1623* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1624* Liu Bo <bo.liu@linux.alibaba.com> 1625* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1626* Rob Bradford <robert.bradford@intel.com> 1627* Samuel Ortiz <sameo@linux.intel.com> 1628* Sebastien Boeuf <sebastien.boeuf@intel.com> 1629* Sergio Lopez <slp@redhat.com> 1630 1631# v0.5.1 1632 1633This is a bugfix release branched off v0.5.0. It contains the following fixes: 1634 1635* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez 1636* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf 1637* Fixed error reporting for resize command - Samuel Ortiz 1638* Double reboot workaround (#783) - Rob Bradford 1639* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford 1640 1641# v0.5.0 1642 1643This release has been tracked through the [0.5.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/6). 1644 1645Highlights for `cloud-hypervisor` version 0.5.0 include: 1646 1647### Virtual Machine Dynamic Resizing 1648 1649With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and 1650memory hot plug as well. This allows to dynamically resize Cloud Hypervisor 1651guests which is needed for e.g. Kubernetes related use cases. 1652The memory hot plug implementation is based on the same framework as the CPU hot 1653plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest. 1654 1655Next on our VM resizing roadmap is the PCI devices hotplug feature. 1656 1657### Multi-Queue, Multi-Threaded Paravirtualization 1658 1659We enhanced our virtio networking and block support by having both devices use 1660multiple I/O queues handled by multiple threads. This improves our default 1661paravirtualized networking and block devices throughput. 1662 1663### New Interrupt Management Framework 1664 1665We improved our interrupt management implementation by introducing an Interrupt 1666Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device) 1667crates discussions. This move made the code significantly cleaner, and allowed 1668us to remove several KVM related dependencies from crates like the PCI and 1669virtio ones. 1670 1671### Development Tools 1672 1673In order to provide a better developer experience, we worked on improving our 1674build, development and testing tools. 1675Somehow similar to the excellent 1676[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool), 1677we now provide a [dev_cli script](scripts/dev_cli.sh). 1678 1679With this new tool, our users and contributors will be able to build and test 1680Cloud Hypervisor through a containerized environment. 1681 1682### Kata Containers Integration 1683 1684We spent some significant time and efforts debugging and fixing our integration 1685with the [Kata Containers](https://github.com/kata-containers) project. Cloud 1686Hypervisor is now a fully supported Kata Containers hypervisor, and is 1687integrated into the project's CI. 1688 1689### Contributors 1690 1691Many thanks to everyone that contributed to the 0.5.0 release: 1692 1693* Bo Chen <chen.bo@intel.com> 1694* Cathy Zhang <cathy.zhang@intel.com> 1695* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1696* Rob Bradford <robert.bradford@intel.com> 1697* Samuel Ortiz <sameo@linux.intel.com> 1698* Sebastien Boeuf <sebastien.boeuf@intel.com> 1699* Sergio Lopez <slp@redhat.com> 1700* Yang Zhong <yang.zhong@intel.com> 1701 1702# v0.4.0 1703 1704This release has been tracked through the [0.4.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/4). 1705 1706Highlights for `cloud-hypervisor` version 0.4.0 include: 1707 1708### Dynamic virtual CPUs addition 1709 1710As a way to vertically scale Cloud Hypervisor guests, we now support dynamically 1711adding virtual CPUs to the guests, a mechanism also known as CPU hot plug. 1712Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs 1713to an already running guest and the high level operations for that process are 1714documented [here](docs/hotplug.md) 1715 1716During the next release cycles we are planning to extend Cloud Hypervisor 1717hot plug framework to other resources, namely PCI devices and memory. 1718 1719### Programmatic firmware tables generation 1720 1721As part of the CPU hot plug feature enablement, and as a requirement for hot 1722plugging other resources like devices or RAM, we added support for 1723programmatically generating the needed ACPI tables. Through a dedicated 1724`acpi-tables` crate, we now have a flexible and clean way of generating those 1725tables based on the VMM device model and topology. 1726 1727### Filesystem and block devices vhost-user backends 1728 1729Our objective of running all Cloud Hypervisor paravirtualized I/O to a 1730vhost-user based framework is getting closer as we've added Rust based 1731implementations for vhost-user-blk and virtiofs backends. Together with the 1732vhost-user-net backend that came with the 0.3.0 release, this will form the 1733default Cloud Hypervisor I/O architecture. 1734 1735### Guest pause and resume 1736 1737As an initial requirement for enabling live migration, we added support for 1738pausing and resuming any VMM components. As an intermediate step towards live 1739migration, the upcoming guest snapshotting feature will be based on the pause 1740and resume capabilities. 1741 1742### Userspace IOAPIC by default 1743 1744As a way to simplify our device manager implementation, but also in order to 1745stay away from privileged rings as often as possible, any device that relies on 1746pin based interrupts will be using the userspace IOAPIC implementation by 1747default. 1748 1749### PCI BAR reprogramming 1750 1751In order to allow for a more flexible device model, and also support guests 1752that would want to move PCI devices, we added support for PCI devices BAR 1753reprogramming. 1754 1755### New `cloud-hypervisor` organization 1756 1757As we wanted to be more flexible on how we manage the Cloud Hypervisor project, 1758we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor). 1759Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) 1760project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux) 1761and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1762repositories. We may also use it to host any rust-vmm that we'd need to 1763temporarily fork. 1764Thanks to GitHub's seamless repository redirections, the move is completely 1765transparent to all Cloud Hypervisor contributors, users and followers. 1766 1767### Contributors 1768 1769Many thanks to everyone that contributed to the 0.4.0 release: 1770 1771* Cathy Zhang <cathy.zhang@intel.com> 1772* Emin Ghuliev <drmint80@gmail.com> 1773* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1774* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1775* Rob Bradford <robert.bradford@intel.com> 1776* Samuel Ortiz <sameo@linux.intel.com> 1777* Sebastien Boeuf <sebastien.boeuf@intel.com> 1778* Sergio Lopez <slp@redhat.com> 1779* Wu Zongyong <wuzongyong@linux.alibaba.com> 1780 1781# v0.3.0 1782 1783This release has been tracked through the [0.3.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/3). 1784 1785Highlights for `cloud-hypervisor` version 0.3.0 include: 1786 1787### Block device offloading 1788 1789We continue to work on offloading paravirtualized I/O to external processes, 1790and we added support for 1791[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends. 1792This enables `cloud-hypervisor` users to plug a `vhost-user` based block device 1793like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage 1794backend. 1795 1796### Network device backend 1797 1798The previous release provided support for 1799[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we 1800also provide a TAP based vhost-user-net backend, implemented in Rust. Together 1801with the vhost-user-net device implementation, this will eventually become the 1802Cloud Hypervisor default paravirtualized networking architecture. 1803 1804### Virtual sockets 1805 1806In order to more efficiently and securely communicate between host and guest, 1807we added an hybrid implementation of the 1808[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address 1809family over virtio. Credits go to the 1810[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md) 1811project as our implementation is a copy of theirs. 1812 1813### HTTP based API 1814 1815In anticipation of the need to support asynchronous operations to Cloud 1816Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP 1817based API to the VMM. The API will be more extensively documented during the 1818next release cycle. 1819 1820### Memory mapped virtio transport 1821 1822In order to support potential PCI-free use cases, we added support for the 1823[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002) 1824transport layer. This will allow us to support simple, minimal guest 1825configurations that do not require a PCI bus emulation. 1826 1827### Paravirtualized IOMMU 1828 1829As we want to improve our nested guests support, we added support for exposing 1830a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows 1831for a safer nested virtio and directly assigned devices support. 1832 1833To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor 1834users to be able to specify if devices had to be handled through this virtual 1835IOMMU or not. In particular, the `--disk` option now expects disk paths to be 1836prefixed with a `path=` string, and supports an optional `iommu=[on|off]` 1837setting. 1838 1839### Ubuntu 19.10 1840 1841With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware), 1842we can now support the latest 1843[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images. 1844 1845### Large memory guests 1846 1847After simplifying and changing our guest address space handling, we can now 1848support guests with large amount of memory (more than 64GB). 1849 1850# v0.2.0 1851 1852This release has been tracked through the [0.2.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/2). 1853 1854Highlights for `cloud-hypervisor` version 0.2.0 include: 1855 1856### Network device offloading 1857 1858As part of our general effort to offload paravirtualized I/O to external 1859processes, we added support for 1860[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This 1861enables `cloud-hypervisor` users to plug a `vhost-user` based networking device 1862(e.g. [DPDK](https://dpdk.org)) into the VMM as their virtio network backend. 1863 1864### Minimal hardware-reduced ACPI 1865 1866In order to properly implement and guest reset and shutdown, we implemented 1867a minimal version of the hardware-reduced ACPI specification. Together with 1868a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to 1869cleanly reboot and shutdown. 1870 1871The ACPI implementation is a `cloud-hypervisor` build time option that is 1872enabled by default. 1873 1874### Debug I/O port 1875 1876Based on the Firecracker idea of using a dedicated I/O port to measure guest 1877boot times, we added support for logging guest events through the 1878[0x80](https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html) 1879PC debug port. This allows, among other things, for granular guest boot time 1880measurements. See our [debug port documentation](docs/debug-port.md) for more 1881details. 1882 1883### Improved direct device assignment 1884 1885We fixed a major performance issue with our initial VFIO implementation: When 1886enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads 1887were (in many cases) not cached. After correctly tagging the guest memory from 1888`cloud-hypervisor` we're now able to reach the expected performance from 1889directly assigned devices. 1890 1891### Improved shared filesystem 1892 1893We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt) 1894support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system. 1895This provides better shared filesystem IO performance with a smaller guest 1896memory footprint. 1897 1898### Ubuntu bionic based CI 1899 1900Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1901improvements, we are now able to boot Ubuntu bionic images. We added those to 1902our CI pipeline. 1903 1904# v0.1.0 1905 1906This release has been tracked through the [0.1.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/1). 1907 1908Highlights for `cloud-hypervisor` version 0.1.0 include: 1909 1910### Shared filesystem 1911 1912We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file 1913system, allowing for an efficient and reliable way of sharing a filesystem 1914between the host and the `cloud-hypervisor` guest. 1915 1916See our [filesystem sharing](docs/fs.md) documentation for more details on how 1917to use virtio-fs with `cloud-hypervisor`. 1918 1919### Initial direct device assignment support 1920 1921VFIO (Virtual Function I/O) is a kernel framework that exposes direct device 1922access to userspace. `cloud-hypervisor` uses VFIO to directly assign host 1923physical devices into its guest. 1924 1925See our [VFIO](docs/vfio.md) documentation for more detail on how to directly 1926assign host devices to `cloud-hypervisor` guests. 1927 1928### Userspace IOAPIC 1929 1930`cloud-hypervisor` supports a so-called split IRQ chip implementation by 1931implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC). 1932By moving part of the IRQ chip implementation from kernel space to user space, 1933the IRQ chip emulation does not always run in a fully privileged mode. 1934 1935### Virtual persistent memory 1936 1937The `virtio-pmem` implementation emulates a virtual persistent memory device 1938that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device 1939allows to bypass the guest page cache and improve the guest memory footprint. 1940 1941### Linux kernel bzImage 1942 1943The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from 1944`bzImage` kernel images, which is usually the format that Linux distributions 1945use to ship their kernels. For example, this allows for booting from the host 1946distribution kernel image. 1947 1948### Console over virtio 1949 1950`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although 1951using this device as a guest console can potentially cut some early boot 1952messages, it can reduce the guest boot time and provides a complete console 1953implementation. 1954 1955The `virtio-console` device is enabled by default for the guest console. 1956Switching back to the legacy serial port is done by selecting 1957`--serial tty --console off` from the command line. 1958 1959### Unit testing 1960 1961We now run all unit tests from all our crates directly from our CI. 1962 1963### Integration tests parallelization 1964 1965The CI cycle run time has been significantly reduced by refactoring our 1966integration tests; allowing them to all be run in parallel. 1967