1- [v26.0](#v260) 2 - [SMBIOS Improvements via `--platform`](#smbios-improvements-via---platform) 3 - [Unified Binary MSHV and KVM Support](#unified-binary-mshv-and-kvm-support) 4 - [Notable Bug Fixes](#notable-bug-fixes) 5 - [Deprecations](#deprecations) 6 - [Removals](#removals) 7 - [Contributors](#contributors) 8- [v25.0](#v250) 9 - [`ch-remote` Improvements](#ch-remote-improvements) 10 - [VM "Coredump" Support](#vm-coredump-support) 11 - [Notable Bug Fixes](#notable-bug-fixes-1) 12 - [Removals](#removals-1) 13 - [Contributors](#contributors-1) 14- [v24.0](#v240) 15 - [Bypass Mode for `virtio-iommu`](#bypass-mode-for-virtio-iommu) 16 - [Ensure Identifiers Uniqueness](#ensure-identifiers-uniqueness) 17 - [Sparse Mmap support](#sparse-mmap-support) 18 - [Expose Platform Serial Number](#expose-platform-serial-number) 19 - [Notable Bug Fixes](#notable-bug-fixes-2) 20 - [Notable Improvements](#notable-improvements) 21 - [Deprecations](#deprecations-1) 22 - [New on the Website](#new-on-the-website) 23 - [Contributors](#contributors-2) 24- [v23.1](#v231) 25- [v23.0](#v230) 26 - [vDPA Support](#vdpa-support) 27 - [Updated OS Support list](#updated-os-support-list) 28 - [`AArch64` Memory Map Improvements](#aarch64-memory-map-improvements) 29 - [`AMX` Support](#amx-support) 30 - [Notable Bug Fixes](#notable-bug-fixes-3) 31 - [Deprecations](#deprecations-2) 32 - [Contributors](#contributors-3) 33- [v22.1](#v221) 34- [v22.0](#v220) 35 - [GDB Debug Stub Support](#gdb-debug-stub-support) 36 - [`virtio-iommu` Backed Segments](#virtio-iommu-backed-segments) 37 - [Before Boot Configuration Changes](#before-boot-configuration-changes) 38 - [`virtio-balloon` Free Page Reporting](#virtio-balloon-free-page-reporting) 39 - [Support for Direct Kernel Booting with TDX](#support-for-direct-kernel-booting-with-tdx) 40 - [PMU Support for AArch64](#pmu-support-for-aarch64) 41 - [Documentation Under CC-BY-4.0 License](#documentation-under-cc-by-40-license) 42 - [Deprecation of "Classic" `virtiofsd`](#deprecation-of-classic-virtiofsd) 43 - [Notable Bug Fixes](#notable-bug-fixes-4) 44 - [Contributors](#contributors-4) 45- [v21.0](#v210) 46 - [Efficient Local Live Migration (for Live Upgrade)](#efficient-local-live-migration-for-live-upgrade) 47 - [Recommended Kernel is Now 5.15](#recommended-kernel-is-now-515) 48 - [Notable Bug fixes](#notable-bug-fixes-5) 49 - [Contributors](#contributors-5) 50- [v20.2](#v202) 51- [v20.1](#v201) 52- [v20.0](#v200) 53 - [Multiple PCI segments support](#multiple-pci-segments-support) 54 - [CPU pinning](#cpu-pinning) 55 - [Improved VFIO support](#improved-vfio-support) 56 - [Safer code](#safer-code) 57 - [Extended documentation](#extended-documentation) 58 - [Notable bug fixes](#notable-bug-fixes-6) 59 - [Contributors](#contributors-6) 60- [v19.0](#v190) 61 - [Improved PTY handling for serial and `virtio-console`](#improved-pty-handling-for-serial-and-virtio-console) 62 - [PCI boot time optimisations](#pci-boot-time-optimisations) 63 - [Improved TDX support](#improved-tdx-support) 64 - [Live migration enhancements](#live-migration-enhancements) 65 - [`virtio-mem` support with `vfio-user`](#virtio-mem-support-with-vfio-user) 66 - [AArch64 for `virtio-iommu`](#aarch64-for-virtio-iommu) 67 - [Notable bug fixes](#notable-bug-fixes-7) 68 - [Contributors](#contributors-7) 69- [v18.0](#v180) 70 - [Experimental User Device (`vfio-user`) support](#experimental-user-device-vfio-user-support) 71 - [Migration support for `vhost-user` devices](#migration-support-for-vhost-user-devices) 72 - [VHDX disk image support](#vhdx-disk-image-support) 73 - [Device pass through on MSHV hypervisor](#device-pass-through-on-mshv-hypervisor) 74 - [AArch64 for support `virtio-mem`](#aarch64-for-support-virtio-mem) 75 - [Live migration on MSHV hypervisor](#live-migration-on-mshv-hypervisor) 76 - [AArch64 CPU topology support](#aarch64-cpu-topology-support) 77 - [Power button support on AArch64](#power-button-support-on-aarch64) 78 - [Notable bug fixes](#notable-bug-fixes-8) 79 - [Contributors](#contributors-8) 80- [v17.0](#v170) 81 - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi) 82 - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend) 83 - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices) 84 - [Improved SGX support](#improved-sgx-support) 85 - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices) 86 - [Notable bug fixes](#notable-bug-fixes-9) 87 - [Contributors](#contributors-9) 88- [v16.0](#v160) 89 - [Improved live migration support](#improved-live-migration-support) 90 - [Improved `vhost-user` support](#improved-vhost-user-support) 91 - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support) 92 - [Notable bug fixes](#notable-bug-fixes-10) 93 - [Removed functionality](#removed-functionality) 94 - [Contributors](#contributors-10) 95- [v15.0](#v150) 96 - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees) 97 - [Network device rate limiting](#network-device-rate-limiting) 98 - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload) 99 - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter) 100 - [Bug fixes](#bug-fixes) 101 - [Deprecations](#deprecations-3) 102 - [Contributors](#contributors-11) 103- [v0.14.1](#v0141) 104- [v0.14.0](#v0140) 105 - [Structured event monitoring](#structured-event-monitoring) 106 - [MSHV improvements](#mshv-improvements) 107 - [Improved aarch64 platform](#improved-aarch64-platform) 108 - [Updated hotplug documentation](#updated-hotplug-documentation) 109 - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console) 110 - [Block device rate limiting](#block-device-rate-limiting) 111 - [Deprecations](#deprecations-4) 112 - [Contributors](#contributors-12) 113- [v0.13.0](#v0130) 114 - [Wider VFIO device support](#wider-vfio-device-support) 115 - [Improved huge page support](#improved-huge-page-support) 116 - [MACvTAP support](#macvtap-support) 117 - [VHD disk image support](#vhd-disk-image-support) 118 - [Improved Virtio device threading](#improved-virtio-device-threading) 119 - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button) 120 - [Contributors](#contributors-13) 121- [v0.12.0](#v0120) 122 - [ARM64 enhancements](#arm64-enhancements) 123 - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning) 124 - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend) 125 - [Enhanced "info" API](#enhanced-info-api) 126 - [Contributors](#contributors-14) 127- [v0.11.0](#v0110) 128 - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block) 129 - [Windows Guest Support](#windows-guest-support) 130 - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation) 131 - [`virtio-mmio` Removal](#virtio-mmio-removal) 132 - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64) 133 - [Improved Linux Boot Time](#improved-linux-boot-time) 134 - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling) 135 - [Default Log Level Changed](#default-log-level-changed) 136 - [New `--balloon` Parameter Added](#new---balloon-parameter-added) 137 - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support) 138 - [Notable Bug Fixes](#notable-bug-fixes-11) 139 - [Contributors](#contributors-15) 140- [v0.10.0](#v0100) 141 - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors) 142 - [Memory Zones](#memory-zones) 143 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements) 144 - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control) 145 - [Notable Bug Fixes](#notable-bug-fixes-12) 146 - [Contributors](#contributors-16) 147- [v0.9.0](#v090) 148 - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support) 149 - [Block and Network Device Statistics](#block-and-network-device-statistics) 150 - [HTTP API Responses](#http-api-responses) 151 - [CPU Topology](#cpu-topology) 152 - [Release Build Optimization](#release-build-optimization) 153 - [Hypervisor Abstraction](#hypervisor-abstraction) 154 - [Snapshot/Restore Improvements](#snapshotrestore-improvements) 155 - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support) 156 - [Enhancements to ARM64 Support](#enhancements-to-arm64-support) 157 - [Intel SGX Support](#intel-sgx-support) 158 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1) 159 - [Notable Bug Fixes](#notable-bug-fixes-13) 160 - [Contributors](#contributors-17) 161- [v0.8.0](#v080) 162 - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support) 163 - [Experimental ARM64 Support](#experimental-arm64-support) 164 - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests) 165 - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices) 166 - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements) 167 - [Notable Bug Fixes](#notable-bug-fixes-14) 168 - [Command Line and API Changes](#command-line-and-api-changes) 169 - [Contributors](#contributors-18) 170- [v0.7.0](#v070) 171 - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug) 172 - [Alternative `libc` Support](#alternative-libc-support) 173 - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends) 174 - [Initial RamFS Support](#initial-ramfs-support) 175 - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem) 176 - [`Seccomp` Sandboxing](#seccomp-sandboxing) 177 - [Updated Distribution Support](#updated-distribution-support) 178 - [Command Line and API Changes](#command-line-and-api-changes-1) 179 - [Contributors](#contributors-19) 180- [v0.6.0](#v060) 181 - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug) 182 - [Shared Filesystem Improvements](#shared-filesystem-improvements) 183 - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading) 184 - [Command Line Interface](#command-line-interface) 185 - [PVH Boot](#pvh-boot) 186 - [Contributors](#contributors-20) 187- [v0.5.1](#v051) 188- [v0.5.0](#v050) 189 - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing) 190 - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization) 191 - [New Interrupt Management Framework](#new-interrupt-management-framework) 192 - [Development Tools](#development-tools) 193 - [Kata Containers Integration](#kata-containers-integration) 194 - [Contributors](#contributors-21) 195- [v0.4.0](#v040) 196 - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition) 197 - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation) 198 - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends) 199 - [Guest pause and resume](#guest-pause-and-resume) 200 - [Userspace IOAPIC by default](#userspace-ioapic-by-default) 201 - [PCI BAR reprogramming](#pci-bar-reprogramming) 202 - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization) 203 - [Contributors](#contributors-22) 204- [v0.3.0](#v030) 205 - [Block device offloading](#block-device-offloading) 206 - [Network device backend](#network-device-backend) 207 - [Virtual sockets](#virtual-sockets) 208 - [HTTP based API](#http-based-api) 209 - [Memory mapped virtio transport](#memory-mapped-virtio-transport) 210 - [Paravirtualized IOMMU](#paravirtualized-iommu) 211 - [Ubuntu 19.10](#ubuntu-1910) 212 - [Large memory guests](#large-memory-guests) 213- [v0.2.0](#v020) 214 - [Network device offloading](#network-device-offloading) 215 - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi) 216 - [Debug I/O port](#debug-io-port) 217 - [Improved direct device assignment](#improved-direct-device-assignment) 218 - [Improved shared filesystem](#improved-shared-filesystem) 219 - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci) 220- [v0.1.0](#v010) 221 - [Shared filesystem](#shared-filesystem) 222 - [Initial direct device assignment support](#initial-direct-device-assignment-support) 223 - [Userspace IOAPIC](#userspace-ioapic) 224 - [Virtual persistent memory](#virtual-persistent-memory) 225 - [Linux kernel bzImage](#linux-kernel-bzimage) 226 - [Console over virtio](#console-over-virtio) 227 - [Unit testing](#unit-testing) 228 - [Integration tests parallelization](#integration-tests-parallelization) 229 230# v26.0 231 232This release has been tracked through the [v26.0 233project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/30). 234 235### SMBIOS Improvements via `--platform` 236 237`--platform` and the appropriate API structure has gained support for supplying 238OEM strings (primarily used to communicate metadata to systemd in the guest) 239(#4319, #4446) and support for specifying the UUID (#4389.) 240 241### Unified Binary MSHV and KVM Support 242 243Support for both the MSHV and KVM hypervisors can be compiled into the same 244binary with the detection of the hypervisor to use made at runtime. 245 246### Notable Bug Fixes 247 248* The prefetchable flag is preserved on BARs for VFIO devices (#4353, #4454) 249* PCI Express capabilties for functionality we do not support are now filtered 250 out (#4456) 251* GDB breakpoint support is more reliable (#4354, #4363) 252* `SIGINT` and `SIGTERM` signals are now handled before the VM has booted 253 (#4269, #4293) 254* Multiple API event loop handling bug fixes (#4309, #4362) 255* Incorrect assumptions in virtio queue numbering were addressed, allowing 256 the`virtio-fs` driver in OVMF to be used (#4341, #4314) 257* VHDX file format header fix (#4291) 258* The same VFIO device cannot be added twice (#4453, #4463) 259* SMBIOS tables were being incorrectly generated (#4442) 260 261### Deprecations 262 263Deprecated features will be removed in a subsequent release and users should 264plan to use alternatives. 265 266* The top-level `kernel` and `initramfs` members on the `VmConfig` have been 267 moved inside a `PayloadConfig` as the `payload` member. The OpenAPI document 268 has been updated to reflect the change and the old API members continue to 269 function and are mapped to the new version. The expectation is that these old 270 versions will be removed in the v28.0 release. 271 272### Removals 273 274The following functionality has been removed: 275 276* The unused `poll_queue` parameter has been removed from `--disk` and 277 equivalent. This was residual from the the removal of the `vhost-user-block` 278 spawning feature (#4402.) 279 280### Contributors 281 282Many thanks to everyone who has contributed to our release: 283 284* Alyssa Ross <hi@alyssa.is> 285* Anatol Belski <ab@php.net> 286* Archana Shinde <archana.m.shinde@intel.com> 287* Bo Chen <chen.bo@intel.com> 288* lizhaoxin1 <Lxiaoyouling@163.com> 289* Maximilian Nitsch <maximilian.nitsch@d3tn.com> 290* Michael Zhao <michael.zhao@arm.com> 291* Rob Bradford <robert.bradford@intel.com> 292* Sebastien Boeuf <sebastien.boeuf@intel.com> 293* Steven Dake <sdake@lambdal.com> 294* Wei Liu <liuwe@microsoft.com> 295 296# v25.0 297 298This release has been tracked through the [v25.0 299project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/29). 300 301### `ch-remote` Improvements 302 303The `ch-remote` command has gained support for creating the VM from a JSON 304config and support for booting and deleting the VM from the VMM. 305 306### VM "Coredump" Support 307 308Under the `guest_debug` feature flag it is now possible to extract the memory 309of the guest for use in debugging with e.g. the `crash` utility. (#4012) 310 311### Notable Bug Fixes 312 313* Always restore console mode on exit (#4249, #4248) 314* Restore vCPUs in numerical order which fixes aarch64 snapshot/restore (#4244) 315* Don't try and configure `IFF_RUNNING` on TAP devices (#4279) 316* Propagate configured queue size through to vhost-user backend (#4286) 317* Always Program vCPU CPUID before running the vCPU to fix running on Linux 318 5.16 (#4156) 319* Enable ACPI MADT "Online Capable" flag for hotpluggable vCPUs to fix newer 320 Linux guest 321 322### Removals 323 324The following functionality has been removed: 325 326* The `mergeable` option from the `virtio-pmem` support has been removed 327 (#3968) 328* The `dax` option from the `virtio-fs` support has been removed (#3889) 329 330### Contributors 331 332Many thanks to everyone who has contributed to our release: 333 334* Dylan Bargatze <dbargatz@users.noreply.github.com> 335* Jinank Jain <jinankjain@microsoft.com> 336* Michael Zhao <michael.zhao@arm.com> 337* Rob Bradford <robert.bradford@intel.com> 338* Sebastien Boeuf <sebastien.boeuf@intel.com> 339* Wei Liu <liuwe@microsoft.com> 340* Yi Wang <wang.yi59@zte.com.cn> 341 342# v24.0 343 344This release has been tracked through the [v24.0 345project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/28). 346 347### Bypass Mode for `virtio-iommu` 348 349`virtio-iommu` specification describes how a device can be attached by default 350to a bypass domain. This feature is particularly helpful for booting a VM with 351guest software which doesn't support `virtio-iommu` but still need to access 352the device. Now that Cloud Hypervisor supports this feature, it can boot a VM 353with Rust Hypervisor Firmware or OVMF even if the `virtio-block` device exposing 354the disk image is placed behind a virtual IOMMU. 355 356### Ensure Identifiers Uniqueness 357 358Multiple checks have been added to the code to prevent devices with identical 359identifiers from being created, and therefore avoid unexpected behaviors at boot 360or whenever a device was hot plugged into the VM. 361 362### Sparse Mmap support 363 364Sparse mmap support has been added to both VFIO and vfio-user devices. This 365allows the device regions that are not fully mappable to be partially mapped. 366And the more a device region can be mapped into the guest address space, the 367fewer VM exits will be generated when this device is accessed. This directly 368impacts the performance related to this device. 369 370### Expose Platform Serial Number 371 372A new `serial_number` option has been added to `--platform`, allowing a user to 373set a specific serial number for the platform. This number is exposed to the 374guest through the SMBIOS. 375 376### Notable Bug Fixes 377 378* Fix loading RAW firmware (#4072) 379* Reject compressed QCOW images (#4055) 380* Reject virtio-mem resize if device is not activated (#4003) 381* Fix potential mmap leaks from VFIO/vfio-user MMIO regions (#4069) 382* Fix algorithm finding HOB memory resources (#3983) 383 384### Notable Improvements 385 386* Refactor interrupt handling (#4083) 387* Load kernel asynchronously (#4022) 388* Only create ACPI memory manager DSDT when resizable (#4013) 389 390### Deprecations 391 392Deprecated features will be removed in a subsequent release and users should 393plan to use alternatives 394 395* The `mergeable` option from the `virtio-pmem` support has been deprecated 396 (#3968) 397* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 398 399### New on the Website 400 401A new blog post [Achieving Bare Metal Performance Within a Virtual 402Machine](https://www.cloudhypervisor.org/blog/achieving-bare-metal-performance-within-a-virtual-machine) 403has been added to the Cloud Hypervisor website. 404 405### Contributors 406 407Many thanks to everyone who has contributed to our release: 408 409* Anatol Belski <anbelski@linux.microsoft.com> 410* Bo Chen <chen.bo@intel.com> 411* Fabiano Fidêncio <fabiano.fidencio@intel.com> 412* LiHui <andrewli@kubesphere.io> 413* Maksym Pavlenko <pavlenko.maksym@gmail.com> 414* Rob Bradford <robert.bradford@intel.com> 415* Sebastien Boeuf <sebastien.boeuf@intel.com> 416* Steven Dake <steven.dake@gmail.com> 417* Vincent Batts <vbatts@hashbangbash.com> 418* Wei Liu <liuwe@microsoft.com> 419 420# v23.1 421 422This is a bug fix release. The following issues have been addressed: 423 424* Add some missing seccomp rules 425* Remove `virtio-fs` filesystem entries from config on removal 426* Do not delete API socket on API server start (#4026) 427* Reject `virtio-mem` resize if the guest doesn't activate the device 428* Fix OpenAPI naming of I/O throttling knobs 429 430# v23.0 431 432This release has been tracked through the [v23.0 433project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/27). 434 435### vDPA Support 436 437A vDPA device has a datapath that complies with the virtio specification but 438with a vendor specific control path. The addition of `--vdpa` and the REST API 439equivalent allows the use of these devices with Cloud Hypervisor. 440 441### Updated OS Support list 442 443The list of officially supported and tested OS versions has been updated to 444include Ubuntu "jammy" 22.04 and EOLed versions removed. 445 446### `AArch64` Memory Map Improvements 447 448The memory map when running on `AArch64` has been improved for the handling of 449the UEFI region which means that the booted guest OS now has full access to its 450allocated RAM. (#3938) 451 452### `AMX` Support 453 454Under a compile time gate of `amx` it is possible compile in support for the 455`AMX` instruction set extension for guest use. This also requires runtime 456enabling with `--cpu features=amx`. 457 458### Notable Bug Fixes 459 460* Generate error when incorrect HTTP method used for some API endpoints (#3887) 461* CMOS based reset is now available to support rebooting on "jammy" (#3914) 462* ACPI tables are not produced for memory hotplug when running with 463 `virtio-mem` (#3883) 464* `virtio-iommu` backed PCI segments are now comprehensively placed behind the 465 vIOMMU (#3870) 466* Seccomp rules have been extended for `virtio-fs` to support direct access 467 (#3848) 468 469### Deprecations 470 471Deprecated features will be removed in a subsequent release and users should 472plan to use alternatives 473 474* The `mergeable` option from the `virtio-pmem` support has been deprecated 475 (#3968) 476* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 477 478### Contributors 479 480Many thanks to everyone who has contributed to our release: 481 482* Bo Chen <chen.bo@intel.com> 483* Fabiano Fidêncio <fabiano.fidencio@intel.com> 484* Henry Wang <Henry.Wang@arm.com> 485* Jianyong Wu <jianyong.wu@arm.com> 486* LiHui <andrewli@kubesphere.io> 487* Michael Zhao <michael.zhao@arm.com> 488* Rob Bradford <robert.bradford@intel.com> 489* Sebastien Boeuf <sebastien.boeuf@intel.com> 490* Wei Liu <liuwe@microsoft.com> 491* William Douglas <william.douglas@intel.com> 492* Yi Wang <wang.yi59@zte.com.cn> 493 494# v22.1 495 496This is a bug fix release. The following issues have been addressed: 497 498* VFIO ioctl reordering to fix MSI on AMD platforms (#3827) 499* Fix `virtio-net` control queue (#3829) 500 501# v22.0 502 503This release has been tracked through the [v22.0 504project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/25). 505 506### GDB Debug Stub Support 507 508Cloud Hypervisor can now be used as debug target with GDB. This is controlled 509by the `gdb` compile time feature and details of how to use it can be found in 510the [gdb 511documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/gdb.md). 512 513### `virtio-iommu` Backed Segments 514 515In order to facilitate hotplug devices that require being behind an IOMMU (e.g. 516QAT) there is a new option `--platform iommu_segments=<list_of_segments>` that 517will place all the specified segments behind the IOMMU. 518 519### Before Boot Configuration Changes 520 521It is now possible to change the VM configuration (e.g. add or remove devices, 522resize) before the VM is booted. 523 524### `virtio-balloon` Free Page Reporting 525 526If `--balloon free_page_reporting=on` is used then the guest can report pages 527that is it not using to the VMM. The VMM will then notify the host OS that 528those pages are no longer in use and can be freed. This can result in improved 529memory density. 530 531### Support for Direct Kernel Booting with TDX 532 533Through the use of `TD-Shim` lightweight firmware it is now possible to 534directly boot into the kernel with TDX. The [TDX 535documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/intel_tdx.md#tdshim) 536has been updated for this usage. 537 538### PMU Support for AArch64 539 540A PMU is now available on AArch64 for guest performance profiling. This will be 541exposed automatically if available from the host. 542 543### Documentation Under CC-BY-4.0 License 544 545The documentation is now licensed under the "Creative Commons Attribution 4.0 546International" license which is aligned with the project charter under the 547Linux Foundation. 548 549### Deprecation of "Classic" `virtiofsd` 550 551The use of the Rust based [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd) 552is now recommended and we are no longer testing against the C based "classic" 553version. 554 555### Notable Bug Fixes 556 557* Can now be used on kernels without `AF_INET` support (#3785) 558* `virtio-balloon` size is now validated against guest RAM size (#3689) 559* Ensure that I/O related KVM VM Exits are correctly handled (#3677) 560* Multiple TAP file descriptors can be used for `virtio-net` device hotplug (#3607) 561* Minor API improvements and fixes (#3756, #3766, #3647, #3578) 562* Fix sporadic seccomp violation from glibc memory freeing (#3610, #3609) 563* Fix Windows 11 on AArch64 due to wider MSI-X register accesses (#3714, #3720) 564* Ensure `vhost-user` features are correct across migration (#3737) 565* Improved vCPU topology on AArch64 (#3735, #3733) 566 567### Contributors 568 569Many thanks to everyone who has contributed to our release: 570 571* Akira Moroo <retrage01@gmail.com> 572* Barret Rhoden <brho@google.com> 573* Bo Chen <chen.bo@intel.com> 574* Fabiano Fidêncio <fabiano.fidencio@intel.com> 575* Feng Ye <yefeng@smartx.com> 576* Henry Wang <Henry.Wang@arm.com> 577* Jianyong Wu <jianyong.wu@arm.com> 578* lizhaoxin1 <Lxiaoyouling@163.com> 579* Michael Zhao <michael.zhao@arm.com> 580* Rob Bradford <robert.bradford@intel.com> 581* Sebastien Boeuf <sebastien.boeuf@intel.com> 582* Wei Liu <liuwe@microsoft.com> 583 584# v21.0 585 586This release has been tracked through the [v21.0 587project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/24). 588 589### Efficient Local Live Migration (for Live Upgrade) 590 591In order to support fast live upgrade of the VMM an optimised path has been 592added in which the memory for the VM is not compared from source to 593destination. This is activated by passing `--local` to the `ch-remote 594send-migration` command. This means that the live upgrade can complete in the 595order of 50ms vs 3s. (#3566) 596 597### Recommended Kernel is Now 5.15 598 599Due to an issue in the `virtio-net` code in 5.14 the recommended Linux kernel 600is now 5.15. (#3530) 601 602### Notable Bug fixes 603 604* Multiple fixes were made to the OpenAPI YAML file to match the implementaion (#3555,#3562) 605* Avoid live migration deadlock when triggered during the kernel boot (#3585) 606* Support live migration within firmware (#3586) 607* Validate the `virtio-net` desciptor chain (#3548) 608* `direct=on` (`O_DIRECT`) can now be used with a guest that makes unaligned accesses (e.g. firmware) (#3587) 609 610### Contributors 611 612Many thanks to everyone who has contributed to our release: 613 614* Anatol Belski <anbelski@linux.microsoft.com> 615* Barret Rhoden <brho@google.com> 616* Bo Chen <chen.bo@intel.com> 617* Fabiano Fidêncio <fabiano.fidencio@intel.com> 618* Henry Wang <Henry.Wang@arm.com> 619* Liang Zhou <zhoul110@chinatelecom.cn> 620* Michael Zhao <michael.zhao@arm.com> 621* Muhammad Falak R Wani <falakreyaz@gmail.com> 622* Muminul Islam <muislam@microsoft.com> 623* Rob Bradford <robert.bradford@intel.com> 624* Sebastien Boeuf <sebastien.boeuf@intel.com> 625* Wei Liu <liuwe@microsoft.com> 626* Ziye Yang <ziye.yang@intel.com> 627 628# v20.2 629 630This is a bug fix release. The following issues have been addressed: 631 632* Don't error out when setting up the SIGWINCH handler (for console resize) 633 when this fails due to older kernel (#3456) 634* Seccomp rules were refined to remove syscalls that are now unused 635* Fix reboot on older host kernels when SIGWINCH handler was not initialised 636 (#3496) 637* Fix virtio-vsock blocking issue (#3497) 638 639# v20.1 640 641This is a bug fix release. The following issues have been addressed: 642 643* Networking performance regression with `virtio-net` (#3450) 644* Limit file descriptors sent in `vfio-user` support (#3401) 645* Fully advertise PCI MMIO config regions in ACPI tables (#3432) 646* Set the TSS and KVM identity maps so they don't overlap with firmware RAM 647* Correctly update the `DeviceTree` on restore 648 649# v20.0 650 651This release has been tracked through the [v20.0 652project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/23). 653 654### Multiple PCI segments support 655 656Cloud Hypervisor is no longer limited to 31 PCI devices. For both `x86_64` and 657`aarch64` architectures, it is now possible to create up to 16 PCI segments, 658increasing the total amount of supported PCI devices to 496. 659 660### CPU pinning 661 662For each vCPU, the user can define a limited set of host CPUs on which it is 663allowed to run. This can be useful when assigning a 1:1 mapping between host and 664guest resources, or when running a VM on a specific NUMA node. 665 666### Improved VFIO support 667 668Based on VFIO region capabilities, all regions can be memory mapped, limiting 669the amount of triggered VM exits, and therefore increasing the performance of 670the passthrough device. 671 672### Safer code 673 674Several sections containing unsafe Rust code have been replaced with safe 675alternatives, and multiple comments have been added to clarify why the remaining 676unsafe sections are safe to use. 677 678### Extended documentation 679 680The documentation related to VFIO has been updated while some new documents have 681been introduced to cover the usage of `--cpus` parameter as well as how to run 682Cloud Hypervisor on Intel TDX. 683 684### Notable bug fixes 685 686* Naturally align PCI BARs on relocation (#3244) 687* Fix panic in SIGWINCH listener thread when no seccomp filter set (#3338) 688* Use the tty raw mode implementation from libc (#3344) 689* Fix the emulation of register D for CMOS/RTC device (#3393) 690 691### Contributors 692 693Many thanks to everyone who has contributed to our release: 694 695* Alyssa Ross <hi@alyssa.is> 696* Bo Chen <chen.bo@intel.com> 697* Fabiano Fidêncio <fabiano.fidencio@intel.com> 698* Michael Zhao <michael.zhao@arm.com> 699* Muminul Islam <muislam@microsoft.com> 700* Rob Bradford <robert.bradford@intel.com> 701* Sebastien Boeuf <sebastien.boeuf@intel.com> 702* Wei Liu <liuwe@microsoft.com> 703* Willen Yang <willenyang@gmail.com> 704* William Douglas <william.douglas@intel.com> 705* Ziye Yang <ziye.yang@intel.com> 706 707# v19.0 708 709This release has been tracked through the [v19.0 710project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/22). 711 712### Improved PTY handling for serial and `virtio-console` 713 714The PTY support for serial has been enhanced with improved buffering when the 715the PTY is not yet connected to. Using `virtio-console` with PTY now results in 716the console being resized if the PTY window is also resized. 717 718### PCI boot time optimisations 719 720Multiple optimisations have been made to the PCI handling resulting in 721significant improvements in the boot time of the guest. 722 723### Improved TDX support 724 725When using the latest TDVF firmware the ACPI tables created by the VMM are now 726exposed via the firmware to the guest. 727 728### Live migration enhancements 729 730Live migration support has been enhanced to support migration with `virtio-mem` 731based memory hotplug and the `virtio-balloon` device now supports live 732migration. 733 734### `virtio-mem` support with `vfio-user` 735 736The use of `vfio-user` userspaces devices can now be used in conjunction with 737`virtio-mem` based memory hotplug and unplug. 738 739### AArch64 for `virtio-iommu` 740 741A paravirtualised IOMMU can now be used on the AArch64 platform. 742 743### Notable bug fixes 744 745* ACPI hotplugged memory is correctly restored after a live migration or 746 snapshot/restore (#3165) 747* Multiple devices from the same IOMMU group can be passed through via VFIO 748 (#3078 #3113) 749* Live migration with large blocks of memory was buggy due to an in issue in 750 the underlying crate (#3157) 751 752### Contributors 753 754Many thanks to everyone who has contributed to our release: 755 756* Alyssa Ross <hi@alyssa.is> 757* Bo Chen <chen.bo@intel.com> 758* Henry Wang <Henry.Wang@arm.com> 759* Hui Zhu <teawater@antfin.com> 760* Jianyong Wu <jianyong.wu@arm.com> 761* Li Yu <liyu.yukiteru@bytedance.com> 762* Michael Zhao <michael.zhao@arm.com> 763* Muminul Islam <muislam@microsoft.com> 764* Rob Bradford <robert.bradford@intel.com> 765* Sebastien Boeuf <sebastien.boeuf@intel.com> 766* Wei Liu <liuwe@microsoft.com> 767* William Douglas <william.douglas@intel.com> 768* Yu Li <liyu.yukiteru@bytedance.com> 769 770# v18.0 771 772This release has been tracked through the [v18.0 773project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/21). 774 775### Experimental User Device (`vfio-user`) support 776 777Experimental support for running PCI devices in userspace via `vfio-user` 778has been included. This allows the use of the SPDK NVMe `vfio-user` controller 779with Cloud Hypervisor. This is enabled by `--user-device` on the command line. 780 781### Migration support for `vhost-user` devices 782 783Devices exposed into the VM via `vhost-user` can now be migrated using the live 784migration support. This requires support from the backend however the commonly 785used DPDK `vhost-user` backend does support this. 786 787### VHDX disk image support 788 789Images using the VHDX disk image format can now be used with Cloud Hypervisor. 790 791### Device pass through on MSHV hypervisor 792 793When running on the MSHV hypervisor it is possible to pass through devices from 794the host through to the guest (e.g with `--device`) 795 796### AArch64 for support `virtio-mem` 797 798The reference Linux kernel we recommend for using with Cloud Hypervisor now supports `virtio-mem` on AArch64. 799 800### Live migration on MSHV hypervisor 801 802Live migration is now supported when running on the MSHV hypervisor including 803efficient tracking of dirty pages. 804 805### AArch64 CPU topology support 806 807The CPU topology (as configured through `--cpu topology=`) can now be 808configured on AArch64 platforms and is conveyed through either ACPI or device 809tree. 810 811### Power button support on AArch64 812 813Use of the ACPI power button (e.g `ch-remote --api-socket=<API socket> power-button`) 814is now supported when running on AArch64. 815 816### Notable bug fixes 817 818* Using two PTY outputs e.g. `--serial pty --console pty` now works correctly (#3012) 819* TTY input is now always sent to the correct destination (#3005) 820* The boot is no longer blocked when using a unattached PTY on the serial console (#3004) 821* Live migration is now supported on AArch64 (#3049) 822* Ensure signal handlers are run on the correct thread (#3069) 823 824### Contributors 825 826Many thanks to everyone who has contributed to our release: 827 828* Alyssa Ross <hi@alyssa.is> 829* Anatol Belski <anbelski@linux.microsoft.com> 830* Arafatms <arafatms@outlook.com> 831* Bo Chen <chen.bo@intel.com> 832* Fazla Mehrab <akm.fazla.mehrab@vt.edu> 833* Henry Wang <Henry.Wang@arm.com> 834* Jianyong Wu <jianyong.wu@arm.com> 835* Jiaqi Gao <jiaqi.gao@intel.com> 836* Markus Theil <markus.theil@tu-ilmenau.de> 837* Muminul Islam <muislam@microsoft.com> 838* Rob Bradford <robert.bradford@intel.com> 839* Sebastien Boeuf <sebastien.boeuf@intel.com> 840* Wei Liu <liuwe@microsoft.com> 841* Yu Li <liyu.yukiteru@bytedance.com> 842 843# v17.0 844 845This release has been tracked through the [v17.0 846project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/20). 847 848### ARM64 NUMA support using ACPI 849 850The support for ACPI on ARM64 has been enhanced to include support for 851specifying a NUMA configuration using the existing control options. 852 853### `Seccomp` support for MSHV backend 854 855The `seccomp` rules have now been extended to support running against the MSHV 856hypervisor backend. 857 858### Hotplug of `macvtap` devices 859 860Hotplug of `macvtap` devices is now supported with the file descriptor for the 861network device if opened by the user and passed to the VMM. The `ch-remote` 862tool supports this functionality when adding a network device. 863 864### Improved SGX support 865 866The SGX support has been updated to match the latest Linux kernel support and 867now supports SGX provisioning and associating EPC sections to NUMA nodes. 868 869### Inflight tracking for `vhost-user` devices 870 871Support for handling inflight tracking of I/O requests has been added to the 872`vhost-user` devices allowing recovery after device reconnection. 873 874### Notable bug fixes 875 876* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821). 877* The VMM side of `vhost-user` devices no longer advertise the 878 `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM 879(#2833). 880* On ARM64 VMs can be created with more than 16 vCPUs (#2763). 881 882### Contributors 883 884Many thanks to everyone who has contributed to our release: 885 886* Anatol Belski <anbelski@linux.microsoft.com> 887* Arafatms <arafatms@outlook.com> 888* Bo Chen <chen.bo@intel.com> 889* Fei Li <lifei.shirley@bytedance.com> 890* Henry Wang <Henry.Wang@arm.com> 891* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 892* Jianyong Wu <jianyong.wu@arm.com> 893* Li Hangjing <lihangjing@bytedance.com> 894* Michael Zhao <michael.zhao@arm.com> 895* Muminul Islam <muislam@microsoft.com> 896* Rob Bradford <robert.bradford@intel.com> 897* Sebastien Boeuf <sebastien.boeuf@intel.com> 898* Wei Liu <liuwe@microsoft.com> 899* Yukiteru <wfly1998@sina.com> 900 901# v16.0 902 903This release has been tracked through the [v16.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/19). 904 905### Improved live migration support 906 907The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details. 908 909### Improved `vhost-user` support 910 911When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch. 912 913### ARM64 ACPI and UEFI support 914 915Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot. 916 917### Notable bug fixes 918 919* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578). 920* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563). 921* The KVM clock is now correctly handled during live migration / snapshot & restore. 922 923### Removed functionality 924 925The following formerly deprecated features have been removed: 926 927* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 928 binaries has been deprecated. When using direct boot users should configure 929 their kernel with `CONFIG_PVH=y`. 930 931### Contributors 932 933Many thanks to everyone who has contributed to our release including some new faces. 934 935* Anatol Belski <anbelski@linux.microsoft.com> 936* Bo Chen <chen.bo@intel.com> 937* Dayu Liu <liu.dayu@zte.com.cn> 938* Henry Wang <Henry.Wang@arm.com> 939* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 940* Jianyong Wu <jianyong.wu@arm.com> 941* Michael Zhao <michael.zhao@arm.com> 942* Mikko Ylinen <mikko.ylinen@intel.com> 943* Muminul Islam <muislam@microsoft.com> 944* Ren Lei <ren.lei4@zte.com.cn> 945* Rob Bradford <robert.bradford@intel.com> 946* Sebastien Boeuf <sebastien.boeuf@intel.com> 947* Wei Liu <liuwe@microsoft.com> 948* Yi Wang <wang.yi59@zte.com.cn> 949 950# v15.0 951 952This release has been tracked through the [v15.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/18). 953 954Highlights for `cloud-hypervisor` version v15.0 include: 955 956### Version numbering and stability guarantees 957 958This release is the first in a new version numbering scheme to represent that 959we believe Cloud Hypervisor is maturing and entering a period of stability. 960With this new release we are beginning our new stability guarantees: 961 962* The API (including command line options) will not be removed or changed in a 963 breaking way without a minimum of 2 releases notice. Where possible warnings 964 will be given about the use of deprecated functionality and the deprecations 965 will be documented in the release notes. 966* Point releases will be made between individual releases where there are 967 substantial bug fixes or security issues that need to be fixed. 968 969Currently the following items are **not** guaranteed across updates: 970 971* Snapshot/restore is not supported across different versions 972* Live migration is not supported across different versions 973* The following features are considered experimental and may change 974 substantially between releases: TDX, SGX. 975 976### Network device rate limiting 977 978Building on our existing support for rate limiting block activity the network 979device also now supports rate limiting. Full details of the controls are in the 980[IO throttling documentation.](docs/io_throttling.md) 981 982### Support for runtime control of `virtio-net` guest offload 983 984The guest is now able to change the offload settings for the `virtio-net` 985device. As well as providing a useful control this mitigates an issue in the 986Linux kernel where the guest will attempt to reprogram the offload settings 987even if they are not advertised as configurable (#2528). 988 989### `--api-socket` supports file descriptor parameter 990 991The `--api-socket` can now take an `fd=` parameter to specify an existing file 992descriptor to use. This is particularly beneficial for frameworks that need to 993programmatically control Cloud Hypervisor. 994 995### Bug fixes 996 997* A workaround has been put in place to mitigate a Linux kernel issues that 998 results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277). 999* PCI BARs are now correctly aligned removing the need for the guest to 1000 reprogram them (#1797,#1798) 1001* Handle TAP interface not being writable within virtio-net (due to the buffer 1002 exhaustion on the host) (#2517) 1003* The recommended Linux kernel is now v5.12.0 as it contains a fix that 1004 prevents snapshot & restore working (#2535) 1005 1006### Deprecations 1007 1008Deprecated features will be removed in a subsequent release and users should plan to use alternatives 1009 1010* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 1011 binaries has been deprecated. When using direct boot users should configure 1012 their kernel with `CONFIG_PVH=y`. Will be removed in v16.0. 1013 1014### Contributors 1015 1016Many thanks to everyone who has contributed to our release including some new faces. 1017 1018* Alyssa Ross <hi@alyssa.is> 1019* Anatol Belski <anbelski@linux.microsoft.com> 1020* Bo Chen <chen.bo@intel.com> 1021* Gaelan Steele <gbs@canishe.com> 1022* Jianyong Wu <jianyong.wu@arm.com> 1023* Michael Zhao <michael.zhao@arm.com> 1024* Muminul Islam <muislam@microsoft.com> 1025* Rob Bradford <robert.bradford@intel.com> 1026* Sebastien Boeuf <sebastien.boeuf@intel.com> 1027* Wei Liu <liuwe@microsoft.com> 1028* William Douglas <william.douglas@intel.com> 1029 1030# v0.14.1 1031 1032Bug fix release branched off the v0.14.0 release. The following bugs were fixed 1033in this release: 1034 1035* CPU hotplug on Windows failed due to misreported CPU state information and 1036 the lack of HyperV CPUID bit enabled (#2437, #2449, #2436) 1037* A seccomp rule was missing that was triggered on CPU unplug (#2455) 1038* A bounds check in VIRTIO queue validation was erroneously generating 1039 DescriptorChainTooShort errors in certain circumstances (#2450, #2424) 1040 1041# v0.14.0 1042 1043This release has been tracked through the [0.14.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/17). 1044 1045Highlights for `cloud-hypervisor` version 0.14.0 include: 1046 1047### Structured event monitoring 1048 1049A new option was added to the VMM `--event-monitor` which reports structured 1050events (JSON) over a file or file descriptor at key events in the lifecycle of 1051the VM. The list of events is limited at the moment but will be further 1052extended over subsequent releases. The events exposed form part of the Cloud 1053Hypervisor API surface. 1054 1055### MSHV improvements 1056 1057Basic support has been added for running Windows guests atop the MSHV 1058hypervisor as an alternative to KVM and further improvements have been made to 1059the MSHV support. 1060 1061### Improved aarch64 platform 1062 1063The aarch64 platform has been enhanced with more devices exposed to the running 1064VM including an enhanced serial UART. 1065 1066### Updated hotplug documentation 1067 1068The documentation for the hotplug support has been updated to reflect the use 1069of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as 1070well as documenting hotplug of paravirtualised and VFIO devices. 1071 1072### PTY control for serial and `virtio-console` 1073 1074The `--serial` and `--console` parameters can now direct the console to a PTY 1075allowing programmatic control of the console from another process through the 1076PTY subsystem. 1077 1078### Block device rate limiting 1079 1080The block device performance can now be constrained as part of the VM 1081configuration allowing rate limiting. Full details of the controls are in the 1082[IO throttling documentation.](docs/io_throttling.md) 1083 1084 1085### Deprecations 1086 1087Deprecated features will be removed in a subsequent release and users should plan to use alternatives 1088 1089* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 1090 binaries has been deprecated. When using direct boot users should configure 1091 their kernel with `CONFIG_PVH=y`. 1092 1093 1094### Contributors 1095 1096Many thanks to everyone who has contributed to our 0.14.0 release including 1097some new faces. 1098 1099Bo Chen <chen.bo@intel.com> 1100Henry Wang <Henry.Wang@arm.com> 1101Iggy Jackson <iggy@theiggy.com> 1102Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 1103Michael Zhao <michael.zhao@arm.com> 1104Muminul Islam <muislam@microsoft.com> 1105Penny Zheng <Penny.Zheng@arm.com> 1106Rob Bradford <robert.bradford@intel.com> 1107Sebastien Boeuf <sebastien.boeuf@intel.com> 1108Vineeth Pillai <viremana@linux.microsoft.com> 1109Wei Liu <liuwe@microsoft.com> 1110William Douglas <william.r.douglas@gmail.com> 1111Zide Chen <zide.chen@intel.com> 1112 1113# v0.13.0 1114 1115This release has been tracked through the [0.13.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/16). 1116 1117Highlights for `cloud-hypervisor` version 0.13.0 include: 1118 1119### Wider VFIO device support 1120 1121It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI 1122devices that do not support MSI or MSI-X and instead rely on INTx interrupts. 1123Most notably this widens the support to most NVIDIA cards with the proprietary 1124drivers. 1125 1126### Improved huge page support 1127 1128Through the addition of `hugepage_size` on `--memory` it is now possible to 1129specify the desired size of the huge pages used when allocating the guest 1130memory. The user is required to ensure they have sufficient pages of the 1131desired size in their pool. 1132 1133### MACvTAP support 1134 1135It is now possible to provide file descriptors using the `fd` parameter to 1136`--net` which point at TAP devices that have already been opened by the user. 1137This aids integration with `libvirt` but also permits the use of MACvTAP 1138support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md). 1139 1140### VHD disk image support 1141 1142It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw 1143disk image with Cloud Hypervisor. 1144 1145### Improved Virtio device threading 1146 1147Device threads are now derived from the main VMM thread which allows more 1148restrictive seccomp filters to be applied to them. The threads also have a 1149predictable name derived from the device id. 1150 1151### Clean shutdown support via synthetic power button 1152 1153It is now possible to request that the guest VM shut itself down by triggering 1154a synthetic ACPI power button press from the VMM. If the guest is listening for 1155such an event (e.g. using systemd) then it will process the event and cleanly 1156shut down. This functionality is exposed through the HTTP API and can be 1157triggered via `ch-remote --api-socket=<API socket> power-button`. 1158 1159### Contributors 1160 1161Many thanks to everyone who has contributed to our 0.13.0 release including 1162some new faces. 1163 1164* Bo Chen <chen.bo@intel.com> 1165* Mikko Ylinen <mikko.ylinen@intel.com> 1166* Muminul Islam <muislam@microsoft.com> 1167* Rob Bradford <robert.bradford@intel.com> 1168* Samuel Ortiz <sameo@linux.intel.com> 1169* Sebastien Boeuf <sebastien.boeuf@intel.com> 1170* Vineeth Pillai <viremana@linux.microsoft.com> 1171* Wei Liu <liuwe@microsoft.com> 1172* William Douglas <william.r.douglas@gmail.com> 1173* Xie Yongji <xieyongji@bytedance.com> 1174 1175# v0.12.0 1176 1177This release has been tracked through the [0.12.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/15). 1178 1179Highlights for `cloud-hypervisor` version 0.12.0 include: 1180 1181### ARM64 enhancements 1182 1183The use of `--watchdog` is now fully supported as is the ability to reboot the 1184VM from within the guest when running Cloud Hypervisor on an ARM64 system. 1185 1186### Removal of `vhost-user-net` and `vhost-user-block` self spawning 1187 1188In order to use `vhost-user-net` or `vhost-user-block` backends the user is now 1189responsible for starting the backend and providing the socket for the VMM to 1190use. This functionality was deprecated in the last release and how now been 1191removed. 1192 1193### Migration of `vhost-user-fs` backend 1194 1195The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is 1196instead hosted in [it's own 1197repository](https://gitlab.com/virtio-fs/virtiofsd-rs) 1198 1199### Enhanced "info" API 1200 1201The `vm.info` HTTP API endpoint has been extended to include the details of the 1202devices used by the VM including any VFIO devices used. 1203 1204### Contributors 1205 1206Many thanks to everyone who has contributed to our 0.12.0 release: 1207 1208* Anatol Belski <anbelski@linux.microsoft.com> 1209* Julio Montes <julio.montes@intel.com> 1210* Michael Zhao <michael.zhao@arm.com> 1211* Muminul Islam <muislam@microsoft.com> 1212* Rob Bradford <robert.bradford@intel.com> 1213* Samuel Ortiz <sameo@linux.intel.com> 1214* Wei Liu <liuwe@microsoft.com> 1215 1216# v0.11.0 1217 1218This release has been tracked through the [0.11.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/14). 1219 1220Highlights for `cloud-hypervisor` version 0.11.0 include: 1221 1222### `io_uring` support by default for `virtio-block` 1223 1224Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will 1225be used for a significantly higher performance block device. 1226 1227### Windows Guest Support 1228 1229This is the first release where we officially support Windows running as a 1230guest. Full details of how to setup the image and run Cloud Hypervisor with a 1231Windows guest can be found in the dedicated [Windows 1232documentation](docs/windows.md). 1233 1234### `vhost-user` "Self Spawning" Deprecation 1235 1236Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now 1237deprecated. Users of this functionality will receive a warning and should make 1238adjustments. The functionality will be removed in the next release. 1239 1240### `virtio-mmio` Removal 1241 1242Support for using the `virtio-mmio` transport, rather than using PCI, has been 1243removed. This has been to simplify the code and significantly 1244reduce the testing burden of the project. 1245 1246### Snapshot/Restore support for ARM64 1247 1248When running on the ARM64 architecture snapshot and restore has now been 1249implemented. 1250 1251### Improved Linux Boot Time 1252 1253The time to boot the Linux kernel has been significantly improved by the 1254identifying some areas of delays around PCI bus probing, IOAPIC programming and 1255MPTABLE issues. Full details can be seen in #1728. 1256 1257### `SIGTERM/SIGINT` Interrupt Signal Handling 1258 1259When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will 1260trigger the VMM process to cleanly deallocate resources before exiting. The 1261guest VM will not be cleanly shutdown but the VMM process will clean up its 1262resources. 1263 1264### Default Log Level Changed 1265 1266The default logging level was changed to include warnings which should make it 1267easier to see potential issues. New [logging 1268documentation](docs/logging) was also added. 1269 1270### New `--balloon` Parameter Added 1271 1272Control of the setup of `virtio-balloon` has been moved from `--memory` to its 1273own dedicated parameter. This makes it easier to add more balloon specific 1274controls without overloading `--memory`. 1275 1276### Experimental `virtio-watchdog` Support 1277 1278Support for using a new `virtio-watchdog` has been added which can be used to 1279have the VMM reboot the guest if the guest userspace fails to ping the 1280watchdog. This is enabled with `--watchdog` and requires kernel support. 1281 1282### Notable Bug Fixes 1283 1284* MTRR bit was missing from CPUID advertised to guest 1285* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170) 1286* CPU identification string is now exposed to the guest 1287* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as 1288 read only so avoids excessive VM exits (#1795) 1289* PCI device hotplug after an unplug was fixed (#1802) 1290* When using the ACPI method to resize the guest memory the full reserved size 1291 can be used (#1803) 1292* Snapshot and restore followed by a second snapshot and restore now works 1293 correctly 1294* Snapshot and restore of VMs with more than 2GiB in one region now work 1295 correctly 1296 1297### Contributors 1298 1299Many thanks to everyone who has contributed to our 0.11.0 release including some new faces. 1300 1301* Anatol Belski <anbelski@linux.microsoft.com> 1302* Bo Chen <chen.bo@intel.com> 1303* Daniel Verkamp <dverkamp@chromium.org> 1304* Henry Wang <Henry.Wang@arm.com> 1305* Hui Zhu <teawater@antfin.com> 1306* Jiangbo Wu <jiangbo.wu@intel.com> 1307* Josh Soref <jsoref@users.noreply.github.com> 1308* Julio Montes <julio.montes@intel.com> 1309* Michael Zhao <michael.zhao@arm.com> 1310* Muminul Islam <muislam@microsoft.com> 1311* pierwill <19642016+pierwill@users.noreply.github.com> 1312* Praveen Paladugu <prapal@microsoft.com> 1313* Rob Bradford <robert.bradford@intel.com> 1314* Sebastien Boeuf <sebastien.boeuf@intel.com> 1315* Wei Liu <liuwe@microsoft.com> 1316 1317 1318# v0.10.0 1319 1320This release has been tracked through the [0.10.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/13). 1321 1322Highlights for `cloud-hypervisor` version 0.10.0 include: 1323 1324### `virtio-block` Support for Multiple Descriptors 1325 1326Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers. 1327 1328### Memory Zones 1329 1330Support has been added for fine grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md). 1331 1332### `Seccomp` Sandbox Improvements 1333 1334All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`. 1335 1336### Preliminary KVM HyperV Emulation Control 1337 1338A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices. 1339 1340### Notable Bug Fixes 1341 1342- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596) 1343- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564) 1344- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593) 1345- `--version` shows the version for released binaries (#1669) 1346- Errors generated by worker threads for `virtio` devices are now printed out (#1551) 1347 1348### Contributors 1349 1350Many thanks to everyone who has contributed to our 0.10.0 release including some new faces. 1351 1352* Alyssa Ross <hi@alyssa.is> 1353* Amey Narkhede <ameynarkhede02@gmail.com> 1354* Anatol Belski <ab@php.net> 1355* Bo Chen <chen.bo@intel.com> 1356* Hui Zhu <teawater@antfin.com> 1357* Michael Zhao <michael.zhao@arm.com> 1358* Muminul Islam <muislam@microsoft.com> 1359* Rob Bradford <robert.bradford@intel.com> 1360* Samuel Ortiz <sameo@linux.intel.com> 1361* Sebastien Boeuf <sebastien.boeuf@intel.com> 1362* Wei Liu <liuwe@microsoft.com> 1363 1364# v0.9.0 1365 1366This release has been tracked through the [0.9.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/12). 1367 1368Highlights for `cloud-hypervisor` version 0.9.0 include: 1369 1370### `io_uring` Based Block Device Support 1371 1372If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement. 1373 1374### Block and Network Device Statistics 1375 1376Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM. 1377 1378### HTTP API Responses 1379 1380The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF. 1381 1382### CPU Topology 1383 1384A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core. 1385 1386### Release Build Optimization 1387 1388Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size. 1389 1390### Hypervisor Abstraction 1391 1392A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`. 1393 1394### Snapshot/Restore Improvements 1395 1396Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs. 1397 1398### Virtio Memory Ballooning Support 1399 1400A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest. 1401 1402### Enhancements to ARM64 Support 1403 1404The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting. 1405 1406### Intel SGX Support 1407 1408The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md). 1409 1410### `Seccomp` Sandbox Improvements 1411 1412The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development. 1413 1414### Notable Bug Fixes 1415 1416* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes. 1417* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338) 1418* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543). 1419* `virtio-mmio` based devices are now more widely tested (#275). 1420* Multiple issues have been fixed with virtio device configuration (#1217) 1421* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521) 1422 1423### Contributors 1424 1425Many thanks to everyone who has contributed to our 0.9.0 release including some new faces. 1426 1427* Anatol Belski <ab@php.net> 1428* Bo Chen <chen.bo@intel.com> 1429* Dr. David Alan Gilbert <dgilbert@redhat.com> 1430* Henry Wang <Henry.Wang@arm.com> 1431* Howard Zhang <howard.zhang@arm.com> 1432* Hui Zhu <teawater@antfin.com> 1433* Jianyong Wu <jianyong.wu@arm.com> 1434* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1435* LiYa'nan <oliverliyn@gmail.com> 1436* Michael Zhao <michael.zhao@arm.com> 1437* Muminul Islam <muislam@microsoft.com> 1438* Praveen Paladugu <prapal@microsoft.com> 1439* Ricardo Koller <ricarkol@gmail.com> 1440* Rob Bradford <robert.bradford@intel.com> 1441* Samuel Ortiz <sameo@linux.intel.com> 1442* Sebastien Boeuf <sebastien.boeuf@intel.com> 1443* Stefano Garzarella <sgarzare@redhat.com> 1444* Wei Liu <liuwe@microsoft.com> 1445 1446 1447# v0.8.0 1448 1449This release has been tracked through the [0.8.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/10). 1450 1451Highlights for `cloud-hypervisor` version 0.8.0 include: 1452 1453### Experimental Snapshot and Restore Support 1454 1455This release includes the first version of the snapshot and restore feature. 1456This allows a VM to be paused and then subsequently snapshotted. At a later 1457point that snapshot may be restored into a new running VM identical to the 1458original VM at the point it was paused. 1459 1460This feature can be used for offline migration from one VM host to another, to 1461allow the upgrading or rebooting of the host machine transparently to the guest 1462or for templating the VM. This is an experimental feature and cannot be used on 1463a VM using passthrough (VFIO) devices. Issues with SMP have also been observed 1464(#1176). 1465 1466### Experimental ARM64 Support 1467 1468Included in this release is experimental support for running on ARM64. 1469Currently only `virtio-mmio` devices and a serial port are supported. Full 1470details can be found in the [ARM64 documentation](docs/arm64.md). 1471 1472### Support for Using 5-level Paging in Guests 1473 1474If the host supports it the guest is now enabled for 5-level paging (aka LA57). 1475This works when booting the Linux kernel with a vmlinux, bzImage or firmware 1476based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not 1477work due to current limitations in the PVH boot process. 1478 1479### Virtio Device Interrupt Suppression for Network Devices 1480 1481With `virtio-net` and `vhost-user-net` devices the guest can suppress 1482interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This 1483can lead to an improvement in performance by reducing the number of interrupts 1484the guest must service. 1485 1486### `vhost_user_fs` Improvements 1487 1488The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`. 1489 1490 1491### Notable Bug Fixes 1492 1493* VMs that have not yet been booted can now be deleted (#1110). 1494* By creating the `tap` device ahead of creating the VM it is not required to 1495 run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273). 1496* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to 1497 the specification and synchronizes to the underlying filesystem as required 1498 based on guest feature negotiation. This avoids potential data loss (#399, 1499 #1216). 1500* When booting with a large number of vCPUs then the ACPI table would be 1501 overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the 1502 `MPTABLE` will no longer be generated (#1132). 1503* Shutting down VMs that have been paused is now supported (#816). 1504* Created socket files are deleted on shutdown (#1083). 1505* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds 1506 (#751). 1507 1508### Command Line and API Changes 1509 1510This is non exhaustive list of HTTP API and command line changes: 1511 1512* All user visible socket parameters are now consistently called `socket` 1513 rather than `sock` in some cases. 1514* The `ch-remote` tool now shows any error message generated by the VMM 1515* The `wce` parameter has been removed from `--disk` as the feature is always 1516 offered for negotiation. 1517* `--net` has gained a `host_mac` option that allows the setting of the MAC 1518 address for the `tap` device on the host. 1519 1520### Contributors 1521 1522Many thanks to everyone who has contributed to our 0.8.0 release including some new faces. 1523 1524* Anatol Belski <ab@php.net> 1525* Arron Wang <arron.wang@intel.com> 1526* Bo Chen <chen.bo@intel.com> 1527* Dr. David Alan Gilbert <dgilbert@redhat.com> 1528* Henry Wang <Henry.Wang@arm.com> 1529* Hui Zhu <teawater@antfin.com> 1530* LiYa'nan <oliverliyn@gmail.com> 1531* Michael Zhao <michael.zhao@arm.com> 1532* Rob Bradford <robert.bradford@intel.com> 1533* Samuel Ortiz <sameo@linux.intel.com> 1534* Sebastien Boeuf <sebastien.boeuf@intel.com> 1535* Sergio Lopez <slp@redhat.com> 1536 1537# v0.7.0 1538 1539This release has been tracked through the [0.7.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 1540 1541Highlights for `cloud-hypervisor` version 0.7.0 include: 1542 1543### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug 1544 1545Further to our effort to support modifying a running guest we now support 1546hotplug and unplug of the following virtio backed devices: block, network, 1547pmem, virtio-fs and vsock. This functionality is available on the (default) PCI 1548based transport and is exposed through the HTTP API. The `ch-remote` utility 1549provides a CLI for adding or removing these device types after the VM has 1550booted. User can use the `id` parameter on the devices to choose names for 1551devices to ease their removal. 1552 1553### Alternative `libc` Support 1554 1555Cloud Hypervisor can now be compiled with the `musl` C library and this release 1556contains a static binary compiled using that toolchain. 1557 1558### Multithreaded Multi Queued `vhost-user` Backends 1559 1560The `vhost-user` backends for network and block support that are shipped by 1561Cloud Hypervisor have been enhanced to support multiple threads and queues to 1562improve throughput. These backends are used automatically if `vhost_user=true` 1563is passed when the devices are created. 1564 1565### Initial RamFS Support 1566 1567By passing the `--initramfs` command line option the user can specify a file to 1568be loaded into the guest memory to be used as the kernel initial filesystem. 1569This is usually used to allow the loading of drivers needed to be able to 1570access the real root filesystem but it can also be used standalone for a very 1571minimal image. 1572 1573### Alternative Memory Hotplug: `virtio-mem` 1574 1575As well as supporting ACPI based hotplug Cloud Hypervisor now supports using 1576the `virtio-mem` hotplug alternative. This can be controlled by the 1577`hotplug_method` parameter on the `--memory` command line option. It currently 1578requires kernel patches to be able to support it. 1579 1580### `Seccomp` Sandboxing 1581 1582Cloud Hypervisor now has support for restricting the system calls that the 1583process can use via the `seccomp` security API. This on by default and is 1584controlled by the `--seccomp` command line option. 1585 1586### Updated Distribution Support 1587 1588With the release of Ubuntu 20.04 we have added that to the list of supported 1589distributions and is part of our regular testing programme. 1590 1591### Command Line and API Changes 1592 1593This is non exhaustive list of HTTP API and command line changes 1594 1595* New `id` fields added for devices to allow them to be named to ease removal. 1596 If no name is specified the VMM chooses one. 1597* Use `--memory`'s `shared` and `hugepages` controls for determining backing 1598 memory instead of providing a path. 1599* The `--vsock` parameter only takes one device as the Linux kernel only 1600 supports a single Vsock device. The REST API has removed the vector for this 1601 option and replaced it with a single optional field. 1602* There is enhanced validation of the command line and API provided 1603 configurations to ensure that the provided options are compatible e.g. that 1604 shared memory is in use if any attempt is made to used a `vhost-user` backed 1605 device. 1606* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and 1607 `add-vsock` subcommands. For removal `remove-device` is used. The REST API 1608 has appropriate new HTTP endpoints too. 1609* Specifying a `size` with `--pmem` is no longer required and instead the size 1610 will be obtained from the file. A `discard_writes` option has also been added 1611 to provide the equivalent of a read-only file. 1612* The parameters to `--block-backend` have been changed to more closely align 1613 with those used by `--disk`. 1614 1615### Contributors 1616 1617Many thanks to everyone who has contributed to our 0.7.0 release including some new faces. 1618 1619* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 1620* Bo Chen <chen.bo@intel.com> 1621* Cathy Zhang <cathy.zhang@intel.com> 1622* Damjan Georgievski <gdamjan@gmail.com> 1623* Dean Sheather <dean@coder.com> 1624* Eryu Guan <eguan@linux.alibaba.com> 1625* Hui Zhu <teawater@antfin.com> 1626* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1627* Martin Xu <martin.xu@intel.com> 1628* Muminul Islam <muislam@microsoft.com> 1629* Rob Bradford <robert.bradford@intel.com> 1630* Samuel Ortiz <sameo@linux.intel.com> 1631* Sebastien Boeuf <sebastien.boeuf@intel.com> 1632* Sergio Lopez <slp@redhat.com> 1633* Yang Zhong <yang.zhong@intel.com> 1634* Yi Sun <yi.y.sun@linux.intel.com> 1635 1636# v0.6.0 1637 1638This release has been tracked through the [0.6.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 1639 1640Highlights for `cloud-hypervisor` version 0.6.0 include: 1641 1642### Directly Assigned Devices Hotplug 1643 1644We continued our efforts around supporting dynamically changing the guest 1645resources. After adding support for CPU and memory hotplug, Cloud Hypervisor 1646now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`) 1647devices into an already running guest. This closes the features gap for 1648providing a complete Kata Containers workloads support with Cloud Hypervisor. 1649 1650### Shared Filesystem Improvements 1651 1652We enhanced our shared filesystem support through many `virtio-fs` improvements. 1653By adding support for DAX, parallel processing of multiple requests, `FS_IO`, 1654`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we 1655improved our filesystem sharing performance, but also made it more stable and 1656compatible with other `virtio-fs` implementations. 1657 1658### Block and Networking IO Self Offloading 1659 1660When choosing to offload the paravirtualized block and networking I/O to an 1661external process (through the `vhost-user` protocol), Cloud Hypervisor now 1662automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends 1663into their own, separate processes. 1664This provides a seamless paravirtualized I/O user experience for those who want 1665to run their guest I/O into separate executions contexts. 1666 1667### Command Line Interface 1668 1669More and more Cloud Hypervisor services are exposed through the 1670[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only 1671accessible via relatively cumbersome HTTP calls. In order to abstract 1672those calls into a more user friendly tool, we created a Cloud Hypervisor 1673Command Line Interface (CLI) called `ch-remote`. The `ch-remote` binary 1674is created with each build and available e.g. at 1675`cloud-hypervisor/target/debug/ch-remote` when doing a debug build. 1676 1677Please check `ch-remote --help` for a complete description of all available 1678commands. 1679 1680### PVH Boot 1681 1682In addition to the traditional Linux boot protocol, Cloud Hypervisor now 1683supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html). 1684 1685### Contributors 1686 1687With the 0.6.0 release, we are welcoming a few new contributors. Many thanks 1688to them and to everyone that contributed to this release: 1689 1690* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 1691* Arron Wang <arron.wang@intel.com> 1692* Bin Liu <liubin0329@gmail.com> 1693* Bo Chen <chen.bo@intel.com> 1694* Cathy Zhang <cathy.zhang@intel.com> 1695* Eryu Guan <eguan@linux.alibaba.com> 1696* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1697* Liu Bo <bo.liu@linux.alibaba.com> 1698* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1699* Rob Bradford <robert.bradford@intel.com> 1700* Samuel Ortiz <sameo@linux.intel.com> 1701* Sebastien Boeuf <sebastien.boeuf@intel.com> 1702* Sergio Lopez <slp@redhat.com> 1703 1704# v0.5.1 1705 1706This is a bugfix release branched off v0.5.0. It contains the following fixes: 1707 1708* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez 1709* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf 1710* Fixed error reporting for resize command - Samuel Ortiz 1711* Double reboot workaround (#783) - Rob Bradford 1712* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford 1713 1714# v0.5.0 1715 1716This release has been tracked through the [0.5.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/6). 1717 1718Highlights for `cloud-hypervisor` version 0.5.0 include: 1719 1720### Virtual Machine Dynamic Resizing 1721 1722With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and 1723memory hot plug as well. This allows to dynamically resize Cloud Hypervisor 1724guests which is needed for e.g. Kubernetes related use cases. 1725The memory hot plug implementation is based on the same framework as the CPU hot 1726plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest. 1727 1728Next on our VM resizing roadmap is the PCI devices hotplug feature. 1729 1730### Multi-Queue, Multi-Threaded Paravirtualization 1731 1732We enhanced our virtio networking and block support by having both devices use 1733multiple I/O queues handled by multiple threads. This improves our default 1734paravirtualized networking and block devices throughput. 1735 1736### New Interrupt Management Framework 1737 1738We improved our interrupt management implementation by introducing an Interrupt 1739Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device) 1740crates discussions. This move made the code significantly cleaner, and allowed 1741us to remove several KVM related dependencies from crates like the PCI and 1742virtio ones. 1743 1744### Development Tools 1745 1746In order to provide a better developer experience, we worked on improving our 1747build, development and testing tools. 1748Somehow similar to the excellent 1749[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool), 1750we now provide a [dev_cli script](scripts/dev_cli.sh). 1751 1752With this new tool, our users and contributors will be able to build and test 1753Cloud Hypervisor through a containerized environment. 1754 1755### Kata Containers Integration 1756 1757We spent some significant time and efforts debugging and fixing our integration 1758with the [Kata Containers](https://github.com/kata-containers) project. Cloud 1759Hypervisor is now a fully supported Kata Containers hypervisor, and is 1760integrated into the project's CI. 1761 1762### Contributors 1763 1764Many thanks to everyone that contributed to the 0.5.0 release: 1765 1766* Bo Chen <chen.bo@intel.com> 1767* Cathy Zhang <cathy.zhang@intel.com> 1768* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1769* Rob Bradford <robert.bradford@intel.com> 1770* Samuel Ortiz <sameo@linux.intel.com> 1771* Sebastien Boeuf <sebastien.boeuf@intel.com> 1772* Sergio Lopez <slp@redhat.com> 1773* Yang Zhong <yang.zhong@intel.com> 1774 1775# v0.4.0 1776 1777This release has been tracked through the [0.4.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/4). 1778 1779Highlights for `cloud-hypervisor` version 0.4.0 include: 1780 1781### Dynamic virtual CPUs addition 1782 1783As a way to vertically scale Cloud Hypervisor guests, we now support dynamically 1784adding virtual CPUs to the guests, a mechanism also known as CPU hot plug. 1785Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs 1786to an already running guest and the high level operations for that process are 1787documented [here](docs/hotplug.md) 1788 1789During the next release cycles we are planning to extend Cloud Hypervisor 1790hot plug framework to other resources, namely PCI devices and memory. 1791 1792### Programmatic firmware tables generation 1793 1794As part of the CPU hot plug feature enablement, and as a requirement for hot 1795plugging other resources like devices or RAM, we added support for 1796programmatically generating the needed ACPI tables. Through a dedicated 1797`acpi-tables` crate, we now have a flexible and clean way of generating those 1798tables based on the VMM device model and topology. 1799 1800### Filesystem and block devices vhost-user backends 1801 1802Our objective of running all Cloud Hypervisor paravirtualized I/O to a 1803vhost-user based framework is getting closer as we've added Rust based 1804implementations for vhost-user-blk and virtiofs backends. Together with the 1805vhost-user-net backend that came with the 0.3.0 release, this will form the 1806default Cloud Hypervisor I/O architecture. 1807 1808### Guest pause and resume 1809 1810As an initial requirement for enabling live migration, we added support for 1811pausing and resuming any VMM components. As an intermediate step towards live 1812migration, the upcoming guest snapshotting feature will be based on the pause 1813and resume capabilities. 1814 1815### Userspace IOAPIC by default 1816 1817As a way to simplify our device manager implementation, but also in order to 1818stay away from privileged rings as often as possible, any device that relies on 1819pin based interrupts will be using the userspace IOAPIC implementation by 1820default. 1821 1822### PCI BAR reprogramming 1823 1824In order to allow for a more flexible device model, and also support guests 1825that would want to move PCI devices, we added support for PCI devices BAR 1826reprogramming. 1827 1828### New `cloud-hypervisor` organization 1829 1830As we wanted to be more flexible on how we manage the Cloud Hypervisor project, 1831we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor). 1832Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) 1833project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux) 1834and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1835repositories. We may also use it to host any rust-vmm that we'd need to 1836temporarily fork. 1837Thanks to GitHub's seamless repository redirections, the move is completely 1838transparent to all Cloud Hypervisor contributors, users and followers. 1839 1840### Contributors 1841 1842Many thanks to everyone that contributed to the 0.4.0 release: 1843 1844* Cathy Zhang <cathy.zhang@intel.com> 1845* Emin Ghuliev <drmint80@gmail.com> 1846* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1847* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1848* Rob Bradford <robert.bradford@intel.com> 1849* Samuel Ortiz <sameo@linux.intel.com> 1850* Sebastien Boeuf <sebastien.boeuf@intel.com> 1851* Sergio Lopez <slp@redhat.com> 1852* Wu Zongyong <wuzongyong@linux.alibaba.com> 1853 1854# v0.3.0 1855 1856This release has been tracked through the [0.3.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/3). 1857 1858Highlights for `cloud-hypervisor` version 0.3.0 include: 1859 1860### Block device offloading 1861 1862We continue to work on offloading paravirtualized I/O to external processes, 1863and we added support for 1864[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends. 1865This enables `cloud-hypervisor` users to plug a `vhost-user` based block device 1866like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage 1867backend. 1868 1869### Network device backend 1870 1871The previous release provided support for 1872[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we 1873also provide a TAP based vhost-user-net backend, implemented in Rust. Together 1874with the vhost-user-net device implementation, this will eventually become the 1875Cloud Hypervisor default paravirtualized networking architecture. 1876 1877### Virtual sockets 1878 1879In order to more efficiently and securely communicate between host and guest, 1880we added an hybrid implementation of the 1881[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address 1882family over virtio. Credits go to the 1883[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md) 1884project as our implementation is a copy of theirs. 1885 1886### HTTP based API 1887 1888In anticipation of the need to support asynchronous operations to Cloud 1889Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP 1890based API to the VMM. The API will be more extensively documented during the 1891next release cycle. 1892 1893### Memory mapped virtio transport 1894 1895In order to support potential PCI-free use cases, we added support for the 1896[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002) 1897transport layer. This will allow us to support simple, minimal guest 1898configurations that do not require a PCI bus emulation. 1899 1900### Paravirtualized IOMMU 1901 1902As we want to improve our nested guests support, we added support for exposing 1903a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows 1904for a safer nested virtio and directly assigned devices support. 1905 1906To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor 1907users to be able to specify if devices had to be handled through this virtual 1908IOMMU or not. In particular, the `--disk` option now expects disk paths to be 1909prefixed with a `path=` string, and supports an optional `iommu=[on|off]` 1910setting. 1911 1912### Ubuntu 19.10 1913 1914With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware), 1915we can now support the latest 1916[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images. 1917 1918### Large memory guests 1919 1920After simplifying and changing our guest address space handling, we can now 1921support guests with large amount of memory (more than 64GB). 1922 1923# v0.2.0 1924 1925This release has been tracked through the [0.2.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/2). 1926 1927Highlights for `cloud-hypervisor` version 0.2.0 include: 1928 1929### Network device offloading 1930 1931As part of our general effort to offload paravirtualized I/O to external 1932processes, we added support for 1933[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This 1934enables `cloud-hypervisor` users to plug a `vhost-user` based networking device 1935(e.g. [DPDK](https://dpdk.org)) into the VMM as their virtio network backend. 1936 1937### Minimal hardware-reduced ACPI 1938 1939In order to properly implement and guest reset and shutdown, we implemented 1940a minimal version of the hardware-reduced ACPI specification. Together with 1941a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to 1942cleanly reboot and shutdown. 1943 1944The ACPI implementation is a `cloud-hypervisor` build time option that is 1945enabled by default. 1946 1947### Debug I/O port 1948 1949Based on the Firecracker idea of using a dedicated I/O port to measure guest 1950boot times, we added support for logging guest events through the 1951[0x80](https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html) 1952PC debug port. This allows, among other things, for granular guest boot time 1953measurements. See our [debug port documentation](docs/debug-port.md) for more 1954details. 1955 1956### Improved direct device assignment 1957 1958We fixed a major performance issue with our initial VFIO implementation: When 1959enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads 1960were (in many cases) not cached. After correctly tagging the guest memory from 1961`cloud-hypervisor` we're now able to reach the expected performance from 1962directly assigned devices. 1963 1964### Improved shared filesystem 1965 1966We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt) 1967support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system. 1968This provides better shared filesystem IO performance with a smaller guest 1969memory footprint. 1970 1971### Ubuntu bionic based CI 1972 1973Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1974improvements, we are now able to boot Ubuntu bionic images. We added those to 1975our CI pipeline. 1976 1977# v0.1.0 1978 1979This release has been tracked through the [0.1.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/1). 1980 1981Highlights for `cloud-hypervisor` version 0.1.0 include: 1982 1983### Shared filesystem 1984 1985We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file 1986system, allowing for an efficient and reliable way of sharing a filesystem 1987between the host and the `cloud-hypervisor` guest. 1988 1989See our [filesystem sharing](docs/fs.md) documentation for more details on how 1990to use virtio-fs with `cloud-hypervisor`. 1991 1992### Initial direct device assignment support 1993 1994VFIO (Virtual Function I/O) is a kernel framework that exposes direct device 1995access to userspace. `cloud-hypervisor` uses VFIO to directly assign host 1996physical devices into its guest. 1997 1998See our [VFIO](docs/vfio.md) documentation for more detail on how to directly 1999assign host devices to `cloud-hypervisor` guests. 2000 2001### Userspace IOAPIC 2002 2003`cloud-hypervisor` supports a so-called split IRQ chip implementation by 2004implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC). 2005By moving part of the IRQ chip implementation from kernel space to user space, 2006the IRQ chip emulation does not always run in a fully privileged mode. 2007 2008### Virtual persistent memory 2009 2010The `virtio-pmem` implementation emulates a virtual persistent memory device 2011that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device 2012allows to bypass the guest page cache and improve the guest memory footprint. 2013 2014### Linux kernel bzImage 2015 2016The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from 2017`bzImage` kernel images, which is usually the format that Linux distributions 2018use to ship their kernels. For example, this allows for booting from the host 2019distribution kernel image. 2020 2021### Console over virtio 2022 2023`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although 2024using this device as a guest console can potentially cut some early boot 2025messages, it can reduce the guest boot time and provides a complete console 2026implementation. 2027 2028The `virtio-console` device is enabled by default for the guest console. 2029Switching back to the legacy serial port is done by selecting 2030`--serial tty --console off` from the command line. 2031 2032### Unit testing 2033 2034We now run all unit tests from all our crates directly from our CI. 2035 2036### Integration tests parallelization 2037 2038The CI cycle run time has been significantly reduced by refactoring our 2039integration tests; allowing them to all be run in parallel. 2040