1- [v42.0](#v420) 2 - [SVE/SVE2 Support on AArch64](#svesve2-support-on-aarch64) 3 - [Notable Bug Fixes](#notable-bug-fixes) 4 - [Sponsorships](#sponsorships) 5 - [Contributors](#contributors) 6- [v41.0](#v410) 7 - [Experimental "Pvmemcontrol" Support](#experimental-pvmemcontrol-support) 8 - [Sandboxing With Landlock Support](#sandboxing-with-landlock-support) 9 - [Notable Performance Improvements](#notable-performance-improvements) 10 - [Notable Bug Fixes](#notable-bug-fixes-1) 11 - [Contributors](#contributors-1) 12- [v40.0](#v400) 13 - [Support for Restoring File Descriptor Backed Network Devices](#support-for-restoring-file-descriptor-backed-network-devices) 14 - [Notable Bug Fixes](#notable-bug-fixes-2) 15 - [Contributors](#contributors-2) 16- [v39.0](#v390) 17 - [Variable Sizing of PCI Apertures for Segments](#variable-sizing-of-pci-apertures-for-segments) 18 - [Direct Booting with bzImages](#direct-booting-with-bzimages) 19 - [Support for NVIDIA GPUDirect P2P Support](#support-for-nvidia-gpudirect-p2p-support) 20 - [Guest NMI Injection Support](#guest-nmi-injection-support) 21 - [Notable Bug Fixes](#notable-bug-fixes-3) 22 - [Contributors](#contributors-3) 23- [v38.0](#v380) 24 - [Group Rate Limiter on Block Devices](#group-rate-limiter-on-block-devices) 25 - [CPU Pinning Support for Block Device Worker Thread](#cpu-pinning-support-for-block-device-worker-thread) 26 - [Optimized Boot Time with Parallel Memory Prefault](#optimized-boot-time-with-parallel-memory-prefault) 27 - [New 'debug-console' Device](#new-debug-console-device) 28 - [Improved VFIO Device Support](#improved-vfio-device-support) 29 - [Extended CPU Affinity Support](#extended-cpu-affinity-support) 30 - [Notable Bug Fixes](#notable-bug-fixes-4) 31 - [Contributors](#contributors-4) 32- [v37.0](#v370) 33 - [Long Term Support (LTS) Release](#long-term-support-lts-release) 34 - [Multiple PCI segments Support for 32-bit VFIO devices](#multiple-pci-segments-support-for-32-bit-vfio-devices) 35 - [Configurable Named TAP Devices](#configurable-named-tap-devices) 36 - [TTY Output from Both Serial Device and Virtio Console](#tty-output-from-both-serial-device-and-virtio-console) 37 - [Faster VM Restoration from Snapshots](#faster-vm-restoration-from-snapshots) 38 - [Notable Bug Fixes](#notable-bug-fixes-5) 39 - [Contributors](#contributors-5) 40- [v36.0](#v360) 41 - [Command Line Changes](#command-line-changes) 42 - [Enabled Features Reported via API Endpoint and CLI](#enabled-features-reported-via-api-endpoint-and-cli) 43 - [NUMA Support for PCI segments](#numa-support-for-pci-segments) 44 - [CPU Topology Support on AMD Platforms](#cpu-topology-support-on-amd-platforms) 45 - [Unix Socket Backend for Serial Port](#unix-socket-backend-for-serial-port) 46 - [AIO Backend for Block Devices](#aio-backend-for-block-devices) 47 - [Documentation Improvements](#documentation-improvements) 48 - [Notable Bug Fixes](#notable-bug-fixes-6) 49 - [Contributors](#contributors-6) 50- [v35.0](#v350) 51 - [`virtio-vsock` Support for Linux Guest Kernel v6.3+](#virtio-vsock-support-for-linux-guest-kernel-v63) 52 - [User Specified Serial Number for `virtio-block`](#user-specified-serial-number-for-virtio-block) 53 - [vCPU TSC Frequency Included in Migration State](#vcpu-tsc-frequency-included-in-migration-state) 54 - [Notable Bug Fixes](#notable-bug-fixes-7) 55 - [Contributors](#contributors-7) 56- [v34.0](#v340) 57 - [Paravirtualised Panic Device Support](#paravirtualised-panic-device-support) 58 - [Improvements to VM Core Dump](#improvements-to-vm-core-dump) 59 - [QCOW2 Support for Backing Files](#qcow2-support-for-backing-files) 60 - [Minimum Host Kernel Bump](#minimum-host-kernel-bump) 61 - [Notable Bug Fixes](#notable-bug-fixes-8) 62 - [Contributors](#contributors-8) 63- [v33.0](#v330) 64 - [D-Bus based API](#d-bus-based-api) 65 - [Expose Host CPU Cache Details for AArch64](#expose-host-cpu-cache-details-for-aarch64) 66 - [Notable Bug Fixes](#notable-bug-fixes-9) 67 - [Contributors](#contributors-9) 68- [v32.0](#v320) 69 - [Increased PCI Segment Limit](#increased-pci-segment-limit) 70 - [API Changes](#api-changes) 71 - [Notable Bug Fixes](#notable-bug-fixes-10) 72 - [Contributors](#contributors-10) 73- [v31.1](#v311) 74- [v31.0](#v310) 75 - [Update to Latest `acpi_tables`](#update-to-latest-acpi_tables) 76 - [Update Reference Kernel to 6.2](#update-reference-kernel-to-62) 77 - [Improvements on Console `SIGWINCH` Handler](#improvements-on-console-sigwinch-handler) 78 - [Remove Directory Support from `MemoryZoneConfig::file`](#remove-directory-support-from-memoryzoneconfigfile) 79 - [Documentation Improvements](#documentation-improvements-1) 80 - [Notable Bug Fixes](#notable-bug-fixes-11) 81 - [Contributors](#contributors-11) 82- [v30.0](#v300) 83 - [Command Line Changes for Reduced Binary Size](#command-line-changes-for-reduced-binary-size) 84 - [Basic vfio-user Server Support](#basic-vfio-user-server-support) 85 - [Heap Profiling Support](#heap-profiling-support) 86 - [Documentation Improvements](#documentation-improvements-2) 87 - [Notable Bug Fixes](#notable-bug-fixes-12) 88 - [Contributors](#contributors-12) 89- [v28.2](#v282) 90- [v29.0](#v290) 91 - [Release Binary Supports Both MSHV and KVM](#release-binary-supports-both-mshv-and-kvm) 92 - [Snapshot/Restore and Live Migration Improvements](#snapshotrestore-and-live-migration-improvements) 93 - [Heap Allocation Improvements](#heap-allocation-improvements) 94 - [`ch-remote` Improvements](#ch-remote-improvements) 95 - [`AArch64` Documentation Integration](#aarch64-documentation-integration) 96 - [`virtio-block` Counters Enhancement](#virtio-block-counters-enhancement) 97 - [TCP Offload Control](#tcp-offload-control) 98 - [Notable Bug Fixes](#notable-bug-fixes-13) 99 - [Removals](#removals) 100 - [Deprecations](#deprecations) 101 - [Contributors](#contributors-13) 102- [v28.1](#v281) 103- [v28.0](#v280) 104 - [Community Engagement (Reminder)](#community-engagement-reminder) 105 - [Long Term Support (LTS) Release](#long-term-support-lts-release-1) 106 - [Virtualised TPM Support](#virtualised-tpm-support) 107 - [Transparent Huge Page Support](#transparent-huge-page-support) 108 - [README Quick Start Improved](#readme-quick-start-improved) 109 - [Notable Bug Fixes](#notable-bug-fixes-14) 110 - [Removals](#removals-1) 111 - [Contributors](#contributors-14) 112- [v27.0](#v270) 113 - [Community Engagement](#community-engagement) 114 - [Prebuilt Packages](#prebuilt-packages) 115 - [Network Device MTU Exposed to Guest](#network-device-mtu-exposed-to-guest) 116 - [Boot Tracing](#boot-tracing) 117 - [Simplified Build Feature Flags](#simplified-build-feature-flags) 118 - [Asynchronous Kernel Loading](#asynchronous-kernel-loading) 119 - [GDB Support for AArch64](#gdb-support-for-aarch64) 120 - [Notable Bug Fixes](#notable-bug-fixes-15) 121 - [Deprecations](#deprecations-1) 122 - [Contributors](#contributors-15) 123- [v26.0](#v260) 124 - [SMBIOS Improvements via `--platform`](#smbios-improvements-via---platform) 125 - [Unified Binary MSHV and KVM Support](#unified-binary-mshv-and-kvm-support) 126 - [Notable Bug Fixes](#notable-bug-fixes-16) 127 - [Deprecations](#deprecations-2) 128 - [Removals](#removals-2) 129 - [Contributors](#contributors-16) 130- [v25.0](#v250) 131 - [`ch-remote` Improvements](#ch-remote-improvements-1) 132 - [VM "Coredump" Support](#vm-coredump-support) 133 - [Notable Bug Fixes](#notable-bug-fixes-17) 134 - [Removals](#removals-3) 135 - [Contributors](#contributors-17) 136- [v24.0](#v240) 137 - [Bypass Mode for `virtio-iommu`](#bypass-mode-for-virtio-iommu) 138 - [Ensure Identifiers Uniqueness](#ensure-identifiers-uniqueness) 139 - [Sparse Mmap support](#sparse-mmap-support) 140 - [Expose Platform Serial Number](#expose-platform-serial-number) 141 - [Notable Bug Fixes](#notable-bug-fixes-18) 142 - [Notable Improvements](#notable-improvements) 143 - [Deprecations](#deprecations-3) 144 - [New on the Website](#new-on-the-website) 145 - [Contributors](#contributors-18) 146- [v23.1](#v231) 147- [v23.0](#v230) 148 - [vDPA Support](#vdpa-support) 149 - [Updated OS Support list](#updated-os-support-list) 150 - [`AArch64` Memory Map Improvements](#aarch64-memory-map-improvements) 151 - [`AMX` Support](#amx-support) 152 - [Notable Bug Fixes](#notable-bug-fixes-19) 153 - [Deprecations](#deprecations-4) 154 - [Contributors](#contributors-19) 155- [v22.1](#v221) 156- [v22.0](#v220) 157 - [GDB Debug Stub Support](#gdb-debug-stub-support) 158 - [`virtio-iommu` Backed Segments](#virtio-iommu-backed-segments) 159 - [Before Boot Configuration Changes](#before-boot-configuration-changes) 160 - [`virtio-balloon` Free Page Reporting](#virtio-balloon-free-page-reporting) 161 - [Support for Direct Kernel Booting with TDX](#support-for-direct-kernel-booting-with-tdx) 162 - [PMU Support for AArch64](#pmu-support-for-aarch64) 163 - [Documentation Under CC-BY-4.0 License](#documentation-under-cc-by-40-license) 164 - [Deprecation of "Classic" `virtiofsd`](#deprecation-of-classic-virtiofsd) 165 - [Notable Bug Fixes](#notable-bug-fixes-20) 166 - [Contributors](#contributors-20) 167- [v21.0](#v210) 168 - [Efficient Local Live Migration (for Live Upgrade)](#efficient-local-live-migration-for-live-upgrade) 169 - [Recommended Kernel is Now 5.15](#recommended-kernel-is-now-515) 170 - [Notable Bug fixes](#notable-bug-fixes-21) 171 - [Contributors](#contributors-21) 172- [v20.2](#v202) 173- [v20.1](#v201) 174- [v20.0](#v200) 175 - [Multiple PCI segments support](#multiple-pci-segments-support) 176 - [CPU pinning](#cpu-pinning) 177 - [Improved VFIO support](#improved-vfio-support) 178 - [Safer code](#safer-code) 179 - [Extended documentation](#extended-documentation) 180 - [Notable bug fixes](#notable-bug-fixes-22) 181 - [Contributors](#contributors-22) 182- [v19.0](#v190) 183 - [Improved PTY handling for serial and `virtio-console`](#improved-pty-handling-for-serial-and-virtio-console) 184 - [PCI boot time optimisations](#pci-boot-time-optimisations) 185 - [Improved TDX support](#improved-tdx-support) 186 - [Live migration enhancements](#live-migration-enhancements) 187 - [`virtio-mem` support with `vfio-user`](#virtio-mem-support-with-vfio-user) 188 - [AArch64 for `virtio-iommu`](#aarch64-for-virtio-iommu) 189 - [Notable bug fixes](#notable-bug-fixes-23) 190 - [Contributors](#contributors-23) 191- [v18.0](#v180) 192 - [Experimental User Device (`vfio-user`) support](#experimental-user-device-vfio-user-support) 193 - [Migration support for `vhost-user` devices](#migration-support-for-vhost-user-devices) 194 - [VHDX disk image support](#vhdx-disk-image-support) 195 - [Device pass through on MSHV hypervisor](#device-pass-through-on-mshv-hypervisor) 196 - [AArch64 for support `virtio-mem`](#aarch64-for-support-virtio-mem) 197 - [Live migration on MSHV hypervisor](#live-migration-on-mshv-hypervisor) 198 - [AArch64 CPU topology support](#aarch64-cpu-topology-support) 199 - [Power button support on AArch64](#power-button-support-on-aarch64) 200 - [Notable bug fixes](#notable-bug-fixes-24) 201 - [Contributors](#contributors-24) 202- [v17.0](#v170) 203 - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi) 204 - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend) 205 - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices) 206 - [Improved SGX support](#improved-sgx-support) 207 - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices) 208 - [Notable bug fixes](#notable-bug-fixes-25) 209 - [Contributors](#contributors-25) 210- [v16.0](#v160) 211 - [Improved live migration support](#improved-live-migration-support) 212 - [Improved `vhost-user` support](#improved-vhost-user-support) 213 - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support) 214 - [Notable bug fixes](#notable-bug-fixes-26) 215 - [Removed functionality](#removed-functionality) 216 - [Contributors](#contributors-26) 217- [v15.0](#v150) 218 - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees) 219 - [Network device rate limiting](#network-device-rate-limiting) 220 - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload) 221 - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter) 222 - [Bug fixes](#bug-fixes) 223 - [Deprecations](#deprecations-5) 224 - [Contributors](#contributors-27) 225- [v0.14.1](#v0141) 226- [v0.14.0](#v0140) 227 - [Structured event monitoring](#structured-event-monitoring) 228 - [MSHV improvements](#mshv-improvements) 229 - [Improved aarch64 platform](#improved-aarch64-platform) 230 - [Updated hotplug documentation](#updated-hotplug-documentation) 231 - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console) 232 - [Block device rate limiting](#block-device-rate-limiting) 233 - [Deprecations](#deprecations-6) 234 - [Contributors](#contributors-28) 235- [v0.13.0](#v0130) 236 - [Wider VFIO device support](#wider-vfio-device-support) 237 - [Improved huge page support](#improved-huge-page-support) 238 - [MACvTAP support](#macvtap-support) 239 - [VHD disk image support](#vhd-disk-image-support) 240 - [Improved Virtio device threading](#improved-virtio-device-threading) 241 - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button) 242 - [Contributors](#contributors-29) 243- [v0.12.0](#v0120) 244 - [ARM64 enhancements](#arm64-enhancements) 245 - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning) 246 - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend) 247 - [Enhanced "info" API](#enhanced-info-api) 248 - [Contributors](#contributors-30) 249- [v0.11.0](#v0110) 250 - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block) 251 - [Windows Guest Support](#windows-guest-support) 252 - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation) 253 - [`virtio-mmio` Removal](#virtio-mmio-removal) 254 - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64) 255 - [Improved Linux Boot Time](#improved-linux-boot-time) 256 - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling) 257 - [Default Log Level Changed](#default-log-level-changed) 258 - [New `--balloon` Parameter Added](#new---balloon-parameter-added) 259 - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support) 260 - [Notable Bug Fixes](#notable-bug-fixes-27) 261 - [Contributors](#contributors-31) 262- [v0.10.0](#v0100) 263 - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors) 264 - [Memory Zones](#memory-zones) 265 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements) 266 - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control) 267 - [Notable Bug Fixes](#notable-bug-fixes-28) 268 - [Contributors](#contributors-32) 269- [v0.9.0](#v090) 270 - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support) 271 - [Block and Network Device Statistics](#block-and-network-device-statistics) 272 - [HTTP API Responses](#http-api-responses) 273 - [CPU Topology](#cpu-topology) 274 - [Release Build Optimization](#release-build-optimization) 275 - [Hypervisor Abstraction](#hypervisor-abstraction) 276 - [Snapshot/Restore Improvements](#snapshotrestore-improvements) 277 - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support) 278 - [Enhancements to ARM64 Support](#enhancements-to-arm64-support) 279 - [Intel SGX Support](#intel-sgx-support) 280 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1) 281 - [Notable Bug Fixes](#notable-bug-fixes-29) 282 - [Contributors](#contributors-33) 283- [v0.8.0](#v080) 284 - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support) 285 - [Experimental ARM64 Support](#experimental-arm64-support) 286 - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests) 287 - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices) 288 - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements) 289 - [Notable Bug Fixes](#notable-bug-fixes-30) 290 - [Command Line and API Changes](#command-line-and-api-changes) 291 - [Contributors](#contributors-34) 292- [v0.7.0](#v070) 293 - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug) 294 - [Alternative `libc` Support](#alternative-libc-support) 295 - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends) 296 - [Initial RamFS Support](#initial-ramfs-support) 297 - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem) 298 - [`Seccomp` Sandboxing](#seccomp-sandboxing) 299 - [Updated Distribution Support](#updated-distribution-support) 300 - [Command Line and API Changes](#command-line-and-api-changes-1) 301 - [Contributors](#contributors-35) 302- [v0.6.0](#v060) 303 - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug) 304 - [Shared Filesystem Improvements](#shared-filesystem-improvements) 305 - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading) 306 - [Command Line Interface](#command-line-interface) 307 - [PVH Boot](#pvh-boot) 308 - [Contributors](#contributors-36) 309- [v0.5.1](#v051) 310- [v0.5.0](#v050) 311 - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing) 312 - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization) 313 - [New Interrupt Management Framework](#new-interrupt-management-framework) 314 - [Development Tools](#development-tools) 315 - [Kata Containers Integration](#kata-containers-integration) 316 - [Contributors](#contributors-37) 317- [v0.4.0](#v040) 318 - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition) 319 - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation) 320 - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends) 321 - [Guest pause and resume](#guest-pause-and-resume) 322 - [Userspace IOAPIC by default](#userspace-ioapic-by-default) 323 - [PCI BAR reprogramming](#pci-bar-reprogramming) 324 - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization) 325 - [Contributors](#contributors-38) 326- [v0.3.0](#v030) 327 - [Block device offloading](#block-device-offloading) 328 - [Network device backend](#network-device-backend) 329 - [Virtual sockets](#virtual-sockets) 330 - [HTTP based API](#http-based-api) 331 - [Memory mapped virtio transport](#memory-mapped-virtio-transport) 332 - [Paravirtualized IOMMU](#paravirtualized-iommu) 333 - [Ubuntu 19.10](#ubuntu-1910) 334 - [Large memory guests](#large-memory-guests) 335- [v0.2.0](#v020) 336 - [Network device offloading](#network-device-offloading) 337 - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi) 338 - [Debug I/O port](#debug-io-port) 339 - [Improved direct device assignment](#improved-direct-device-assignment) 340 - [Improved shared filesystem](#improved-shared-filesystem) 341 - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci) 342- [v0.1.0](#v010) 343 - [Shared filesystem](#shared-filesystem) 344 - [Initial direct device assignment support](#initial-direct-device-assignment-support) 345 - [Userspace IOAPIC](#userspace-ioapic) 346 - [Virtual persistent memory](#virtual-persistent-memory) 347 - [Linux kernel bzImage](#linux-kernel-bzimage) 348 - [Console over virtio](#console-over-virtio) 349 - [Unit testing](#unit-testing) 350 - [Integration tests parallelization](#integration-tests-parallelization) 351 352# v42.0 353 354This release has been tracked in our [roadmap 355project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 356v42.0. The following user visible changes have been made: 357 358### SVE/SVE2 Support on AArch64 359 360The SVE and SVE2 feature bits are now propagated through to the guest on 361AArch64. (#6678, #6691) 362 363### Notable Bug Fixes 364 365* Reduce latency notification when rate limited (#6672) 366* Fix `virtio-console` resizing (#6704) 367* Fix resizing when console uses TTY (#6754) 368* Avoid deadlock in PCI BAR reprogramming that can occur when adding a new 369 `virtio` device to a VM that has been restored (#6775) 370* Fix console resizing after VM restore (#6748) 371* Fix memory resize error due to incorrect bounds checks (#6736) 372 373### Sponsorships 374 375During this release cycle a new VFIO CI worker has been provided by Crusoe 376Energy and a new ARM64 CI worker has been provided by Ubicloud. 377 378### Contributors 379 380Many thanks to everyone who has contributed to our release: 381 382* Alexandru Matei <alexandru.matei@uipath.com> 383* Alyssa Ross <hi@alyssa.is> 384* Anirudh Rayabharam <anrayabh@microsoft.com> 385* BharatNarasimman <bharatn@microsoft.com> 386* Bo Chen <chen.bo@intel.com> 387* Jinank Jain <jinankjain@microsoft.com> 388* Jonas Scholz <Jonas.Scholz@bbscholz.de> 389* Nuno Das Neves <nudasnev@microsoft.com> 390* Praveen K Paladugu <prapal@linux.microsoft.com> 391* Purna Pavan Chandra <paekkaladevi@microsoft.com> 392* Rob Bradford <rbradford@rivosinc.com> 393* Ruoqing He <heruoqing@iscas.ac.cn> 394* Songqian Li <sionli@tencent.com> 395* Tom Dohrmann <erbse.13@gmx.de> 396* Wei Liu <liuwe@microsoft.com> 397* Wenyu Huang <huangwenyuu@outlook.com> 398* Yuhong Zhong <yz@cs.columbia.edu> 399* wuxinyue <wuxinyue.wxy@antgroup.com> 400 401# v41.0 402 403This release has been tracked in our [roadmap 404project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 405v41.0. The following user visible changes have been made: 406 407### Experimental "Pvmemcontrol" Support 408 409VMM support has been added for this experimental functionality (requires 410currently out of tree Linux kernel patches) to allow guests to control its 411physical memory properties to allow optimisations and security features. 412(#6318, #6467) 413 414### Sandboxing With Landlock Support 415 416Support for restricting the VMM process using the Linux kernel "Landlock" API 417has been added - this can be used to restrict the files (and the read/write 418permissions) that the VMM process can access. This adds another layer of 419security alongside the existing sycall filters (`seccomp`) - this can be 420enabled with `--landlock` and [fully documentated](docs/landlock.md). (#5170) 421 422### Notable Performance Improvements 423 424* Reduced heap allocations in `virtio-net` via the use of a cache of `Iovec` 425 structures (#6636) 426* Notification suppression ("`EVENT_IDX`") support has been added to 427 `virtio-block` giving a 60% improvement in single queue block throughput and 428 IOPs performance (#6580) 429* Correct size used for `status` field in `virtio-block` state (#6586) 430 431### Notable Bug Fixes 432 433* Avoid panic on out-of-bounds PCI MSI-X access (#6657) 434* Fix undefined behaviour on AArch64 leading to wrong optimisation on KVM API 435 access (#6647) 436* Rust v1.80.0 added use of `fcntl` syscall on debug assertions so this is now 437 included in the virtio-device seccomp filters for tests that use this (#6648) 438* Short reads are now handled correctly in the `virtio-vsock` device (#6621) 439* Fix undefined behaviour on TTY ioctl leading to wrong optimisation (#6568) 440 441### Contributors 442 443Many thanks to everyone who has contributed to our release: 444 445* Alyssa Ross <hi@alyssa.is> 446* Bo Chen <chen.bo@intel.com> 447* Changyuan Lyu <changyuanl@google.com> 448* Jinank Jain <jinankjain@microsoft.com> 449* Julian Stecklina <julian.stecklina@cyberus-technology.de> 450* Muminul Islam <muislam@microsoft.com> 451* Nuno Das Neves <nudasnev@microsoft.com> 452* Praveen K Paladugu <prapal@linux.microsoft.com> 453* Rob Bradford <rbradford@rivosinc.com> 454* Songqian Li <sionli@tencent.com> 455* Wei Liu <liuwe@microsoft.com> 456* Yuanchu Xie <yuanchu@google.com> 457* ihciah <ihciah@gmail.com> 458* wuxinyue <wuxinyue.wxy@antgroup.com> 459 460# v40.0 461 462This release has been tracked in our [roadmap 463project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 464v40.0. The following user visible changes have been made: 465 466### Support for Restoring File Descriptor Backed Network Devices 467 468It is now possible to pass file descriptors over the HTTP API (and using 469`ch-remote`) when restoring to update the file descriptors for network devices. 470This enables snapshot & restore functionality for guests using `macvtap` or 471other file descriptor backed network devices. (#6286) 472 473### Notable Bug Fixes 474 475* Default values have been removed from required fields in the OpenAPI metadata 476 (#6495) 477* The help syntax of `ch-remote remove-device` has been improved (#6456) 478* A double close of file descriptors has been fixed when using `--serial` 479 (#6486) 480* To prevent loops a limit on the nesting level for QCOW2 backing files has 481 been introduced (#6482) 482* Boot time performance has been improved with multiple cores by avoiding 483 `cpuid` instructions and by seeding the in kernel file descriptor table 484 (#6498, #6478) 485* L1 cache details are more likely to be propagated into the guest (#6523) 486* The default topology for guests now uses multiple cores rather than sockets 487 (#6504) 488 489### Contributors 490 491Many thanks to everyone who has contributed to our release: 492 493* Alexandru Matei <alexandru.matei@uipath.com> 494* Bo Chen <chen.bo@intel.com> 495* Jinank Jain <jinankjain@microsoft.com> 496* Josh Soref <2119212+jsoref@users.noreply.github.com> 497* Muminul Islam <muislam@microsoft.com> 498* Nuno Das Neves <nudasnev@microsoft.com> 499* Omer Faruk Bayram <omer.faruk@sartura.hr> 500* Praveen K Paladugu <prapal@linux.microsoft.com> 501* Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> 502* Rob Bradford <rbradford@rivosinc.com> 503* SamrutGadde <samrut.gadde@gmail.com> 504* Sean Banko <sbanko@crusoe.ai> 505* Songqian Li <sionli@tencent.com> 506* Wei Liu <liuwe@microsoft.com> 507* Yi Wang <foxywang@tencent.com> 508* Yu Li <liyu.yukiteru@bytedance.com> 509* 510 511# v39.0 512 513This release has been tracked in our [roadmap 514project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 515v39.0. The following user visible changes have been made: 516 517### Variable Sizing of PCI Apertures for Segments 518 519It is now possible to use `--pci-segment` to adjust the aperture size that 520devices 32-bit and 64-bit PCI device BARs will be allocated from. Previously 521the address space was equally distributed across all the segments which may 522leave insufficient space for devices that require a large 32-bit space. With 523this change the weighting per segment can be adjusted. (#6387) 524 525### Direct Booting with bzImages 526 527Support for directly booting Linux from bzImages has been added.(#6200) 528 529### Support for NVIDIA GPUDirect P2P Support 530 531The `x_nv_gpudirect_clique` option was added to `--device` to allow the 532configuration of device P2P support with NVIDIA GPUs. (#6235) 533 534### Guest NMI Injection Support 535 536A new API endpoint and `ch-remote` option added for injecting an NMI into the 537guest. (#6047) 538 539### Notable Bug Fixes 540 541* Workaround for kernel bug affecting guest IRQ masking on AMD (#6353) 542* Correctly cleanup `sigwinch_listener` process (#6208) 543* Graceful shutdown of HTTP API thread (#6248, #6247) 544* Fix `queue_affinity` option in OpenAPI metadata (#6268) 545* Fix documentation to indicate only stream mode is supported by `virtio-vsock` 546 (#6306) 547* Fix `virtio-fs` tag validation (#6358, #6359) 548* Add missing `pvpanic` device to OpenAPI metadata (#6372) 549* Fixes for nested virtualization with VFIO devices (#6110, #6298, #6297, 550 #6319) 551* Fix for backing file for `virtio-mem` regions with snapshot/restore (#6337, 552 #6338) 553* Explicitly mark FDs used for network devices as invalid across 554 snapshot/restore (#6332, #6286) 555* Improve `event-monitor` events around reboot (#6277, #6274) 556* Fix potential deadlock around paused devices during live migration (#6293) 557* Fix panic when running `ch-remote` with no subcommand (#6230) 558* Fix hotplug of `virtio` devices after snapshot/restore and live migration 559 (#6326, #6265) 560 561### Contributors 562 563Many thanks to everyone who has contributed to our release: 564 565* Alexandru Matei <alexandru.matei@uipath.com> 566* Andrew Carp <acarp@crusoeenergy.com> 567* Bo Chen <chen.bo@intel.com> 568* Bouke van der Bijl <i@bou.ke> 569* Chris Webb <chris@arachsys.com> 570* Jinank Jain <jinankjain@microsoft.com> 571* Lucas Jacques <contact@lucasjacques.com> 572* Muminul Islam <muislam@microsoft.com> 573* Nuno Das Neves <nudasnev@microsoft.com> 574* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 575* Rob Bradford <rbradford@rivosinc.com> 576* Ruslan Mstoi <ruslan.mstoi@intel.com> 577* Stefan Nuernberger <stefan.nuernberger@cyberus-technology.de> 578* Thomas Barrett <tbarrett@crusoeenergy.com> 579* Wei Liu <liuwe@microsoft.com> 580* Yi Wang <foxywang@tencent.com> 581 582# v38.0 583 584This release has been tracked in our [roadmap 585project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 586v38.0. The following user visible changes have been made: 587 588### Group Rate Limiter on Block Devices 589 590Users now can throttle a group of block devices with the new 591`--rate-limiter-group` option. Details can be found from the [I/O 592Throttling documentation](docs/io_throttling.md) 593 594### CPU Pinning Support for Block Device Worker Thread 595 596Users now have the option to pin virt-queue threads for block devices 597to specific host cpus. 598 599### Optimized Boot Time with Parallel Memory Prefault 600 601The boot time with `prefault` option enabled is optimized via parallel 602memory prefault. 603 604### New 'debug-console' Device 605 606A 'debug-console' device is added to provide a user-configurable debug 607port for logging guest information. Details can be found from the [Debug 608IO Ports documentation](docs/debug-port.md). 609 610### Improved VFIO Device Support 611 612All non-emulated MMIO regions of VFIO devices are now mapped to the VFIO 613container, allowing PCIe P2P between all VFIO devices on the same 614VM. This is required for a wide variety of multi-GPU workloads involving 615GPUDirect P2P (DMA between two GPUs), GPUDirect RDMA (DMA between a GPU 616and an IB device). 617 618### Extended CPU Affinity Support 619 620Users now can set the vcpu affinity to a host CPU with index larger 621than 255. 622 623### Notable Bug Fixes 624 625* Enable HTT flag to avoid crashing cpu topology enumeration software 626such as hwloc in the guest (#6146) 627* Fix several security advisories from dependencies (#6134, #6141) 628* Handle non-power-of-two CPU topology properly (#6062) 629* Various bug fixes around `virtio-vsock`(#6080, #6091, #6095) 630* Enable nested virtualization on AMD if supported (#6106) 631* Align VFIO devices PCI BARs naturally (#6196) 632 633### Contributors 634 635Many thanks to everyone who has contributed to our release: 636* Alyssa Ross <hi@alyssa.is> 637* Bo Chen <chen.bo@intel.com> 638* Daniel Farina <daniel@ubicloud.com> 639* Jinank Jain <jinankjain@microsoft.com> 640* Muminul Islam <muislam@microsoft.com> 641* Peteris Rudzusiks <rye@stripe.com> 642* Philipp Schuster <philipp.schuster@cyberus-technology.de> 643* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 644* Rob Bradford <rbradford@rivosinc.com> 645* Ruslan Mstoi <ruslan.mstoi@intel.com> 646* Sean Banko <sbanko@crusoeenergy.com> 647* Thomas Barrett <tbarrett@crusoeenergy.com> 648* Wei Liu <liuwe@microsoft.com> 649* Yi Wang <foxywang@tencent.com> 650* acarp <acarp@crusoeenergy.com> 651 652# v37.0 653 654This release has been tracked in our [roadmap 655project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 656v37.0. The following user visible changes have been made: 657 658### Long Term Support (LTS) Release 659 660This release is a LTS release. Point releases for bug fixes will be made 661for the next 18 months; live migration and live upgrade will be 662supported between the point releases of the LTS. 663 664### Multiple PCI segments Support for 32-bit VFIO devices 665 666Now VFIO devices with 32-bit memory BARs can be attached to non-zero PCI 667segments on the guest, allowing users to have more 32-bit devices and 668assign such devices to appropriate NUMA nodes for better performance. 669 670### Configurable Named TAP Devices 671 672Named TAP devices now accepts IP configuration from users, such as IP 673and MAC address, as long as the named TAP device is created by Cloud 674Hypervisor (e.g. not existing TAP devices). 675 676### TTY Output from Both Serial Device and Virtio Console 677 678Now legacy serial device and virtio console can be set as TTY mode as 679the same time. This allows users to capture early boot logs with the 680legacy serial device without losing performance benefits of using 681virtio-console, when appropriate kernel configuration is used (such as 682using kernel command-line `console=hvc0 earlyprintk=ttyS0` on x86). 683 684### Faster VM Restoration from Snapshots 685 686The speed of VM restoration from snapshots is improved with a better 687implementation of deserializing JSON files. 688 689### Notable Bug Fixes 690 691* Fix aio backend behavior for block devices when writeback cache 692 disabled (#5930) 693* Fix PvPanic device PCI BAR alignment (#5956) 694* Bug fix to OpenAPI specification file (#5967) 695* Error out early for live migration when TDX is enabled (#6025) 696 697### Contributors 698 699Many thanks to everyone who has contributed to our release: 700 701* Bo Chen <chen.bo@intel.com> 702* Jinank Jain <jinankjain@microsoft.com> 703* Markus Sütter <markus.suetter@secunet.com> 704* Michael Zhao <michael.zhao@arm.com> 705* Muminul Islam <muislam@microsoft.com> 706* Rob Bradford <rbradford@rivosinc.com> 707* Rui Chang <rui.chang@arm.com> 708* Ruslan Mstoi <ruslan.mstoi@intel.com> 709* Thomas Barrett <tbarrett@crusoeenergy.com> 710* Wei Liu <liuwe@microsoft.com> 711* Yi Wang <foxywang@tencent.com> 712* Yong He <alexyonghe@tencent.com> 713 714# v36.0 715 716This release has been tracked in our [roadmap 717project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 718v36.0. The following user visible changes have been made: 719 720### Command Line Changes 721 722We switched back to use the `clap` crate to create our command line, 723since the `argh` crate is barely maintained. There were several syntax 724changes: 725 726* All `--option value` commands now are `--option=value`. 727* The `--disk DISK1 --disk DISK2` command now is `--disk DISK1 DISK2`. 728* The `-v -v -v`command now is `-vvv`. 729 730Note: the released binary size increased around 0.3M due to this change. 731 732### Enabled Features Reported via API Endpoint and CLI 733 734Now the enabled (Cargo) features of the running Cloud Hypervisor 735instance can be queried via API endpoint (`/vmm.ping`) and CLI 736(`--version -v`). 737 738### NUMA Support for PCI segments 739 740The `--numa` command is augmented with a new option `pci_segment=`, so 741that users can define the relationship between PCI segments and NUMA 742nodes. Examples can be found from the [memory documentation](docs/memory.md) 743 744### CPU Topology Support on AMD Platforms 745 746Now the CPU topology on x86_64 platforms supports multiple vendors. 747 748### Unix Socket Backend for Serial Port 749 750The `--serial` command is augmented with a new option `socket=`, allowing 751users to access the serial port using a Unix socket. 752 753### AIO Backend for Block Devices 754 755An AIO backend is added for `virtio-block` devices to improve block 756device performance when the `io_uring` feature is not supported by the 757host Operating System. 758 759### Documentation Improvements 760 761* New [documentation](docs/coverage.md) for collecting coverage data 762* Various typo fixes 763 764### Notable Bug Fixes 765 766* Fix a deadlock when TDX is enabled (#5845) 767* Only advertise AMX feature bits to guest when the AMX cpu feature is 768 enabled (#5834) 769* Correct default value for vCPU topology on AArch64 (#5893) 770 771### Contributors 772 773Many thanks to everyone who has contributed to our release: 774 775* Anatol Belski <anbelski@linux.microsoft.com> 776* Bo Chen <chen.bo@intel.com> 777* Dario Nieuwenhuis <dirbaio@dirbaio.net> 778* Jianyong Wu <jianyong.wu@arm.com> 779* Jinank Jain <jinankjain@microsoft.com> 780* Muminul Islam <muislam@microsoft.com> 781* Praveen K Paladugu <prapal@linux.microsoft.com> 782* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 783* Rob Bradford <rbradford@rivosinc.com> 784* Thomas Barrett <tbarrett@crusoeenergy.com> 785* Wei Liu <liuwe@microsoft.com> 786* Yi Wang <foxywang@tencent.com> 787* dom.song <dom.song@amperecomputing.com> 788 789# v35.0 790 791This release has been tracked in our [roadmap 792project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 793v35.0. The following user visible changes have been made: 794 795### `virtio-vsock` Support for Linux Guest Kernel v6.3+ 796 797Since kernel v6.3, a vsock packet can be included in a single descriptor, 798instead of being split over two descriptors. Our `virtio-vsock` implementation 799now support both situations. 800 801### User Specified Serial Number for `virtio-block` 802 803A new option `serial` is added to the `--block` command that allows users to 804specify a serial number for block devices which will be guest visible. 805 806### vCPU TSC Frequency Included in Migration State 807 808This ensures migration works correctly between hosts that have different TSC 809frequencies if the guest is running with TSC as the source of timekeeping. 810 811### Notable Bug Fixes 812 813* Disallow concurrent CPU resizing (#5668) 814* Handle APIC EOI message for MSHV (#5681) 815* Use wrapping add for memory offset from instruction emulator (#5719) 816* Add global spell check via the 'typos' GitHub action (#5720) 817* Ensure probing reads are block size aligned (#5727) 818* Multiple bug fixes around the latency counter for block devices 819 (#5712, #5750, #5762, #5763) 820* Replace unsound `static mut` with `once_cell` (#5772) 821 822### Contributors 823 824Many thanks to everyone who has contributed to our release: 825 826* Alyssa Ross <hi@alyssa.is> 827* Anatol Belski <anbelski@linux.microsoft.com> 828* Bo Chen <chen.bo@intel.com> 829* Christian Blichmann <cblichmann@google.com> 830* Jianyong Wu <jianyong.wu@arm.com> 831* Jinank Jain <jinankjain@microsoft.com> 832* Julian Stecklina <julian.stecklina@cyberus-technology.de> 833* Omer Faruk Bayram <omer.faruk@sartura.hr> 834* Philipp Schuster <philipp.schuster@cyberus-technology.de> 835* Rob Bradford <rbradford@rivosinc.com> 836* Ruslan Mstoi <ruslan.mstoi@intel.com> 837* Thomas Barrett <tbarrett@crusoeenergy.com> 838* Wei Liu <liuwe@microsoft.com> 839* Yi Wang <foxywang@tencent.com> 840* zhongbingnan <zhongbingnan@bytedance.com> 841 842# v34.0 843 844This release has been tracked in our [roadmap 845project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 846v34.0. The following user visible changes have been made: 847 848### Paravirtualised Panic Device Support 849 850A new device has been added that can communicate when the guest kernel has 851panicked and share those details with the VMM. This is controlled with a new 852`--pvpanic` command line option and JSON API change equivalent. (#5526) 853 854### Improvements to VM Core Dump 855 856Requesting to dump the guest memory as core dump will now transparently pause 857the VM if required; returning to the original state after. (#5604) 858 859### QCOW2 Support for Backing Files 860 861The support for QCOW2 files has been enhanced to include support for using 862backing files. (#5573) 863 864### Minimum Host Kernel Bump 865 866The minimum supported host kernel is now 5.13 in order to incorporate a bug fix 867for `KVM_FEATURE_ASYNC_PF_INT` functionality. (#5626) 868 869### Notable Bug Fixes 870 871* The x86 emulator is only compiled in if MSHV is compiled in (the kernel 872 carries out this job with KVM) (#5561). 873* A regression has been fixed in VFIO support for devices that use MSI rather 874 than MSI-X (#5658). 875* When triggering a VM shutdown or reset via I/O the vCPU thread will be 876 blocked until that asynchronous event has been received (#5645). 877* Pausing a VM is now a synchronous action with the request only completing 878 when all vCPUs are paused (#5611). 879* Event monitor support now correctly supports concurrent access (#5633). 880* Bug fixes for the QCOW2 file support (#5573). 881 882### Contributors 883 884Many thanks to everyone who has contributed to our release: 885 886* Alyssa Ross <hi@alyssa.is> 887* Anatol Belski <anbelski@linux.microsoft.com> 888* Bo Chen <chen.bo@intel.com> 889* Changyuan Lyu <changyuanl@google.com> 890* Christian Blichmann <cblichmann@google.com> 891* Manish Goregaokar <manishsmail@gmail.com> 892* Omer Faruk Bayram <omer.faruk@sartura.hr> 893* Philipp Schuster <philipp.schuster@cyberus-technology.de> 894* Praveen K Paladugu <prapal@linux.microsoft.com> 895* Rob Bradford <rbradford@rivosinc.com> 896* Ruslan Mstoi <ruslan.mstoi@intel.com> 897* Yi Wang <foxywang@tencent.com> 898* Yong He <alexyonghe@tencent.com> 899* Yu Li <liyu.yukiteru@bytedance.com> 900* dom.song <dom.song@amperecomputing.com> 901 902# v33.0 903 904This release has been tracked in our [roadmap 905project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 906v33.0. The following user visible changes have been made: 907 908### D-Bus based API 909 910A D-Bus based API has been added as an alternative to the existing REST 911API. This feature is gated by the `dbus_api` feature. Details can be 912found in the [API documentation](docs/api.md). 913 914### Expose Host CPU Cache Details for AArch64 915 916Now the CPU cache information on the host is properly exposed to the 917guest on AArch64. 918 919### Notable Bug Fixes 920 921* Report errors explicitly to users when VM failed to boot (#5453) 922* Fix VFIO on platforms with non-4k page size (#5450, #5469) 923* Fix TDX initialization (#5454) 924* Ensure all guest memory regions are page-size aligned (#5496) 925* Fix seccomp filter lists related to virtio-console, serial and pty 926 (#5506, #5524) 927* Populate APIC ID properly (#5512) 928* Ignore and warn TAP FDs in more situations (#5522) 929 930### Contributors 931 932Many thanks to everyone who has contributed to our release: 933 934* Alyssa Ross <hi@alyssa.is> 935* Anatol Belski <anbelski@linux.microsoft.com> 936* Bo Chen <chen.bo@intel.com> 937* Jianyong Wu <jianyong.wu@arm.com> 938* Omer Faruk Bayram <omer.faruk@sartura.hr> 939* Rafael Mendonca <rafaelmendsr@gmail.com> 940* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 941* Rob Bradford <rbradford@rivosinc.com> 942* Ruslan Mstoi <ruslan.mstoi@intel.com> 943* Yu Li <liyu.yukiteru@bytedance.com> 944* zhongbingnan <zhongbingnan@bytedance.com> 945 946# v32.0 947 948This release has been tracked in our [roadmap 949project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 950v32.0. The following user visible changes have been made: 951 952### Increased PCI Segment Limit 953 954The maximum number of PCI segments that can be used is now 96 (up from 16). 955 956### API Changes 957 958* The VmmPingResponse now includes the PID as well as the build details. 959 (#5348) 960 961### Notable Bug Fixes 962 963* Ignore and warn TAP FDs sent via the HTTP request body (#5350) 964* Properly preserve and close valid FDs for TAP devices (#5373) 965* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360) 966* Only touch the tty flags if it's being used (#5343) 967* Fix seccomp filter lists for vhost-user devices (#5361) 968* The number of vCPUs is capped at the hypervisor maximum (#5357) 969* Fixes for TTY reset (#5414) 970* CPU topology fixes on MSHV (#5325) 971* Seccomp fixes for older distributions (#5397) 972 973### Contributors 974 975Many thanks to everyone who has contributed to our release: 976 977* Alyssa Ross <hi@alyssa.is> 978* Anatol Belski <anbelski@linux.microsoft.com> 979* Bo Chen <chen.bo@intel.com> 980* Hao Xu <howeyxu@tencent.com> 981* Muminul Islam <muislam@microsoft.com> 982* Omer Faruk Bayram <omer.faruk@sartura.hr> 983* Rafael Mendonca <rafaelmendsr@gmail.com> 984* Rob Bradford <rbradford@rivosinc.com> 985* Ruslan Mstoi <ruslan.mstoi@intel.com> 986* Smit Gardhariya <gardhariya.smit@gmail.com> 987* Wei Liu <liuwe@microsoft.com> 988 989# v31.1 990 991This is a bug fix release. The following issues have been addressed: 992 993* Ignore and warn TAP FDs sent via the HTTP request body (#5350) 994* Properly preserve and close valid FDs for TAP devices (#5373) 995* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360) 996* Only touch the tty flags if it's being used (#5343) 997* Fix seccomp filter lists for vhost-user devices (#5361) 998 999# v31.0 1000 1001This release has been tracked in our [roadmap 1002project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 1003v31.0. The following user visible changes have been made: 1004 1005### Update to Latest `acpi_tables` 1006 1007Adapted to the latest [acpi_tables](https://github.com/rust-vmm/acpi_tables). 1008There has been significant API changes in the crate. 1009 1010### Update Reference Kernel to 6.2 1011 1012Updated the recommended guest kernel version from 6.1.6 to 6.2. 1013 1014### Improvements on Console `SIGWINCH` Handler 1015 1016A separate thread had been created to capture the `SIGWINCH` signal and resize 1017the guest console. Now the thread is skipped if the console is not resizable. 1018 1019Two completely different code paths existed for handling console resizing, one 1020for `tty` and the other for `pty`. That makes the understanding of the console 1021handling code unnecessarily complicated. Now the code paths are unified. Both 1022`tty` and `pty` are supported in single `SIGWINCH` handler. And the new handler 1023can works with kernel versions earlier than v5.5. 1024 1025### Remove Directory Support from `MemoryZoneConfig::file` 1026 1027Setting a directory to `MemoryZoneConfig::file` is no longer supported. 1028 1029Before this change, user can set a directory to `file` of the `--memory-zone` 1030option. In that case, a temporary file will be created as the backing file for 1031the `mmap(2)` operation. This functionality has been unnecessary since we had 1032the native support for hugepages and allocating anonymous shared memory. 1033 1034### Documentation Improvements 1035 1036* Various improvements in API document 1037* Improvements in Doc comments 1038* Updated Slack channel information in README 1039 1040### Notable Bug Fixes 1041 1042* Fixed the offset setting while removing the entire mapping of `vhost-user` FS 1043 client. 1044* Fixed the `ShutdownVmm` and `Shutdown` commands to call the correct API 1045 endpoint. 1046 1047### Contributors 1048 1049Many thanks to everyone who has contributed to our release: 1050 1051* Alyssa Ross <hi@alyssa.is> 1052* Bo Chen <chen.bo@intel.com> 1053* Daniel Farina <daniel@fdr.io> 1054* Dom <peng6662001@163.com> 1055* Hao Xu <howeyxu@tencent.com> 1056* Muminul Islam <muislam@microsoft.com> 1057* Omer Faruk Bayram <omer.faruk@sartura.hr> 1058* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1059* Rob Bradford <rbradford@rivosinc.com> 1060* Ruslan Mstoi <ruslan.mstoi@intel.com> 1061* Smit Gardhariya <gardhariya.smit@gmail.com> 1062* Yang <ailin.yang@intel.com> 1063* Yong He <alexyonghe@tencent.com> 1064 1065# v30.0 1066 1067This release has been tracked in our [roadmap 1068project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 1069v30.0. The following user visible changes have been made: 1070 1071### Command Line Changes for Reduced Binary Size 1072 1073The `clap` crate was replaced by the `argh` crate to create our command 1074line, which reduced our release binary size from 3.6MB to 3.3MB. There 1075were several syntax changes: 1076 1077* All `--option=value` commands now are `--option value`. 1078* The `--disk DISK1 DISK2` command now is `--disk DISK1 --disk DISK2`. 1079* The `-vvv` command now is `-v -v -v` 1080 1081### Basic vfio-user Server Support 1082 1083Our `vfio-user` crate is extended to provide basic server side support 1084with an example of gpio vfio-user device. This crate now is moved to [its 1085own repository](https://github.com/rust-vmm/vfio-user) under the 1086`rust-vmm` organization. 1087 1088### Heap Profiling Support 1089 1090A new building target is added for profiling purposes with examples of 1091heap profiling using `dhat` gated by the `dhat-heap` feature. 1092 1093### Documentation Improvements 1094 1095The documentation on Intel TDX is expanded with details of the building 1096and using [TD-Shim](https://github.com/confidential-containers/td-shim), 1097references to [TDX Tools](https://github.com/intel/tdx-tools), and 1098version information of guest/host kernel/TDVF/TDShim being tested. Also, 1099a new 'heap profiling' documentation is added with improvements on the 1100existing 'profiling' documentation. 1101 1102### Notable Bug Fixes 1103 1104* Close FDs for TAP devices that are provided to VM (#5199, #5206) 1105* Set vcpu thread status properly and signal `exit_evt` upon thread exit (#5211) 1106* Populate CPUID leaf 0x4000_0010 (TSC frequency) (#5178, #5179) 1107* Inform the TPM guest driver upon failed TPM requests on the host (#5151) 1108* Bug fix to OpenAPI specification file (#5186) 1109 1110### Contributors 1111 1112Many thanks to everyone who has contributed to our release: 1113 1114* Anatol Belski <anbelski@linux.microsoft.com> 1115* Anirudh Rayabharam <anrayabh@linux.microsoft.com> 1116* Bo Chen <chen.bo@intel.com> 1117* Jinank Jain <jinankjain@microsoft.com> 1118* Kaihang Zhang <kaihang.zhang@smartx.com> 1119* Michael Zhao <michael.zhao@arm.com> 1120* Muminul Islam <muislam@microsoft.com> 1121* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1122* Praveen K Paladugu <prapal@linux.microsoft.com> 1123* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1124* Rob Bradford <robert.bradford@intel.com> 1125* Ruslan Mstoi <ruslan.mstoi@intel.com> 1126* Sebastien Boeuf <sebastien.boeuf@intel.com> 1127* Wei Liu <liuwe@microsoft.com> 1128* Yong He <alexyonghe@tencent.com> 1129* Yu Li <liyu.yukiteru@bytedance.com> 1130 1131# v28.2 1132This is a bug fix release. The following issues have been addressed: 1133 1134* Fix QCOW2 refcount table size (#5034) 1135* Fix unpause support on MSHV in dual binary (#5037) 1136* Threads inside `virtio` devices are now shutdown on reboot (#5095) 1137 1138# v29.0 1139 1140This release has been tracked in our [roadmap 1141project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 1142v29.0. The following user visible changes have been made: 1143 1144### Release Binary Supports Both MSHV and KVM 1145 1146On `x86-64` the binary included in releases supports both the KVM and MSHV 1147hypervisor with runtime detection to identify the correct hypervisor to use. 1148 1149### Snapshot/Restore and Live Migration Improvements 1150 1151Improvements have been made to the data structures used for both live migration 1152and snapshot/restore. Unfortunately this has broken compatibility with older 1153versions (support for migrating between major versions is not yet officially 1154supported but has worked for some versions.) 1155 1156### Heap Allocation Improvements 1157 1158Improvements have been made to the volume of heap allocations when running with 1159`virtio-block` devices along with a reduction in the peak heap size. 1160 1161### `ch-remote` Improvements 1162 1163Support for "pinging" the VMM and shutting the VMM down have been added to 1164`ch-remote`. 1165 1166### `AArch64` Documentation Integration 1167 1168The documentation for `AArch64` support has been integrated into the main 1169README. 1170 1171### `virtio-block` Counters Enhancement 1172 1173The counters for the `virtio-block` device has extended to include min/mean/max 1174latency counters. 1175 1176### TCP Offload Control 1177 1178The `virtio-net` device has gained support for controlling the enabling of 1179checksum and offloading. This allows the device to be used in environments 1180where the hardware lacks support for the offloading. 1181 1182### Notable Bug Fixes 1183 1184* Update dependencies including a version of `linux-loader` that addresses an 1185 infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6)) 1186* Fix bugs related to `virtio-net` including an integer overflow issue 1187 (#4924, #4949) 1188* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920) 1189* Memory mapped into the guest is now marked as non-dumpable which prevents large core files (#5016) 1190* Fix QCOW2 refcount table size (#5034) 1191* Fix unpause support on MSHV in dual binary (#5037) 1192* Threads inside `virtio` devices are now shutdown on reboot (#5095) 1193 1194### Removals 1195 1196No functionality has been removed in this release. 1197 1198### Deprecations 1199 1200* Support for specifying a directory with `MemoryZoneConfig::file` or 1201 `MemoryConfig::file` has been deprecated. This was originally used for 1202 supporting huge pages or shared memory backing which is now natively supported 1203 (#5085) 1204 1205### Contributors 1206 1207Many thanks to everyone who has contributed to our release: 1208 1209* Bo Chen <chen.bo@intel.com> 1210* Claudio Fontana <claudio.fontana@gmail.com> 1211* Hao Xu <howeyxu@tencent.com> 1212* Henry Wang <Henry.Wang@arm.com> 1213* Jinank Jain <jinankjain@microsoft.com> 1214* Michael Zhao <michael.zhao@arm.com> 1215* Muminul Islam <muislam@microsoft.com> 1216* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1217* Rob Bradford <robert.bradford@intel.com> 1218* Sebastien Boeuf <sebastien.boeuf@intel.com> 1219* Shuaiyi Zhang <zhangsy28@lenovo.com> 1220* Wei Liu <liuwe@microsoft.com> 1221* Yong He <alexyonghe@tencent.com> 1222* Yuji Hagiwara <yuuzi41@gmail.com> 1223 1224 1225# v28.1 1226This is a bug fix release. The following issues have been addressed: 1227 1228* Update dependencies including a version of `linux-loader` that 1229addresses an infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6)) 1230* Fix bugs related to `virtio-net` including an integer overflow issue 1231 (#4924, #4949) 1232* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920) 1233* Improve README and documentation 1234 1235# v28.0 1236 1237This release has been tracked in our new [roadmap 1238project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 1239v28.0. 1240 1241### Community Engagement (Reminder) 1242 1243Just a reminder that we have a new mailing list to support broader community 1244discussions. Please consider 1245[subscribing](https://lists.cloudhypervisor.org/g/dev/). We plan to use 1246this to announce a regular meeting for those interested in talking about Cloud 1247Hypervisor development. 1248 1249### Long Term Support (LTS) Release 1250 1251This is the first version of Cloud Hypervisor to be released under the LTS 1252release process. Point releases for bug fixes will be made for the next 18 1253months; live migration and live upgrade will be supported between the point 1254releases of the LTS. 1255 1256### Virtualised TPM Support 1257 1258Support for adding an emulated CRB TPM has been added. This has its own [TPM 1259documentation](docs/tpm.md). 1260 1261### Transparent Huge Page Support 1262 1263By default, but controllable through `--memory thp=off` if it possible to back 1264the guest memory with Transparent Huge Pages (no file backing/`shared=off`) 1265then this will be used resulting in improved boot performance. 1266 1267### README Quick Start Improved 1268 1269The README has been refreshed with the quick start guide updated to reflect the 1270different firmware options and to recommend the use of pre-built binaries. 1271 1272### Notable Bug Fixes 1273 1274* Inappropriate Copy-on-Write of pinned pages (e.g. VFIO) leading to higher 1275 memory consumption (#4835) 1276* Multiple `virtio` device bug fixes found through fuzzing (#4859, #4799) 1277* Large QCOW files (> 4TiB) are now supported (#4767) 1278* Support for > 31 vCPUS on aarch64 (#4863) 1279* Multiple fixes to OpenAPI specification file (#4720, #4811) 1280* Programming of the MSI-X table has been optimised leading to faster boot on 1281 newer Linux kernels (#4744) 1282* Error on reboot from race to close TAP devices (#4871) 1283* Non-spec compliant virtio-block read-only support (#4888) 1284 1285### Removals 1286 1287The following functionality has been removed: 1288 1289* Support for non-PVH firmware booting has been removed (#4511) 1290* I/O ports used for older versions of firmware have been removed (#3926) 1291* Deprecated API options for kernel/cmdline/initramfs have been removed (#4737) 1292 1293### Contributors 1294 1295Many thanks to everyone who has contributed to our release: 1296 1297* Anatol Belski <anbelski@linux.microsoft.com> 1298* Bo Chen <chen.bo@intel.com> 1299* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1300* Jianyong Wu <jianyong.wu@arm.com> 1301* Jinank Jain <jinankjain@microsoft.com> 1302* Jinrong Liang <cloudliang@tencent.com> 1303* lv.mengzhao <lv.mengzhao@zte.com.cn> 1304* Michael Zhao <michael.zhao@arm.com> 1305* Muminul Islam <muislam@microsoft.com> 1306* Praveen K Paladugu <prapal@linux.microsoft.com> 1307* Rob Bradford <robert.bradford@intel.com> 1308* Sebastien Boeuf <sebastien.boeuf@intel.com> 1309* Wei Liu <liuwe@microsoft.com> 1310 1311# v27.0 1312 1313This release has been tracked in our new [roadmap 1314project](https://github.com/orgs/cloud-hypervisor/projects/6) as iteration 1315v27.0. 1316 1317### Community Engagement 1318 1319A new mailing list has been created to support broader community discussions. 1320Please consider [subscribing](https://lists.cloudhypervisor.org/g/dev/); an 1321announcement of a regular meeting will be announced via this list shortly. 1322 1323### Prebuilt Packages 1324 1325Prebuilt packages are now available. Please see [this 1326document](https://github.com/cloud-hypervisor/obs-packaging/blob/main/README.md) 1327on how to install. These packages also include packages for the different 1328firmware options available. 1329 1330### Network Device MTU Exposed to Guest 1331 1332The MTU for the TAP device associated with a `virtio-net` device is now exposed 1333to the guest. If the user provides a MTU with `--net mtu=..` then that MTU is 1334applied to created TAP interfaces. This functionality is also exposed for 1335`vhost-user-net` devices including those created with the reference backend 1336(#4658, #4676.) 1337 1338### Boot Tracing 1339 1340Support for generating a trace report for the boot time has been added 1341including a script for generating an SVG from that trace (#4659.) 1342 1343### Simplified Build Feature Flags 1344 1345The set of feature flags, for e.g. experimental features, have been simplified: 1346 1347* `msvh` and `kvm` features provide support for those specific hypervisors 1348 (with `kvm` enabled by default), 1349* `tdx` provides support for Intel TDX; and although there is no MSHV support 1350 now it is now possible to compile with the `mshv` feature (#4696,) 1351* `tracing` adds support for boot tracing, 1352* `guest_debug` now covers both support for gdbing a guest (formerly `gdb` 1353 feature) and dumping guest memory. 1354 1355The following feature flags were removed as the functionality was enabled by 1356default: `amx`, `fwdebug`, `cmos` and `common` (#4679, #4632.) 1357 1358### Asynchronous Kernel Loading 1359 1360AArch64 has gained support for loading the guest kernel asynchronously like 1361x86-64. (#4538) 1362 1363### GDB Support for AArch64 1364 1365GDB stub support (accessed through `--gdb` under `guest_debug` feature) is now 1366available on AArch64 as well as as x86-64. 1367 1368### Notable Bug Fixes 1369 1370* This version incorporates a version of `virtio-queue` that addresses an issue 1371 where a rogue guest can potentially DoS the VMM (rust-vmm/vm-virtio#196.) 1372* Improvements around PTY handling for `virtio-console` and serial devices 1373 (#4520, #4533, #4535.) 1374* Improved error handling in virtio devices (#4626, #4605, #4509, #4631, #4697) 1375 1376### Deprecations 1377 1378Deprecated features will be removed in a subsequent release and users should 1379plan to use alternatives. 1380 1381* Booting legacy firmware (compiled without a PVH header) has been deprecated. 1382 All the firmware options (Cloud Hypervisor OVMF and Rust Hypervisor Firmware) 1383 support booting with PVH so support for loading firmware in a legacy mode is no 1384 longer needed. This functionality will be removed in the next release. 1385 1386### Contributors 1387 1388Many thanks to everyone who has contributed to our release: 1389 1390* Anatol Belski <anbelski@linux.microsoft.com> 1391* Bo Chen <chen.bo@intel.com> 1392* James O. D. Hunt <james.o.hunt@intel.com> 1393* Jianyong Wu <jianyong.wu@arm.com> 1394* Markus Napierkowski <markus.napierkowski@cyberus-technology.de> 1395* Michael Zhao <michael.zhao@arm.com> 1396* Nuno Das Neves <nudasnev@microsoft.com> 1397* Rob Bradford <robert.bradford@intel.com> 1398* Sebastien Boeuf <sebastien.boeuf@intel.com> 1399* Smit Gardhariya <sgardhariya@microsoft.com> 1400* Wei Liu <liuwe@microsoft.com> 1401 1402# v26.0 1403 1404This release has been tracked through the [v26.0 1405project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/30). 1406 1407### SMBIOS Improvements via `--platform` 1408 1409`--platform` and the appropriate API structure has gained support for supplying 1410OEM strings (primarily used to communicate metadata to systemd in the guest) 1411(#4319, #4446) and support for specifying the UUID (#4389.) 1412 1413### Unified Binary MSHV and KVM Support 1414 1415Support for both the MSHV and KVM hypervisors can be compiled into the same 1416binary with the detection of the hypervisor to use made at runtime. 1417 1418### Notable Bug Fixes 1419 1420* The prefetchable flag is preserved on BARs for VFIO devices (#4353, #4454) 1421* PCI Express capabilities for functionality we do not support are now filtered 1422 out (#4456) 1423* GDB breakpoint support is more reliable (#4354, #4363) 1424* `SIGINT` and `SIGTERM` signals are now handled before the VM has booted 1425 (#4269, #4293) 1426* Multiple API event loop handling bug fixes (#4309, #4362) 1427* Incorrect assumptions in virtio queue numbering were addressed, allowing 1428 the`virtio-fs` driver in OVMF to be used (#4341, #4314) 1429* VHDX file format header fix (#4291) 1430* The same VFIO device cannot be added twice (#4453, #4463) 1431* SMBIOS tables were being incorrectly generated (#4442) 1432 1433### Deprecations 1434 1435Deprecated features will be removed in a subsequent release and users should 1436plan to use alternatives. 1437 1438* The top-level `kernel` and `initramfs` members on the `VmConfig` have been 1439 moved inside a `PayloadConfig` as the `payload` member. The OpenAPI document 1440 has been updated to reflect the change and the old API members continue to 1441 function and are mapped to the new version. The expectation is that these old 1442 versions will be removed in the v28.0 release. 1443 1444### Removals 1445 1446The following functionality has been removed: 1447 1448* The unused `poll_queue` parameter has been removed from `--disk` and 1449 equivalent. This was residual from the removal of the `vhost-user-block` 1450 spawning feature (#4402.) 1451 1452### Contributors 1453 1454Many thanks to everyone who has contributed to our release: 1455 1456* Alyssa Ross <hi@alyssa.is> 1457* Anatol Belski <ab@php.net> 1458* Archana Shinde <archana.m.shinde@intel.com> 1459* Bo Chen <chen.bo@intel.com> 1460* lizhaoxin1 <Lxiaoyouling@163.com> 1461* Maximilian Nitsch <maximilian.nitsch@d3tn.com> 1462* Michael Zhao <michael.zhao@arm.com> 1463* Rob Bradford <robert.bradford@intel.com> 1464* Sebastien Boeuf <sebastien.boeuf@intel.com> 1465* Steven Dake <sdake@lambdal.com> 1466* Wei Liu <liuwe@microsoft.com> 1467 1468# v25.0 1469 1470This release has been tracked through the [v25.0 1471project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/29). 1472 1473### `ch-remote` Improvements 1474 1475The `ch-remote` command has gained support for creating the VM from a JSON 1476config and support for booting and deleting the VM from the VMM. 1477 1478### VM "Coredump" Support 1479 1480Under the `guest_debug` feature flag it is now possible to extract the memory 1481of the guest for use in debugging with e.g. the `crash` utility. (#4012) 1482 1483### Notable Bug Fixes 1484 1485* Always restore console mode on exit (#4249, #4248) 1486* Restore vCPUs in numerical order which fixes aarch64 snapshot/restore (#4244) 1487* Don't try and configure `IFF_RUNNING` on TAP devices (#4279) 1488* Propagate configured queue size through to vhost-user backend (#4286) 1489* Always Program vCPU CPUID before running the vCPU to fix running on Linux 1490 5.16 (#4156) 1491* Enable ACPI MADT "Online Capable" flag for hotpluggable vCPUs to fix newer 1492 Linux guest 1493 1494### Removals 1495 1496The following functionality has been removed: 1497 1498* The `mergeable` option from the `virtio-pmem` support has been removed 1499 (#3968) 1500* The `dax` option from the `virtio-fs` support has been removed (#3889) 1501 1502### Contributors 1503 1504Many thanks to everyone who has contributed to our release: 1505 1506* Dylan Bargatze <dbargatz@users.noreply.github.com> 1507* Jinank Jain <jinankjain@microsoft.com> 1508* Michael Zhao <michael.zhao@arm.com> 1509* Rob Bradford <robert.bradford@intel.com> 1510* Sebastien Boeuf <sebastien.boeuf@intel.com> 1511* Wei Liu <liuwe@microsoft.com> 1512* Yi Wang <wang.yi59@zte.com.cn> 1513 1514# v24.0 1515 1516This release has been tracked through the [v24.0 1517project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/28). 1518 1519### Bypass Mode for `virtio-iommu` 1520 1521`virtio-iommu` specification describes how a device can be attached by default 1522to a bypass domain. This feature is particularly helpful for booting a VM with 1523guest software which doesn't support `virtio-iommu` but still need to access 1524the device. Now that Cloud Hypervisor supports this feature, it can boot a VM 1525with Rust Hypervisor Firmware or OVMF even if the `virtio-block` device exposing 1526the disk image is placed behind a virtual IOMMU. 1527 1528### Ensure Identifiers Uniqueness 1529 1530Multiple checks have been added to the code to prevent devices with identical 1531identifiers from being created, and therefore avoid unexpected behaviors at boot 1532or whenever a device was hot plugged into the VM. 1533 1534### Sparse Mmap support 1535 1536Sparse mmap support has been added to both VFIO and vfio-user devices. This 1537allows the device regions that are not fully mappable to be partially mapped. 1538And the more a device region can be mapped into the guest address space, the 1539fewer VM exits will be generated when this device is accessed. This directly 1540impacts the performance related to this device. 1541 1542### Expose Platform Serial Number 1543 1544A new `serial_number` option has been added to `--platform`, allowing a user to 1545set a specific serial number for the platform. This number is exposed to the 1546guest through the SMBIOS. 1547 1548### Notable Bug Fixes 1549 1550* Fix loading RAW firmware (#4072) 1551* Reject compressed QCOW images (#4055) 1552* Reject virtio-mem resize if device is not activated (#4003) 1553* Fix potential mmap leaks from VFIO/vfio-user MMIO regions (#4069) 1554* Fix algorithm finding HOB memory resources (#3983) 1555 1556### Notable Improvements 1557 1558* Refactor interrupt handling (#4083) 1559* Load kernel asynchronously (#4022) 1560* Only create ACPI memory manager DSDT when resizable (#4013) 1561 1562### Deprecations 1563 1564Deprecated features will be removed in a subsequent release and users should 1565plan to use alternatives 1566 1567* The `mergeable` option from the `virtio-pmem` support has been deprecated 1568 (#3968) 1569* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 1570 1571### New on the Website 1572 1573A new blog post [Achieving Bare Metal Performance Within a Virtual 1574Machine](https://www.cloudhypervisor.org/blog/achieving-bare-metal-performance-within-a-virtual-machine) 1575has been added to the Cloud Hypervisor website. 1576 1577### Contributors 1578 1579Many thanks to everyone who has contributed to our release: 1580 1581* Anatol Belski <anbelski@linux.microsoft.com> 1582* Bo Chen <chen.bo@intel.com> 1583* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1584* LiHui <andrewli@kubesphere.io> 1585* Maksym Pavlenko <pavlenko.maksym@gmail.com> 1586* Rob Bradford <robert.bradford@intel.com> 1587* Sebastien Boeuf <sebastien.boeuf@intel.com> 1588* Steven Dake <steven.dake@gmail.com> 1589* Vincent Batts <vbatts@hashbangbash.com> 1590* Wei Liu <liuwe@microsoft.com> 1591 1592# v23.1 1593 1594This is a bug fix release. The following issues have been addressed: 1595 1596* Add some missing seccomp rules 1597* Remove `virtio-fs` filesystem entries from config on removal 1598* Do not delete API socket on API server start (#4026) 1599* Reject `virtio-mem` resize if the guest doesn't activate the device 1600* Fix OpenAPI naming of I/O throttling knobs 1601 1602# v23.0 1603 1604This release has been tracked through the [v23.0 1605project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/27). 1606 1607### vDPA Support 1608 1609A vDPA device has a datapath that complies with the virtio specification but 1610with a vendor specific control path. The addition of `--vdpa` and the REST API 1611equivalent allows the use of these devices with Cloud Hypervisor. 1612 1613### Updated OS Support list 1614 1615The list of officially supported and tested OS versions has been updated to 1616include Ubuntu "jammy" 22.04 and EOLed versions removed. 1617 1618### `AArch64` Memory Map Improvements 1619 1620The memory map when running on `AArch64` has been improved for the handling of 1621the UEFI region which means that the booted guest OS now has full access to its 1622allocated RAM. (#3938) 1623 1624### `AMX` Support 1625 1626Under a compile time gate of `amx` it is possible compile in support for the 1627`AMX` instruction set extension for guest use. This also requires runtime 1628enabling with `--cpu features=amx`. 1629 1630### Notable Bug Fixes 1631 1632* Generate error when incorrect HTTP method used for some API endpoints (#3887) 1633* CMOS based reset is now available to support rebooting on "jammy" (#3914) 1634* ACPI tables are not produced for memory hotplug when running with 1635 `virtio-mem` (#3883) 1636* `virtio-iommu` backed PCI segments are now comprehensively placed behind the 1637 vIOMMU (#3870) 1638* Seccomp rules have been extended for `virtio-fs` to support direct access 1639 (#3848) 1640 1641### Deprecations 1642 1643Deprecated features will be removed in a subsequent release and users should 1644plan to use alternatives 1645 1646* The `mergeable` option from the `virtio-pmem` support has been deprecated 1647 (#3968) 1648* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 1649 1650### Contributors 1651 1652Many thanks to everyone who has contributed to our release: 1653 1654* Bo Chen <chen.bo@intel.com> 1655* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1656* Henry Wang <Henry.Wang@arm.com> 1657* Jianyong Wu <jianyong.wu@arm.com> 1658* LiHui <andrewli@kubesphere.io> 1659* Michael Zhao <michael.zhao@arm.com> 1660* Rob Bradford <robert.bradford@intel.com> 1661* Sebastien Boeuf <sebastien.boeuf@intel.com> 1662* Wei Liu <liuwe@microsoft.com> 1663* William Douglas <william.douglas@intel.com> 1664* Yi Wang <wang.yi59@zte.com.cn> 1665 1666# v22.1 1667 1668This is a bug fix release. The following issues have been addressed: 1669 1670* VFIO ioctl reordering to fix MSI on AMD platforms (#3827) 1671* Fix `virtio-net` control queue (#3829) 1672 1673# v22.0 1674 1675This release has been tracked through the [v22.0 1676project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/25). 1677 1678### GDB Debug Stub Support 1679 1680Cloud Hypervisor can now be used as debug target with GDB. This is controlled 1681by the `gdb` compile time feature and details of how to use it can be found in 1682the [gdb 1683documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/gdb.md). 1684 1685### `virtio-iommu` Backed Segments 1686 1687In order to facilitate hotplug devices that require being behind an IOMMU (e.g. 1688QAT) there is a new option `--platform iommu_segments=<list_of_segments>` that 1689will place all the specified segments behind the IOMMU. 1690 1691### Before Boot Configuration Changes 1692 1693It is now possible to change the VM configuration (e.g. add or remove devices, 1694resize) before the VM is booted. 1695 1696### `virtio-balloon` Free Page Reporting 1697 1698If `--balloon free_page_reporting=on` is used then the guest can report pages 1699that is it not using to the VMM. The VMM will then notify the host OS that 1700those pages are no longer in use and can be freed. This can result in improved 1701memory density. 1702 1703### Support for Direct Kernel Booting with TDX 1704 1705Through the use of `TD-Shim` lightweight firmware it is now possible to 1706directly boot into the kernel with TDX. The [TDX 1707documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/intel_tdx.md#tdshim) 1708has been updated for this usage. 1709 1710### PMU Support for AArch64 1711 1712A PMU is now available on AArch64 for guest performance profiling. This will be 1713exposed automatically if available from the host. 1714 1715### Documentation Under CC-BY-4.0 License 1716 1717The documentation is now licensed under the "Creative Commons Attribution 4.0 1718International" license which is aligned with the project charter under the 1719Linux Foundation. 1720 1721### Deprecation of "Classic" `virtiofsd` 1722 1723The use of the Rust based [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd) 1724is now recommended and we are no longer testing against the C based "classic" 1725version. 1726 1727### Notable Bug Fixes 1728 1729* Can now be used on kernels without `AF_INET` support (#3785) 1730* `virtio-balloon` size is now validated against guest RAM size (#3689) 1731* Ensure that I/O related KVM VM Exits are correctly handled (#3677) 1732* Multiple TAP file descriptors can be used for `virtio-net` device hotplug (#3607) 1733* Minor API improvements and fixes (#3756, #3766, #3647, #3578) 1734* Fix sporadic seccomp violation from glibc memory freeing (#3610, #3609) 1735* Fix Windows 11 on AArch64 due to wider MSI-X register accesses (#3714, #3720) 1736* Ensure `vhost-user` features are correct across migration (#3737) 1737* Improved vCPU topology on AArch64 (#3735, #3733) 1738 1739### Contributors 1740 1741Many thanks to everyone who has contributed to our release: 1742 1743* Akira Moroo <retrage01@gmail.com> 1744* Barret Rhoden <brho@google.com> 1745* Bo Chen <chen.bo@intel.com> 1746* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1747* Feng Ye <yefeng@smartx.com> 1748* Henry Wang <Henry.Wang@arm.com> 1749* Jianyong Wu <jianyong.wu@arm.com> 1750* lizhaoxin1 <Lxiaoyouling@163.com> 1751* Michael Zhao <michael.zhao@arm.com> 1752* Rob Bradford <robert.bradford@intel.com> 1753* Sebastien Boeuf <sebastien.boeuf@intel.com> 1754* Wei Liu <liuwe@microsoft.com> 1755 1756# v21.0 1757 1758This release has been tracked through the [v21.0 1759project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/24). 1760 1761### Efficient Local Live Migration (for Live Upgrade) 1762 1763In order to support fast live upgrade of the VMM an optimised path has been 1764added in which the memory for the VM is not compared from source to 1765destination. This is activated by passing `--local` to the `ch-remote 1766send-migration` command. This means that the live upgrade can complete in the 1767order of 50ms vs 3s. (#3566) 1768 1769### Recommended Kernel is Now 5.15 1770 1771Due to an issue in the `virtio-net` code in 5.14 the recommended Linux kernel 1772is now 5.15. (#3530) 1773 1774### Notable Bug fixes 1775 1776* Multiple fixes were made to the OpenAPI YAML file to match the implementation (#3555,#3562) 1777* Avoid live migration deadlock when triggered during the kernel boot (#3585) 1778* Support live migration within firmware (#3586) 1779* Validate the `virtio-net` descriptor chain (#3548) 1780* `direct=on` (`O_DIRECT`) can now be used with a guest that makes unaligned accesses (e.g. firmware) (#3587) 1781 1782### Contributors 1783 1784Many thanks to everyone who has contributed to our release: 1785 1786* Anatol Belski <anbelski@linux.microsoft.com> 1787* Barret Rhoden <brho@google.com> 1788* Bo Chen <chen.bo@intel.com> 1789* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1790* Henry Wang <Henry.Wang@arm.com> 1791* Liang Zhou <zhoul110@chinatelecom.cn> 1792* Michael Zhao <michael.zhao@arm.com> 1793* Muhammad Falak R Wani <falakreyaz@gmail.com> 1794* Muminul Islam <muislam@microsoft.com> 1795* Rob Bradford <robert.bradford@intel.com> 1796* Sebastien Boeuf <sebastien.boeuf@intel.com> 1797* Wei Liu <liuwe@microsoft.com> 1798* Ziye Yang <ziye.yang@intel.com> 1799 1800# v20.2 1801 1802This is a bug fix release. The following issues have been addressed: 1803 1804* Don't error out when setting up the SIGWINCH handler (for console resize) 1805 when this fails due to older kernel (#3456) 1806* Seccomp rules were refined to remove syscalls that are now unused 1807* Fix reboot on older host kernels when SIGWINCH handler was not initialised 1808 (#3496) 1809* Fix virtio-vsock blocking issue (#3497) 1810 1811# v20.1 1812 1813This is a bug fix release. The following issues have been addressed: 1814 1815* Networking performance regression with `virtio-net` (#3450) 1816* Limit file descriptors sent in `vfio-user` support (#3401) 1817* Fully advertise PCI MMIO config regions in ACPI tables (#3432) 1818* Set the TSS and KVM identity maps so they don't overlap with firmware RAM 1819* Correctly update the `DeviceTree` on restore 1820 1821# v20.0 1822 1823This release has been tracked through the [v20.0 1824project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/23). 1825 1826### Multiple PCI segments support 1827 1828Cloud Hypervisor is no longer limited to 31 PCI devices. For both `x86_64` and 1829`aarch64` architectures, it is now possible to create up to 16 PCI segments, 1830increasing the total amount of supported PCI devices to 496. 1831 1832### CPU pinning 1833 1834For each vCPU, the user can define a limited set of host CPUs on which it is 1835allowed to run. This can be useful when assigning a 1:1 mapping between host and 1836guest resources, or when running a VM on a specific NUMA node. 1837 1838### Improved VFIO support 1839 1840Based on VFIO region capabilities, all regions can be memory mapped, limiting 1841the amount of triggered VM exits, and therefore increasing the performance of 1842the passthrough device. 1843 1844### Safer code 1845 1846Several sections containing unsafe Rust code have been replaced with safe 1847alternatives, and multiple comments have been added to clarify why the remaining 1848unsafe sections are safe to use. 1849 1850### Extended documentation 1851 1852The documentation related to VFIO has been updated while some new documents have 1853been introduced to cover the usage of `--cpus` parameter as well as how to run 1854Cloud Hypervisor on Intel TDX. 1855 1856### Notable bug fixes 1857 1858* Naturally align PCI BARs on relocation (#3244) 1859* Fix panic in SIGWINCH listener thread when no seccomp filter set (#3338) 1860* Use the tty raw mode implementation from libc (#3344) 1861* Fix the emulation of register D for CMOS/RTC device (#3393) 1862 1863### Contributors 1864 1865Many thanks to everyone who has contributed to our release: 1866 1867* Alyssa Ross <hi@alyssa.is> 1868* Bo Chen <chen.bo@intel.com> 1869* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1870* Michael Zhao <michael.zhao@arm.com> 1871* Muminul Islam <muislam@microsoft.com> 1872* Rob Bradford <robert.bradford@intel.com> 1873* Sebastien Boeuf <sebastien.boeuf@intel.com> 1874* Wei Liu <liuwe@microsoft.com> 1875* Willen Yang <willenyang@gmail.com> 1876* William Douglas <william.douglas@intel.com> 1877* Ziye Yang <ziye.yang@intel.com> 1878 1879# v19.0 1880 1881This release has been tracked through the [v19.0 1882project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/22). 1883 1884### Improved PTY handling for serial and `virtio-console` 1885 1886The PTY support for serial has been enhanced with improved buffering when the 1887the PTY is not yet connected to. Using `virtio-console` with PTY now results in 1888the console being resized if the PTY window is also resized. 1889 1890### PCI boot time optimisations 1891 1892Multiple optimisations have been made to the PCI handling resulting in 1893significant improvements in the boot time of the guest. 1894 1895### Improved TDX support 1896 1897When using the latest TDVF firmware the ACPI tables created by the VMM are now 1898exposed via the firmware to the guest. 1899 1900### Live migration enhancements 1901 1902Live migration support has been enhanced to support migration with `virtio-mem` 1903based memory hotplug and the `virtio-balloon` device now supports live 1904migration. 1905 1906### `virtio-mem` support with `vfio-user` 1907 1908The use of `vfio-user` userspaces devices can now be used in conjunction with 1909`virtio-mem` based memory hotplug and unplug. 1910 1911### AArch64 for `virtio-iommu` 1912 1913A paravirtualised IOMMU can now be used on the AArch64 platform. 1914 1915### Notable bug fixes 1916 1917* ACPI hotplugged memory is correctly restored after a live migration or 1918 snapshot/restore (#3165) 1919* Multiple devices from the same IOMMU group can be passed through via VFIO 1920 (#3078 #3113) 1921* Live migration with large blocks of memory was buggy due to an in issue in 1922 the underlying crate (#3157) 1923 1924### Contributors 1925 1926Many thanks to everyone who has contributed to our release: 1927 1928* Alyssa Ross <hi@alyssa.is> 1929* Bo Chen <chen.bo@intel.com> 1930* Henry Wang <Henry.Wang@arm.com> 1931* Hui Zhu <teawater@antfin.com> 1932* Jianyong Wu <jianyong.wu@arm.com> 1933* Li Yu <liyu.yukiteru@bytedance.com> 1934* Michael Zhao <michael.zhao@arm.com> 1935* Muminul Islam <muislam@microsoft.com> 1936* Rob Bradford <robert.bradford@intel.com> 1937* Sebastien Boeuf <sebastien.boeuf@intel.com> 1938* Wei Liu <liuwe@microsoft.com> 1939* William Douglas <william.douglas@intel.com> 1940* Yu Li <liyu.yukiteru@bytedance.com> 1941 1942# v18.0 1943 1944This release has been tracked through the [v18.0 1945project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/21). 1946 1947### Experimental User Device (`vfio-user`) support 1948 1949Experimental support for running PCI devices in userspace via `vfio-user` 1950has been included. This allows the use of the SPDK NVMe `vfio-user` controller 1951with Cloud Hypervisor. This is enabled by `--user-device` on the command line. 1952 1953### Migration support for `vhost-user` devices 1954 1955Devices exposed into the VM via `vhost-user` can now be migrated using the live 1956migration support. This requires support from the backend however the commonly 1957used DPDK `vhost-user` backend does support this. 1958 1959### VHDX disk image support 1960 1961Images using the VHDX disk image format can now be used with Cloud Hypervisor. 1962 1963### Device pass through on MSHV hypervisor 1964 1965When running on the MSHV hypervisor it is possible to pass through devices from 1966the host through to the guest (e.g with `--device`) 1967 1968### AArch64 for support `virtio-mem` 1969 1970The reference Linux kernel we recommend for using with Cloud Hypervisor now supports `virtio-mem` on AArch64. 1971 1972### Live migration on MSHV hypervisor 1973 1974Live migration is now supported when running on the MSHV hypervisor including 1975efficient tracking of dirty pages. 1976 1977### AArch64 CPU topology support 1978 1979The CPU topology (as configured through `--cpu topology=`) can now be 1980configured on AArch64 platforms and is conveyed through either ACPI or device 1981tree. 1982 1983### Power button support on AArch64 1984 1985Use of the ACPI power button (e.g `ch-remote --api-socket=<API socket> power-button`) 1986is now supported when running on AArch64. 1987 1988### Notable bug fixes 1989 1990* Using two PTY outputs e.g. `--serial pty --console pty` now works correctly (#3012) 1991* TTY input is now always sent to the correct destination (#3005) 1992* The boot is no longer blocked when using a unattached PTY on the serial console (#3004) 1993* Live migration is now supported on AArch64 (#3049) 1994* Ensure signal handlers are run on the correct thread (#3069) 1995 1996### Contributors 1997 1998Many thanks to everyone who has contributed to our release: 1999 2000* Alyssa Ross <hi@alyssa.is> 2001* Anatol Belski <anbelski@linux.microsoft.com> 2002* Arafatms <arafatms@outlook.com> 2003* Bo Chen <chen.bo@intel.com> 2004* Fazla Mehrab <akm.fazla.mehrab@vt.edu> 2005* Henry Wang <Henry.Wang@arm.com> 2006* Jianyong Wu <jianyong.wu@arm.com> 2007* Jiaqi Gao <jiaqi.gao@intel.com> 2008* Markus Theil <markus.theil@tu-ilmenau.de> 2009* Muminul Islam <muislam@microsoft.com> 2010* Rob Bradford <robert.bradford@intel.com> 2011* Sebastien Boeuf <sebastien.boeuf@intel.com> 2012* Wei Liu <liuwe@microsoft.com> 2013* Yu Li <liyu.yukiteru@bytedance.com> 2014 2015# v17.0 2016 2017This release has been tracked through the [v17.0 2018project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/20). 2019 2020### ARM64 NUMA support using ACPI 2021 2022The support for ACPI on ARM64 has been enhanced to include support for 2023specifying a NUMA configuration using the existing control options. 2024 2025### `Seccomp` support for MSHV backend 2026 2027The `seccomp` rules have now been extended to support running against the MSHV 2028hypervisor backend. 2029 2030### Hotplug of `macvtap` devices 2031 2032Hotplug of `macvtap` devices is now supported with the file descriptor for the 2033network device if opened by the user and passed to the VMM. The `ch-remote` 2034tool supports this functionality when adding a network device. 2035 2036### Improved SGX support 2037 2038The SGX support has been updated to match the latest Linux kernel support and 2039now supports SGX provisioning and associating EPC sections to NUMA nodes. 2040 2041### Inflight tracking for `vhost-user` devices 2042 2043Support for handling inflight tracking of I/O requests has been added to the 2044`vhost-user` devices allowing recovery after device reconnection. 2045 2046### Notable bug fixes 2047 2048* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821). 2049* The VMM side of `vhost-user` devices no longer advertise the 2050 `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM 2051(#2833). 2052* On ARM64 VMs can be created with more than 16 vCPUs (#2763). 2053 2054### Contributors 2055 2056Many thanks to everyone who has contributed to our release: 2057 2058* Anatol Belski <anbelski@linux.microsoft.com> 2059* Arafatms <arafatms@outlook.com> 2060* Bo Chen <chen.bo@intel.com> 2061* Fei Li <lifei.shirley@bytedance.com> 2062* Henry Wang <Henry.Wang@arm.com> 2063* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2064* Jianyong Wu <jianyong.wu@arm.com> 2065* Li Hangjing <lihangjing@bytedance.com> 2066* Michael Zhao <michael.zhao@arm.com> 2067* Muminul Islam <muislam@microsoft.com> 2068* Rob Bradford <robert.bradford@intel.com> 2069* Sebastien Boeuf <sebastien.boeuf@intel.com> 2070* Wei Liu <liuwe@microsoft.com> 2071* Yukiteru <wfly1998@sina.com> 2072 2073# v16.0 2074 2075This release has been tracked through the [v16.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/19). 2076 2077### Improved live migration support 2078 2079The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details. 2080 2081### Improved `vhost-user` support 2082 2083When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch. 2084 2085### ARM64 ACPI and UEFI support 2086 2087Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot. 2088 2089### Notable bug fixes 2090 2091* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578). 2092* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563). 2093* The KVM clock is now correctly handled during live migration / snapshot & restore. 2094 2095### Removed functionality 2096 2097The following formerly deprecated features have been removed: 2098 2099* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2100 binaries has been deprecated. When using direct boot users should configure 2101 their kernel with `CONFIG_PVH=y`. 2102 2103### Contributors 2104 2105Many thanks to everyone who has contributed to our release including some new faces. 2106 2107* Anatol Belski <anbelski@linux.microsoft.com> 2108* Bo Chen <chen.bo@intel.com> 2109* Dayu Liu <liu.dayu@zte.com.cn> 2110* Henry Wang <Henry.Wang@arm.com> 2111* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2112* Jianyong Wu <jianyong.wu@arm.com> 2113* Michael Zhao <michael.zhao@arm.com> 2114* Mikko Ylinen <mikko.ylinen@intel.com> 2115* Muminul Islam <muislam@microsoft.com> 2116* Ren Lei <ren.lei4@zte.com.cn> 2117* Rob Bradford <robert.bradford@intel.com> 2118* Sebastien Boeuf <sebastien.boeuf@intel.com> 2119* Wei Liu <liuwe@microsoft.com> 2120* Yi Wang <wang.yi59@zte.com.cn> 2121 2122# v15.0 2123 2124This release has been tracked through the [v15.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/18). 2125 2126Highlights for `cloud-hypervisor` version v15.0 include: 2127 2128### Version numbering and stability guarantees 2129 2130This release is the first in a new version numbering scheme to represent that 2131we believe Cloud Hypervisor is maturing and entering a period of stability. 2132With this new release we are beginning our new stability guarantees: 2133 2134* The API (including command line options) will not be removed or changed in a 2135 breaking way without a minimum of 2 releases notice. Where possible warnings 2136 will be given about the use of deprecated functionality and the deprecations 2137 will be documented in the release notes. 2138* Point releases will be made between individual releases where there are 2139 substantial bug fixes or security issues that need to be fixed. 2140 2141Currently the following items are **not** guaranteed across updates: 2142 2143* Snapshot/restore is not supported across different versions 2144* Live migration is not supported across different versions 2145* The following features are considered experimental and may change 2146 substantially between releases: TDX, SGX. 2147 2148### Network device rate limiting 2149 2150Building on our existing support for rate limiting block activity the network 2151device also now supports rate limiting. Full details of the controls are in the 2152[IO throttling documentation.](docs/io_throttling.md) 2153 2154### Support for runtime control of `virtio-net` guest offload 2155 2156The guest is now able to change the offload settings for the `virtio-net` 2157device. As well as providing a useful control this mitigates an issue in the 2158Linux kernel where the guest will attempt to reprogram the offload settings 2159even if they are not advertised as configurable (#2528). 2160 2161### `--api-socket` supports file descriptor parameter 2162 2163The `--api-socket` can now take an `fd=` parameter to specify an existing file 2164descriptor to use. This is particularly beneficial for frameworks that need to 2165programmatically control Cloud Hypervisor. 2166 2167### Bug fixes 2168 2169* A workaround has been put in place to mitigate a Linux kernel issues that 2170 results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277). 2171* PCI BARs are now correctly aligned removing the need for the guest to 2172 reprogram them (#1797,#1798) 2173* Handle TAP interface not being writable within virtio-net (due to the buffer 2174 exhaustion on the host) (#2517) 2175* The recommended Linux kernel is now v5.12.0 as it contains a fix that 2176 prevents snapshot & restore working (#2535) 2177 2178### Deprecations 2179 2180Deprecated features will be removed in a subsequent release and users should plan to use alternatives 2181 2182* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2183 binaries has been deprecated. When using direct boot users should configure 2184 their kernel with `CONFIG_PVH=y`. Will be removed in v16.0. 2185 2186### Contributors 2187 2188Many thanks to everyone who has contributed to our release including some new faces. 2189 2190* Alyssa Ross <hi@alyssa.is> 2191* Anatol Belski <anbelski@linux.microsoft.com> 2192* Bo Chen <chen.bo@intel.com> 2193* Gaelan Steele <gbs@canishe.com> 2194* Jianyong Wu <jianyong.wu@arm.com> 2195* Michael Zhao <michael.zhao@arm.com> 2196* Muminul Islam <muislam@microsoft.com> 2197* Rob Bradford <robert.bradford@intel.com> 2198* Sebastien Boeuf <sebastien.boeuf@intel.com> 2199* Wei Liu <liuwe@microsoft.com> 2200* William Douglas <william.douglas@intel.com> 2201 2202# v0.14.1 2203 2204Bug fix release branched off the v0.14.0 release. The following bugs were fixed 2205in this release: 2206 2207* CPU hotplug on Windows failed due to misreported CPU state information and 2208 the lack of HyperV CPUID bit enabled (#2437, #2449, #2436) 2209* A seccomp rule was missing that was triggered on CPU unplug (#2455) 2210* A bounds check in VIRTIO queue validation was erroneously generating 2211 DescriptorChainTooShort errors in certain circumstances (#2450, #2424) 2212 2213# v0.14.0 2214 2215This release has been tracked through the [0.14.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/17). 2216 2217Highlights for `cloud-hypervisor` version 0.14.0 include: 2218 2219### Structured event monitoring 2220 2221A new option was added to the VMM `--event-monitor` which reports structured 2222events (JSON) over a file or file descriptor at key events in the lifecycle of 2223the VM. The list of events is limited at the moment but will be further 2224extended over subsequent releases. The events exposed form part of the Cloud 2225Hypervisor API surface. 2226 2227### MSHV improvements 2228 2229Basic support has been added for running Windows guests atop the MSHV 2230hypervisor as an alternative to KVM and further improvements have been made to 2231the MSHV support. 2232 2233### Improved aarch64 platform 2234 2235The aarch64 platform has been enhanced with more devices exposed to the running 2236VM including an enhanced serial UART. 2237 2238### Updated hotplug documentation 2239 2240The documentation for the hotplug support has been updated to reflect the use 2241of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as 2242well as documenting hotplug of paravirtualised and VFIO devices. 2243 2244### PTY control for serial and `virtio-console` 2245 2246The `--serial` and `--console` parameters can now direct the console to a PTY 2247allowing programmatic control of the console from another process through the 2248PTY subsystem. 2249 2250### Block device rate limiting 2251 2252The block device performance can now be constrained as part of the VM 2253configuration allowing rate limiting. Full details of the controls are in the 2254[IO throttling documentation.](docs/io_throttling.md) 2255 2256 2257### Deprecations 2258 2259Deprecated features will be removed in a subsequent release and users should plan to use alternatives 2260 2261* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2262 binaries has been deprecated. When using direct boot users should configure 2263 their kernel with `CONFIG_PVH=y`. 2264 2265 2266### Contributors 2267 2268Many thanks to everyone who has contributed to our 0.14.0 release including 2269some new faces. 2270 2271* Bo Chen <chen.bo@intel.com> 2272* Henry Wang <Henry.Wang@arm.com> 2273* Iggy Jackson <iggy@theiggy.com> 2274* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2275* Michael Zhao <michael.zhao@arm.com> 2276* Muminul Islam <muislam@microsoft.com> 2277* Penny Zheng <Penny.Zheng@arm.com> 2278* Rob Bradford <robert.bradford@intel.com> 2279* Sebastien Boeuf <sebastien.boeuf@intel.com> 2280* Vineeth Pillai <viremana@linux.microsoft.com> 2281* Wei Liu <liuwe@microsoft.com> 2282* William Douglas <william.r.douglas@gmail.com> 2283* Zide Chen <zide.chen@intel.com> 2284 2285# v0.13.0 2286 2287This release has been tracked through the [0.13.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/16). 2288 2289Highlights for `cloud-hypervisor` version 0.13.0 include: 2290 2291### Wider VFIO device support 2292 2293It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI 2294devices that do not support MSI or MSI-X and instead rely on INTx interrupts. 2295Most notably this widens the support to most NVIDIA cards with the proprietary 2296drivers. 2297 2298### Improved huge page support 2299 2300Through the addition of `hugepage_size` on `--memory` it is now possible to 2301specify the desired size of the huge pages used when allocating the guest 2302memory. The user is required to ensure they have sufficient pages of the 2303desired size in their pool. 2304 2305### MACvTAP support 2306 2307It is now possible to provide file descriptors using the `fd` parameter to 2308`--net` which point at TAP devices that have already been opened by the user. 2309This aids integration with `libvirt` but also permits the use of MACvTAP 2310support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md). 2311 2312### VHD disk image support 2313 2314It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw 2315disk image with Cloud Hypervisor. 2316 2317### Improved Virtio device threading 2318 2319Device threads are now derived from the main VMM thread which allows more 2320restrictive seccomp filters to be applied to them. The threads also have a 2321predictable name derived from the device id. 2322 2323### Clean shutdown support via synthetic power button 2324 2325It is now possible to request that the guest VM shut itself down by triggering 2326a synthetic ACPI power button press from the VMM. If the guest is listening for 2327such an event (e.g. using systemd) then it will process the event and cleanly 2328shut down. This functionality is exposed through the HTTP API and can be 2329triggered via `ch-remote --api-socket=<API socket> power-button`. 2330 2331### Contributors 2332 2333Many thanks to everyone who has contributed to our 0.13.0 release including 2334some new faces. 2335 2336* Bo Chen <chen.bo@intel.com> 2337* Mikko Ylinen <mikko.ylinen@intel.com> 2338* Muminul Islam <muislam@microsoft.com> 2339* Rob Bradford <robert.bradford@intel.com> 2340* Samuel Ortiz <sameo@linux.intel.com> 2341* Sebastien Boeuf <sebastien.boeuf@intel.com> 2342* Vineeth Pillai <viremana@linux.microsoft.com> 2343* Wei Liu <liuwe@microsoft.com> 2344* William Douglas <william.r.douglas@gmail.com> 2345* Xie Yongji <xieyongji@bytedance.com> 2346 2347# v0.12.0 2348 2349This release has been tracked through the [0.12.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/15). 2350 2351Highlights for `cloud-hypervisor` version 0.12.0 include: 2352 2353### ARM64 enhancements 2354 2355The use of `--watchdog` is now fully supported as is the ability to reboot the 2356VM from within the guest when running Cloud Hypervisor on an ARM64 system. 2357 2358### Removal of `vhost-user-net` and `vhost-user-block` self spawning 2359 2360In order to use `vhost-user-net` or `vhost-user-block` backends the user is now 2361responsible for starting the backend and providing the socket for the VMM to 2362use. This functionality was deprecated in the last release and how now been 2363removed. 2364 2365### Migration of `vhost-user-fs` backend 2366 2367The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is 2368instead hosted in [its own 2369repository](https://gitlab.com/virtio-fs/virtiofsd-rs) 2370 2371### Enhanced "info" API 2372 2373The `vm.info` HTTP API endpoint has been extended to include the details of the 2374devices used by the VM including any VFIO devices used. 2375 2376### Contributors 2377 2378Many thanks to everyone who has contributed to our 0.12.0 release: 2379 2380* Anatol Belski <anbelski@linux.microsoft.com> 2381* Julio Montes <julio.montes@intel.com> 2382* Michael Zhao <michael.zhao@arm.com> 2383* Muminul Islam <muislam@microsoft.com> 2384* Rob Bradford <robert.bradford@intel.com> 2385* Samuel Ortiz <sameo@linux.intel.com> 2386* Wei Liu <liuwe@microsoft.com> 2387 2388# v0.11.0 2389 2390This release has been tracked through the [0.11.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/14). 2391 2392Highlights for `cloud-hypervisor` version 0.11.0 include: 2393 2394### `io_uring` support by default for `virtio-block` 2395 2396Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will 2397be used for a significantly higher performance block device. 2398 2399### Windows Guest Support 2400 2401This is the first release where we officially support Windows running as a 2402guest. Full details of how to setup the image and run Cloud Hypervisor with a 2403Windows guest can be found in the dedicated [Windows 2404documentation](docs/windows.md). 2405 2406### `vhost-user` "Self Spawning" Deprecation 2407 2408Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now 2409deprecated. Users of this functionality will receive a warning and should make 2410adjustments. The functionality will be removed in the next release. 2411 2412### `virtio-mmio` Removal 2413 2414Support for using the `virtio-mmio` transport, rather than using PCI, has been 2415removed. This has been to simplify the code and significantly 2416reduce the testing burden of the project. 2417 2418### Snapshot/Restore support for ARM64 2419 2420When running on the ARM64 architecture snapshot and restore has now been 2421implemented. 2422 2423### Improved Linux Boot Time 2424 2425The time to boot the Linux kernel has been significantly improved by the 2426identifying some areas of delays around PCI bus probing, IOAPIC programming and 2427MPTABLE issues. Full details can be seen in #1728. 2428 2429### `SIGTERM/SIGINT` Interrupt Signal Handling 2430 2431When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will 2432trigger the VMM process to cleanly deallocate resources before exiting. The 2433guest VM will not be cleanly shutdown but the VMM process will clean up its 2434resources. 2435 2436### Default Log Level Changed 2437 2438The default logging level was changed to include warnings which should make it 2439easier to see potential issues. New [logging 2440documentation](docs/logging) was also added. 2441 2442### New `--balloon` Parameter Added 2443 2444Control of the setup of `virtio-balloon` has been moved from `--memory` to its 2445own dedicated parameter. This makes it easier to add more balloon specific 2446controls without overloading `--memory`. 2447 2448### Experimental `virtio-watchdog` Support 2449 2450Support for using a new `virtio-watchdog` has been added which can be used to 2451have the VMM reboot the guest if the guest userspace fails to ping the 2452watchdog. This is enabled with `--watchdog` and requires kernel support. 2453 2454### Notable Bug Fixes 2455 2456* MTRR bit was missing from CPUID advertised to guest 2457* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170) 2458* CPU identification string is now exposed to the guest 2459* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as 2460 read only so avoids excessive VM exits (#1795) 2461* PCI device hotplug after an unplug was fixed (#1802) 2462* When using the ACPI method to resize the guest memory the full reserved size 2463 can be used (#1803) 2464* Snapshot and restore followed by a second snapshot and restore now works 2465 correctly 2466* Snapshot and restore of VMs with more than 2GiB in one region now work 2467 correctly 2468 2469### Contributors 2470 2471Many thanks to everyone who has contributed to our 0.11.0 release including some new faces. 2472 2473* Anatol Belski <anbelski@linux.microsoft.com> 2474* Bo Chen <chen.bo@intel.com> 2475* Daniel Verkamp <dverkamp@chromium.org> 2476* Henry Wang <Henry.Wang@arm.com> 2477* Hui Zhu <teawater@antfin.com> 2478* Jiangbo Wu <jiangbo.wu@intel.com> 2479* Josh Soref <jsoref@users.noreply.github.com> 2480* Julio Montes <julio.montes@intel.com> 2481* Michael Zhao <michael.zhao@arm.com> 2482* Muminul Islam <muislam@microsoft.com> 2483* pierwill <19642016+pierwill@users.noreply.github.com> 2484* Praveen Paladugu <prapal@microsoft.com> 2485* Rob Bradford <robert.bradford@intel.com> 2486* Sebastien Boeuf <sebastien.boeuf@intel.com> 2487* Wei Liu <liuwe@microsoft.com> 2488 2489 2490# v0.10.0 2491 2492This release has been tracked through the [0.10.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/13). 2493 2494Highlights for `cloud-hypervisor` version 0.10.0 include: 2495 2496### `virtio-block` Support for Multiple Descriptors 2497 2498Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers. 2499 2500### Memory Zones 2501 2502Support has been added for fine-grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md). 2503 2504### `Seccomp` Sandbox Improvements 2505 2506All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`. 2507 2508### Preliminary KVM HyperV Emulation Control 2509 2510A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices. 2511 2512### Notable Bug Fixes 2513 2514- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596) 2515- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564) 2516- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593) 2517- `--version` shows the version for released binaries (#1669) 2518- Errors generated by worker threads for `virtio` devices are now printed out (#1551) 2519 2520### Contributors 2521 2522Many thanks to everyone who has contributed to our 0.10.0 release including some new faces. 2523 2524* Alyssa Ross <hi@alyssa.is> 2525* Amey Narkhede <ameynarkhede02@gmail.com> 2526* Anatol Belski <ab@php.net> 2527* Bo Chen <chen.bo@intel.com> 2528* Hui Zhu <teawater@antfin.com> 2529* Michael Zhao <michael.zhao@arm.com> 2530* Muminul Islam <muislam@microsoft.com> 2531* Rob Bradford <robert.bradford@intel.com> 2532* Samuel Ortiz <sameo@linux.intel.com> 2533* Sebastien Boeuf <sebastien.boeuf@intel.com> 2534* Wei Liu <liuwe@microsoft.com> 2535 2536# v0.9.0 2537 2538This release has been tracked through the [0.9.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/12). 2539 2540Highlights for `cloud-hypervisor` version 0.9.0 include: 2541 2542### `io_uring` Based Block Device Support 2543 2544If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement. 2545 2546### Block and Network Device Statistics 2547 2548Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM. 2549 2550### HTTP API Responses 2551 2552The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF. 2553 2554### CPU Topology 2555 2556A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core. 2557 2558### Release Build Optimization 2559 2560Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size. 2561 2562### Hypervisor Abstraction 2563 2564A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`. 2565 2566### Snapshot/Restore Improvements 2567 2568Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs. 2569 2570### Virtio Memory Ballooning Support 2571 2572A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest. 2573 2574### Enhancements to ARM64 Support 2575 2576The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting. 2577 2578### Intel SGX Support 2579 2580The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md). 2581 2582### `Seccomp` Sandbox Improvements 2583 2584The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development. 2585 2586### Notable Bug Fixes 2587 2588* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes. 2589* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338) 2590* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543). 2591* `virtio-mmio` based devices are now more widely tested (#275). 2592* Multiple issues have been fixed with virtio device configuration (#1217) 2593* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521) 2594 2595### Contributors 2596 2597Many thanks to everyone who has contributed to our 0.9.0 release including some new faces. 2598 2599* Anatol Belski <ab@php.net> 2600* Bo Chen <chen.bo@intel.com> 2601* Dr. David Alan Gilbert <dgilbert@redhat.com> 2602* Henry Wang <Henry.Wang@arm.com> 2603* Howard Zhang <howard.zhang@arm.com> 2604* Hui Zhu <teawater@antfin.com> 2605* Jianyong Wu <jianyong.wu@arm.com> 2606* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2607* LiYa'nan <oliverliyn@gmail.com> 2608* Michael Zhao <michael.zhao@arm.com> 2609* Muminul Islam <muislam@microsoft.com> 2610* Praveen Paladugu <prapal@microsoft.com> 2611* Ricardo Koller <ricarkol@gmail.com> 2612* Rob Bradford <robert.bradford@intel.com> 2613* Samuel Ortiz <sameo@linux.intel.com> 2614* Sebastien Boeuf <sebastien.boeuf@intel.com> 2615* Stefano Garzarella <sgarzare@redhat.com> 2616* Wei Liu <liuwe@microsoft.com> 2617 2618 2619# v0.8.0 2620 2621This release has been tracked through the [0.8.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/10). 2622 2623Highlights for `cloud-hypervisor` version 0.8.0 include: 2624 2625### Experimental Snapshot and Restore Support 2626 2627This release includes the first version of the snapshot and restore feature. 2628This allows a VM to be paused and then subsequently snapshotted. At a later 2629point that snapshot may be restored into a new running VM identical to the 2630original VM at the point it was paused. 2631 2632This feature can be used for offline migration from one VM host to another, to 2633allow the upgrading or rebooting of the host machine transparently to the guest 2634or for templating the VM. This is an experimental feature and cannot be used on 2635a VM using passthrough (VFIO) devices. Issues with SMP have also been observed 2636(#1176). 2637 2638### Experimental ARM64 Support 2639 2640Included in this release is experimental support for running on ARM64. 2641Currently only `virtio-mmio` devices and a serial port are supported. Full 2642details can be found in the [ARM64 documentation](docs/arm64.md). 2643 2644### Support for Using 5-level Paging in Guests 2645 2646If the host supports it the guest is now enabled for 5-level paging (aka LA57). 2647This works when booting the Linux kernel with a vmlinux, bzImage or firmware 2648based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not 2649work due to current limitations in the PVH boot process. 2650 2651### Virtio Device Interrupt Suppression for Network Devices 2652 2653With `virtio-net` and `vhost-user-net` devices the guest can suppress 2654interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This 2655can lead to an improvement in performance by reducing the number of interrupts 2656the guest must service. 2657 2658### `vhost_user_fs` Improvements 2659 2660The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`. 2661 2662 2663### Notable Bug Fixes 2664 2665* VMs that have not yet been booted can now be deleted (#1110). 2666* By creating the `tap` device ahead of creating the VM it is not required to 2667 run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273). 2668* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to 2669 the specification and synchronizes to the underlying filesystem as required 2670 based on guest feature negotiation. This avoids potential data loss (#399, 2671 #1216). 2672* When booting with a large number of vCPUs then the ACPI table would be 2673 overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the 2674 `MPTABLE` will no longer be generated (#1132). 2675* Shutting down VMs that have been paused is now supported (#816). 2676* Created socket files are deleted on shutdown (#1083). 2677* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds 2678 (#751). 2679 2680### Command Line and API Changes 2681 2682This is non exhaustive list of HTTP API and command line changes: 2683 2684* All user visible socket parameters are now consistently called `socket` 2685 rather than `sock` in some cases. 2686* The `ch-remote` tool now shows any error message generated by the VMM 2687* The `wce` parameter has been removed from `--disk` as the feature is always 2688 offered for negotiation. 2689* `--net` has gained a `host_mac` option that allows the setting of the MAC 2690 address for the `tap` device on the host. 2691 2692### Contributors 2693 2694Many thanks to everyone who has contributed to our 0.8.0 release including some new faces. 2695 2696* Anatol Belski <ab@php.net> 2697* Arron Wang <arron.wang@intel.com> 2698* Bo Chen <chen.bo@intel.com> 2699* Dr. David Alan Gilbert <dgilbert@redhat.com> 2700* Henry Wang <Henry.Wang@arm.com> 2701* Hui Zhu <teawater@antfin.com> 2702* LiYa'nan <oliverliyn@gmail.com> 2703* Michael Zhao <michael.zhao@arm.com> 2704* Rob Bradford <robert.bradford@intel.com> 2705* Samuel Ortiz <sameo@linux.intel.com> 2706* Sebastien Boeuf <sebastien.boeuf@intel.com> 2707* Sergio Lopez <slp@redhat.com> 2708 2709# v0.7.0 2710 2711This release has been tracked through the [0.7.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 2712 2713Highlights for `cloud-hypervisor` version 0.7.0 include: 2714 2715### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug 2716 2717Further to our effort to support modifying a running guest we now support 2718hotplug and unplug of the following virtio backed devices: block, network, 2719pmem, virtio-fs and vsock. This functionality is available on the (default) PCI 2720based transport and is exposed through the HTTP API. The `ch-remote` utility 2721provides a CLI for adding or removing these device types after the VM has 2722booted. User can use the `id` parameter on the devices to choose names for 2723devices to ease their removal. 2724 2725### Alternative `libc` Support 2726 2727Cloud Hypervisor can now be compiled with the `musl` C library and this release 2728contains a static binary compiled using that toolchain. 2729 2730### Multithreaded Multi Queued `vhost-user` Backends 2731 2732The `vhost-user` backends for network and block support that are shipped by 2733Cloud Hypervisor have been enhanced to support multiple threads and queues to 2734improve throughput. These backends are used automatically if `vhost_user=true` 2735is passed when the devices are created. 2736 2737### Initial RamFS Support 2738 2739By passing the `--initramfs` command line option the user can specify a file to 2740be loaded into the guest memory to be used as the kernel initial filesystem. 2741This is usually used to allow the loading of drivers needed to be able to 2742access the real root filesystem but it can also be used standalone for a very 2743minimal image. 2744 2745### Alternative Memory Hotplug: `virtio-mem` 2746 2747As well as supporting ACPI based hotplug Cloud Hypervisor now supports using 2748the `virtio-mem` hotplug alternative. This can be controlled by the 2749`hotplug_method` parameter on the `--memory` command line option. It currently 2750requires kernel patches to be able to support it. 2751 2752### `Seccomp` Sandboxing 2753 2754Cloud Hypervisor now has support for restricting the system calls that the 2755process can use via the `seccomp` security API. This on by default and is 2756controlled by the `--seccomp` command line option. 2757 2758### Updated Distribution Support 2759 2760With the release of Ubuntu 20.04 we have added that to the list of supported 2761distributions and is part of our regular testing programme. 2762 2763### Command Line and API Changes 2764 2765This is non exhaustive list of HTTP API and command line changes 2766 2767* New `id` fields added for devices to allow them to be named to ease removal. 2768 If no name is specified the VMM chooses one. 2769* Use `--memory`'s `shared` and `hugepages` controls for determining backing 2770 memory instead of providing a path. 2771* The `--vsock` parameter only takes one device as the Linux kernel only 2772 supports a single Vsock device. The REST API has removed the vector for this 2773 option and replaced it with a single optional field. 2774* There is enhanced validation of the command line and API provided 2775 configurations to ensure that the provided options are compatible e.g. that 2776 shared memory is in use if any attempt is made to used a `vhost-user` backed 2777 device. 2778* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and 2779 `add-vsock` subcommands. For removal `remove-device` is used. The REST API 2780 has appropriate new HTTP endpoints too. 2781* Specifying a `size` with `--pmem` is no longer required and instead the size 2782 will be obtained from the file. A `discard_writes` option has also been added 2783 to provide the equivalent of a read-only file. 2784* The parameters to `--block-backend` have been changed to more closely align 2785 with those used by `--disk`. 2786 2787### Contributors 2788 2789Many thanks to everyone who has contributed to our 0.7.0 release including some new faces. 2790 2791* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 2792* Bo Chen <chen.bo@intel.com> 2793* Cathy Zhang <cathy.zhang@intel.com> 2794* Damjan Georgievski <gdamjan@gmail.com> 2795* Dean Sheather <dean@coder.com> 2796* Eryu Guan <eguan@linux.alibaba.com> 2797* Hui Zhu <teawater@antfin.com> 2798* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2799* Martin Xu <martin.xu@intel.com> 2800* Muminul Islam <muislam@microsoft.com> 2801* Rob Bradford <robert.bradford@intel.com> 2802* Samuel Ortiz <sameo@linux.intel.com> 2803* Sebastien Boeuf <sebastien.boeuf@intel.com> 2804* Sergio Lopez <slp@redhat.com> 2805* Yang Zhong <yang.zhong@intel.com> 2806* Yi Sun <yi.y.sun@linux.intel.com> 2807 2808# v0.6.0 2809 2810This release has been tracked through the [0.6.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 2811 2812Highlights for `cloud-hypervisor` version 0.6.0 include: 2813 2814### Directly Assigned Devices Hotplug 2815 2816We continued our efforts around supporting dynamically changing the guest 2817resources. After adding support for CPU and memory hotplug, Cloud Hypervisor 2818now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`) 2819devices into an already running guest. This closes the features gap for 2820providing a complete Kata Containers workloads support with Cloud Hypervisor. 2821 2822### Shared Filesystem Improvements 2823 2824We enhanced our shared filesystem support through many `virtio-fs` improvements. 2825By adding support for DAX, parallel processing of multiple requests, `FS_IO`, 2826`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we 2827improved our filesystem sharing performance, but also made it more stable and 2828compatible with other `virtio-fs` implementations. 2829 2830### Block and Networking IO Self Offloading 2831 2832When choosing to offload the paravirtualized block and networking I/O to an 2833external process (through the `vhost-user` protocol), Cloud Hypervisor now 2834automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends 2835into their own, separate processes. 2836This provides a seamless paravirtualized I/O user experience for those who want 2837to run their guest I/O into separate executions contexts. 2838 2839### Command Line Interface 2840 2841More and more Cloud Hypervisor services are exposed through the 2842[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only 2843accessible via relatively cumbersome HTTP calls. In order to abstract 2844those calls into a more user friendly tool, we created a Cloud Hypervisor 2845Command Line Interface (CLI) called `ch-remote`. The `ch-remote` binary 2846is created with each build and available e.g. at 2847`cloud-hypervisor/target/debug/ch-remote` when doing a debug build. 2848 2849Please check `ch-remote --help` for a complete description of all available 2850commands. 2851 2852### PVH Boot 2853 2854In addition to the traditional Linux boot protocol, Cloud Hypervisor now 2855supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html). 2856 2857### Contributors 2858 2859With the 0.6.0 release, we are welcoming a few new contributors. Many thanks 2860to them and to everyone that contributed to this release: 2861 2862* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 2863* Arron Wang <arron.wang@intel.com> 2864* Bin Liu <liubin0329@gmail.com> 2865* Bo Chen <chen.bo@intel.com> 2866* Cathy Zhang <cathy.zhang@intel.com> 2867* Eryu Guan <eguan@linux.alibaba.com> 2868* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2869* Liu Bo <bo.liu@linux.alibaba.com> 2870* Qiu Wenbo <qiuwenbo@phytium.com.cn> 2871* Rob Bradford <robert.bradford@intel.com> 2872* Samuel Ortiz <sameo@linux.intel.com> 2873* Sebastien Boeuf <sebastien.boeuf@intel.com> 2874* Sergio Lopez <slp@redhat.com> 2875 2876# v0.5.1 2877 2878This is a bugfix release branched off v0.5.0. It contains the following fixes: 2879 2880* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez 2881* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf 2882* Fixed error reporting for resize command - Samuel Ortiz 2883* Double reboot workaround (#783) - Rob Bradford 2884* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford 2885 2886# v0.5.0 2887 2888This release has been tracked through the [0.5.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/6). 2889 2890Highlights for `cloud-hypervisor` version 0.5.0 include: 2891 2892### Virtual Machine Dynamic Resizing 2893 2894With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and 2895memory hot plug as well. This allows to dynamically resize Cloud Hypervisor 2896guests which is needed for e.g. Kubernetes related use cases. 2897The memory hot plug implementation is based on the same framework as the CPU hot 2898plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest. 2899 2900Next on our VM resizing roadmap is the PCI devices hotplug feature. 2901 2902### Multi-Queue, Multi-Threaded Paravirtualization 2903 2904We enhanced our virtio networking and block support by having both devices use 2905multiple I/O queues handled by multiple threads. This improves our default 2906paravirtualized networking and block devices throughput. 2907 2908### New Interrupt Management Framework 2909 2910We improved our interrupt management implementation by introducing an Interrupt 2911Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device) 2912crates discussions. This move made the code significantly cleaner, and allowed 2913us to remove several KVM related dependencies from crates like the PCI and 2914virtio ones. 2915 2916### Development Tools 2917 2918In order to provide a better developer experience, we worked on improving our 2919build, development and testing tools. 2920Somehow similar to the excellent 2921[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool), 2922we now provide a [dev_cli script](scripts/dev_cli.sh). 2923 2924With this new tool, our users and contributors will be able to build and test 2925Cloud Hypervisor through a containerized environment. 2926 2927### Kata Containers Integration 2928 2929We spent some significant time and efforts debugging and fixing our integration 2930with the [Kata Containers](https://github.com/kata-containers) project. Cloud 2931Hypervisor is now a fully supported Kata Containers hypervisor, and is 2932integrated into the project's CI. 2933 2934### Contributors 2935 2936Many thanks to everyone that contributed to the 0.5.0 release: 2937 2938* Bo Chen <chen.bo@intel.com> 2939* Cathy Zhang <cathy.zhang@intel.com> 2940* Qiu Wenbo <qiuwenbo@phytium.com.cn> 2941* Rob Bradford <robert.bradford@intel.com> 2942* Samuel Ortiz <sameo@linux.intel.com> 2943* Sebastien Boeuf <sebastien.boeuf@intel.com> 2944* Sergio Lopez <slp@redhat.com> 2945* Yang Zhong <yang.zhong@intel.com> 2946 2947# v0.4.0 2948 2949This release has been tracked through the [0.4.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/4). 2950 2951Highlights for `cloud-hypervisor` version 0.4.0 include: 2952 2953### Dynamic virtual CPUs addition 2954 2955As a way to vertically scale Cloud Hypervisor guests, we now support dynamically 2956adding virtual CPUs to the guests, a mechanism also known as CPU hot plug. 2957Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs 2958to an already running guest and the high level operations for that process are 2959documented [here](docs/hotplug.md) 2960 2961During the next release cycles we are planning to extend Cloud Hypervisor 2962hot plug framework to other resources, namely PCI devices and memory. 2963 2964### Programmatic firmware tables generation 2965 2966As part of the CPU hot plug feature enablement, and as a requirement for hot 2967plugging other resources like devices or RAM, we added support for 2968programmatically generating the needed ACPI tables. Through a dedicated 2969`acpi-tables` crate, we now have a flexible and clean way of generating those 2970tables based on the VMM device model and topology. 2971 2972### Filesystem and block devices vhost-user backends 2973 2974Our objective of running all Cloud Hypervisor paravirtualized I/O to a 2975vhost-user based framework is getting closer as we've added Rust based 2976implementations for vhost-user-blk and virtiofs backends. Together with the 2977vhost-user-net backend that came with the 0.3.0 release, this will form the 2978default Cloud Hypervisor I/O architecture. 2979 2980### Guest pause and resume 2981 2982As an initial requirement for enabling live migration, we added support for 2983pausing and resuming any VMM components. As an intermediate step towards live 2984migration, the upcoming guest snapshotting feature will be based on the pause 2985and resume capabilities. 2986 2987### Userspace IOAPIC by default 2988 2989As a way to simplify our device manager implementation, but also in order to 2990stay away from privileged rings as often as possible, any device that relies on 2991pin based interrupts will be using the userspace IOAPIC implementation by 2992default. 2993 2994### PCI BAR reprogramming 2995 2996In order to allow for a more flexible device model, and also support guests 2997that would want to move PCI devices, we added support for PCI devices BAR 2998reprogramming. 2999 3000### New `cloud-hypervisor` organization 3001 3002As we wanted to be more flexible on how we manage the Cloud Hypervisor project, 3003we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor). 3004Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) 3005project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux) 3006and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 3007repositories. We may also use it to host any rust-vmm that we'd need to 3008temporarily fork. 3009Thanks to GitHub's seamless repository redirections, the move is completely 3010transparent to all Cloud Hypervisor contributors, users and followers. 3011 3012### Contributors 3013 3014Many thanks to everyone that contributed to the 0.4.0 release: 3015 3016* Cathy Zhang <cathy.zhang@intel.com> 3017* Emin Ghuliev <drmint80@gmail.com> 3018* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 3019* Qiu Wenbo <qiuwenbo@phytium.com.cn> 3020* Rob Bradford <robert.bradford@intel.com> 3021* Samuel Ortiz <sameo@linux.intel.com> 3022* Sebastien Boeuf <sebastien.boeuf@intel.com> 3023* Sergio Lopez <slp@redhat.com> 3024* Wu Zongyong <wuzongyong@linux.alibaba.com> 3025 3026# v0.3.0 3027 3028This release has been tracked through the [0.3.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/3). 3029 3030Highlights for `cloud-hypervisor` version 0.3.0 include: 3031 3032### Block device offloading 3033 3034We continue to work on offloading paravirtualized I/O to external processes, 3035and we added support for 3036[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends. 3037This enables `cloud-hypervisor` users to plug a `vhost-user` based block device 3038like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage 3039backend. 3040 3041### Network device backend 3042 3043The previous release provided support for 3044[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we 3045also provide a TAP based vhost-user-net backend, implemented in Rust. Together 3046with the vhost-user-net device implementation, this will eventually become the 3047Cloud Hypervisor default paravirtualized networking architecture. 3048 3049### Virtual sockets 3050 3051In order to more efficiently and securely communicate between host and guest, 3052we added an hybrid implementation of the 3053[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address 3054family over virtio. Credits go to the 3055[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md) 3056project as our implementation is a copy of theirs. 3057 3058### HTTP based API 3059 3060In anticipation of the need to support asynchronous operations to Cloud 3061Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP 3062based API to the VMM. The API will be more extensively documented during the 3063next release cycle. 3064 3065### Memory mapped virtio transport 3066 3067In order to support potential PCI-free use cases, we added support for the 3068[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002) 3069transport layer. This will allow us to support simple, minimal guest 3070configurations that do not require a PCI bus emulation. 3071 3072### Paravirtualized IOMMU 3073 3074As we want to improve our nested guests support, we added support for exposing 3075a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows 3076for a safer nested virtio and directly assigned devices support. 3077 3078To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor 3079users to be able to specify if devices had to be handled through this virtual 3080IOMMU or not. In particular, the `--disk` option now expects disk paths to be 3081prefixed with a `path=` string, and supports an optional `iommu=[on|off]` 3082setting. 3083 3084### Ubuntu 19.10 3085 3086With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware), 3087we can now support the latest 3088[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images. 3089 3090### Large memory guests 3091 3092After simplifying and changing our guest address space handling, we can now 3093support guests with large amount of memory (more than 64GB). 3094 3095# v0.2.0 3096 3097This release has been tracked through the [0.2.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/2). 3098 3099Highlights for `cloud-hypervisor` version 0.2.0 include: 3100 3101### Network device offloading 3102 3103As part of our general effort to offload paravirtualized I/O to external 3104processes, we added support for 3105[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This 3106enables `cloud-hypervisor` users to plug a `vhost-user` based networking device 3107(e.g. [DPDK](https://dpdk.org)) into the VMM as their virtio network backend. 3108 3109### Minimal hardware-reduced ACPI 3110 3111In order to properly implement and guest reset and shutdown, we implemented 3112a minimal version of the hardware-reduced ACPI specification. Together with 3113a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to 3114cleanly reboot and shutdown. 3115 3116The ACPI implementation is a `cloud-hypervisor` build time option that is 3117enabled by default. 3118 3119### Debug I/O port 3120 3121Based on the Firecracker idea of using a dedicated I/O port to measure guest 3122boot times, we added support for logging guest events through the 3123[0x80](https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html) 3124PC debug port. This allows, among other things, for granular guest boot time 3125measurements. See our [debug port documentation](docs/debug-port.md) for more 3126details. 3127 3128### Improved direct device assignment 3129 3130We fixed a major performance issue with our initial VFIO implementation: When 3131enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads 3132were (in many cases) not cached. After correctly tagging the guest memory from 3133`cloud-hypervisor` we're now able to reach the expected performance from 3134directly assigned devices. 3135 3136### Improved shared filesystem 3137 3138We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt) 3139support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system. 3140This provides better shared filesystem IO performance with a smaller guest 3141memory footprint. 3142 3143### Ubuntu bionic based CI 3144 3145Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 3146improvements, we are now able to boot Ubuntu bionic images. We added those to 3147our CI pipeline. 3148 3149# v0.1.0 3150 3151This release has been tracked through the [0.1.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/1). 3152 3153Highlights for `cloud-hypervisor` version 0.1.0 include: 3154 3155### Shared filesystem 3156 3157We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file 3158system, allowing for an efficient and reliable way of sharing a filesystem 3159between the host and the `cloud-hypervisor` guest. 3160 3161See our [filesystem sharing](docs/fs.md) documentation for more details on how 3162to use virtio-fs with `cloud-hypervisor`. 3163 3164### Initial direct device assignment support 3165 3166VFIO (Virtual Function I/O) is a kernel framework that exposes direct device 3167access to userspace. `cloud-hypervisor` uses VFIO to directly assign host 3168physical devices into its guest. 3169 3170See our [VFIO](docs/vfio.md) documentation for more detail on how to directly 3171assign host devices to `cloud-hypervisor` guests. 3172 3173### Userspace IOAPIC 3174 3175`cloud-hypervisor` supports a so-called split IRQ chip implementation by 3176implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC). 3177By moving part of the IRQ chip implementation from kernel space to user space, 3178the IRQ chip emulation does not always run in a fully privileged mode. 3179 3180### Virtual persistent memory 3181 3182The `virtio-pmem` implementation emulates a virtual persistent memory device 3183that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device 3184allows to bypass the guest page cache and improve the guest memory footprint. 3185 3186### Linux kernel bzImage 3187 3188The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from 3189`bzImage` kernel images, which is usually the format that Linux distributions 3190use to ship their kernels. For example, this allows for booting from the host 3191distribution kernel image. 3192 3193### Console over virtio 3194 3195`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although 3196using this device as a guest console can potentially cut some early boot 3197messages, it can reduce the guest boot time and provides a complete console 3198implementation. 3199 3200The `virtio-console` device is enabled by default for the guest console. 3201Switching back to the legacy serial port is done by selecting 3202`--serial tty --console off` from the command line. 3203 3204### Unit testing 3205 3206We now run all unit tests from all our crates directly from our CI. 3207 3208### Integration tests parallelization 3209 3210The CI cycle run time has been significantly reduced by refactoring our 3211integration tests; allowing them to all be run in parallel. 3212