1- [v46.0](#v460) 2 - [File-level Locking Support with `--disk`](#file-level-locking-support-with---disk) 3 - [Improved Error Reporting with VM Resizing](#improved-error-reporting-with-vm-resizing) 4 - [IPv6 Address Support with `--net`](#ipv6-address-support-with---net) 5 - [Experimental AArch64 Support with the MSHV Hypervisor](#experimental-aarch64-support-with-the-mshv-hypervisor) 6 - [Deprecated SGX Support](#deprecated-sgx-support) 7 - [Notable Bug Fixes](#notable-bug-fixes) 8 - [Contributors](#contributors) 9- [v45.0](#v450) 10 - [Experimental `riscv64` Architecture Support](#experimental-riscv64-architecture-support) 11 - [Alphabetically Sorted CLI Options](#alphabetically-sorted-cli-options) 12 - [Improved Downtime of VM Live Migration](#improved-downtime-of-vm-live-migration) 13 - [Notable Bug Fixes](#notable-bug-fixes-1) 14 - [Contributors](#contributors-1) 15- [v44.0](#v440) 16 - [Configurable `virtio-iommu` Address Width](#configurable-virtio-iommu-address-width) 17 - [Notable Performance Improvements](#notable-performance-improvements) 18 - [New Fuzzers](#new-fuzzers) 19 - [Notable Bug Fixes](#notable-bug-fixes-2) 20 - [Contributors](#contributors-2) 21- [v43.0](#v430) 22 - [Live Migration over TCP Connections](#live-migration-over-tcp-connections) 23 - [Notable Performance Improvements](#notable-performance-improvements-1) 24 - [Notable Bug Fixes](#notable-bug-fixes-3) 25 - [Contributors](#contributors-3) 26- [v42.0](#v420) 27 - [SVE/SVE2 Support on AArch64](#svesve2-support-on-aarch64) 28 - [Notable Bug Fixes](#notable-bug-fixes-4) 29 - [Sponsorships](#sponsorships) 30 - [Contributors](#contributors-4) 31- [v41.0](#v410) 32 - [Experimental "Pvmemcontrol" Support](#experimental-pvmemcontrol-support) 33 - [Sandboxing With Landlock Support](#sandboxing-with-landlock-support) 34 - [Notable Performance Improvements](#notable-performance-improvements-2) 35 - [Notable Bug Fixes](#notable-bug-fixes-5) 36 - [Contributors](#contributors-5) 37- [v40.0](#v400) 38 - [Support for Restoring File Descriptor Backed Network Devices](#support-for-restoring-file-descriptor-backed-network-devices) 39 - [Notable Bug Fixes](#notable-bug-fixes-6) 40 - [Contributors](#contributors-6) 41- [v39.0](#v390) 42 - [Variable Sizing of PCI Apertures for Segments](#variable-sizing-of-pci-apertures-for-segments) 43 - [Direct Booting with bzImages](#direct-booting-with-bzimages) 44 - [Support for NVIDIA GPUDirect P2P Support](#support-for-nvidia-gpudirect-p2p-support) 45 - [Guest NMI Injection Support](#guest-nmi-injection-support) 46 - [Notable Bug Fixes](#notable-bug-fixes-7) 47 - [Contributors](#contributors-7) 48- [v38.0](#v380) 49 - [Group Rate Limiter on Block Devices](#group-rate-limiter-on-block-devices) 50 - [CPU Pinning Support for Block Device Worker Thread](#cpu-pinning-support-for-block-device-worker-thread) 51 - [Optimized Boot Time with Parallel Memory Prefault](#optimized-boot-time-with-parallel-memory-prefault) 52 - [New 'debug-console' Device](#new-debug-console-device) 53 - [Improved VFIO Device Support](#improved-vfio-device-support) 54 - [Extended CPU Affinity Support](#extended-cpu-affinity-support) 55 - [Notable Bug Fixes](#notable-bug-fixes-8) 56 - [Contributors](#contributors-8) 57- [v37.0](#v370) 58 - [Long Term Support (LTS) Release](#long-term-support-lts-release) 59 - [Multiple PCI segments Support for 32-bit VFIO devices](#multiple-pci-segments-support-for-32-bit-vfio-devices) 60 - [Configurable Named TAP Devices](#configurable-named-tap-devices) 61 - [TTY Output from Both Serial Device and Virtio Console](#tty-output-from-both-serial-device-and-virtio-console) 62 - [Faster VM Restoration from Snapshots](#faster-vm-restoration-from-snapshots) 63 - [Notable Bug Fixes](#notable-bug-fixes-9) 64 - [Contributors](#contributors-9) 65- [v36.0](#v360) 66 - [Command Line Changes](#command-line-changes) 67 - [Enabled Features Reported via API Endpoint and CLI](#enabled-features-reported-via-api-endpoint-and-cli) 68 - [NUMA Support for PCI segments](#numa-support-for-pci-segments) 69 - [CPU Topology Support on AMD Platforms](#cpu-topology-support-on-amd-platforms) 70 - [Unix Socket Backend for Serial Port](#unix-socket-backend-for-serial-port) 71 - [AIO Backend for Block Devices](#aio-backend-for-block-devices) 72 - [Documentation Improvements](#documentation-improvements) 73 - [Notable Bug Fixes](#notable-bug-fixes-10) 74 - [Contributors](#contributors-10) 75- [v35.0](#v350) 76 - [`virtio-vsock` Support for Linux Guest Kernel v6.3+](#virtio-vsock-support-for-linux-guest-kernel-v63) 77 - [User Specified Serial Number for `virtio-block`](#user-specified-serial-number-for-virtio-block) 78 - [vCPU TSC Frequency Included in Migration State](#vcpu-tsc-frequency-included-in-migration-state) 79 - [Notable Bug Fixes](#notable-bug-fixes-11) 80 - [Contributors](#contributors-11) 81- [v34.0](#v340) 82 - [Paravirtualised Panic Device Support](#paravirtualised-panic-device-support) 83 - [Improvements to VM Core Dump](#improvements-to-vm-core-dump) 84 - [QCOW2 Support for Backing Files](#qcow2-support-for-backing-files) 85 - [Minimum Host Kernel Bump](#minimum-host-kernel-bump) 86 - [Notable Bug Fixes](#notable-bug-fixes-12) 87 - [Contributors](#contributors-12) 88- [v33.0](#v330) 89 - [D-Bus based API](#d-bus-based-api) 90 - [Expose Host CPU Cache Details for AArch64](#expose-host-cpu-cache-details-for-aarch64) 91 - [Notable Bug Fixes](#notable-bug-fixes-13) 92 - [Contributors](#contributors-13) 93- [v32.0](#v320) 94 - [Increased PCI Segment Limit](#increased-pci-segment-limit) 95 - [API Changes](#api-changes) 96 - [Notable Bug Fixes](#notable-bug-fixes-14) 97 - [Contributors](#contributors-14) 98- [v31.1](#v311) 99- [v31.0](#v310) 100 - [Update to Latest `acpi_tables`](#update-to-latest-acpi_tables) 101 - [Update Reference Kernel to 6.2](#update-reference-kernel-to-62) 102 - [Improvements on Console `SIGWINCH` Handler](#improvements-on-console-sigwinch-handler) 103 - [Remove Directory Support from `MemoryZoneConfig::file`](#remove-directory-support-from-memoryzoneconfigfile) 104 - [Documentation Improvements](#documentation-improvements-1) 105 - [Notable Bug Fixes](#notable-bug-fixes-15) 106 - [Contributors](#contributors-15) 107- [v30.0](#v300) 108 - [Command Line Changes for Reduced Binary Size](#command-line-changes-for-reduced-binary-size) 109 - [Basic vfio-user Server Support](#basic-vfio-user-server-support) 110 - [Heap Profiling Support](#heap-profiling-support) 111 - [Documentation Improvements](#documentation-improvements-2) 112 - [Notable Bug Fixes](#notable-bug-fixes-16) 113 - [Contributors](#contributors-16) 114- [v28.2](#v282) 115- [v29.0](#v290) 116 - [Release Binary Supports Both MSHV and KVM](#release-binary-supports-both-mshv-and-kvm) 117 - [Snapshot/Restore and Live Migration Improvements](#snapshotrestore-and-live-migration-improvements) 118 - [Heap Allocation Improvements](#heap-allocation-improvements) 119 - [`ch-remote` Improvements](#ch-remote-improvements) 120 - [`AArch64` Documentation Integration](#aarch64-documentation-integration) 121 - [`virtio-block` Counters Enhancement](#virtio-block-counters-enhancement) 122 - [TCP Offload Control](#tcp-offload-control) 123 - [Notable Bug Fixes](#notable-bug-fixes-17) 124 - [Removals](#removals) 125 - [Deprecations](#deprecations) 126 - [Contributors](#contributors-17) 127- [v28.1](#v281) 128- [v28.0](#v280) 129 - [Community Engagement (Reminder)](#community-engagement-reminder) 130 - [Long Term Support (LTS) Release](#long-term-support-lts-release-1) 131 - [Virtualised TPM Support](#virtualised-tpm-support) 132 - [Transparent Huge Page Support](#transparent-huge-page-support) 133 - [README Quick Start Improved](#readme-quick-start-improved) 134 - [Notable Bug Fixes](#notable-bug-fixes-18) 135 - [Removals](#removals-1) 136 - [Contributors](#contributors-18) 137- [v27.0](#v270) 138 - [Community Engagement](#community-engagement) 139 - [Prebuilt Packages](#prebuilt-packages) 140 - [Network Device MTU Exposed to Guest](#network-device-mtu-exposed-to-guest) 141 - [Boot Tracing](#boot-tracing) 142 - [Simplified Build Feature Flags](#simplified-build-feature-flags) 143 - [Asynchronous Kernel Loading](#asynchronous-kernel-loading) 144 - [GDB Support for AArch64](#gdb-support-for-aarch64) 145 - [Notable Bug Fixes](#notable-bug-fixes-19) 146 - [Deprecations](#deprecations-1) 147 - [Contributors](#contributors-19) 148- [v26.0](#v260) 149 - [SMBIOS Improvements via `--platform`](#smbios-improvements-via---platform) 150 - [Unified Binary MSHV and KVM Support](#unified-binary-mshv-and-kvm-support) 151 - [Notable Bug Fixes](#notable-bug-fixes-20) 152 - [Deprecations](#deprecations-2) 153 - [Removals](#removals-2) 154 - [Contributors](#contributors-20) 155- [v25.0](#v250) 156 - [`ch-remote` Improvements](#ch-remote-improvements-1) 157 - [VM "Coredump" Support](#vm-coredump-support) 158 - [Notable Bug Fixes](#notable-bug-fixes-21) 159 - [Removals](#removals-3) 160 - [Contributors](#contributors-21) 161- [v24.0](#v240) 162 - [Bypass Mode for `virtio-iommu`](#bypass-mode-for-virtio-iommu) 163 - [Ensure Identifiers Uniqueness](#ensure-identifiers-uniqueness) 164 - [Sparse Mmap support](#sparse-mmap-support) 165 - [Expose Platform Serial Number](#expose-platform-serial-number) 166 - [Notable Bug Fixes](#notable-bug-fixes-22) 167 - [Notable Improvements](#notable-improvements) 168 - [Deprecations](#deprecations-3) 169 - [New on the Website](#new-on-the-website) 170 - [Contributors](#contributors-22) 171- [v23.1](#v231) 172- [v23.0](#v230) 173 - [vDPA Support](#vdpa-support) 174 - [Updated OS Support list](#updated-os-support-list) 175 - [`AArch64` Memory Map Improvements](#aarch64-memory-map-improvements) 176 - [`AMX` Support](#amx-support) 177 - [Notable Bug Fixes](#notable-bug-fixes-23) 178 - [Deprecations](#deprecations-4) 179 - [Contributors](#contributors-23) 180- [v22.1](#v221) 181- [v22.0](#v220) 182 - [GDB Debug Stub Support](#gdb-debug-stub-support) 183 - [`virtio-iommu` Backed Segments](#virtio-iommu-backed-segments) 184 - [Before Boot Configuration Changes](#before-boot-configuration-changes) 185 - [`virtio-balloon` Free Page Reporting](#virtio-balloon-free-page-reporting) 186 - [Support for Direct Kernel Booting with TDX](#support-for-direct-kernel-booting-with-tdx) 187 - [PMU Support for AArch64](#pmu-support-for-aarch64) 188 - [Documentation Under CC-BY-4.0 License](#documentation-under-cc-by-40-license) 189 - [Deprecation of "Classic" `virtiofsd`](#deprecation-of-classic-virtiofsd) 190 - [Notable Bug Fixes](#notable-bug-fixes-24) 191 - [Contributors](#contributors-24) 192- [v21.0](#v210) 193 - [Efficient Local Live Migration (for Live Upgrade)](#efficient-local-live-migration-for-live-upgrade) 194 - [Recommended Kernel is Now 5.15](#recommended-kernel-is-now-515) 195 - [Notable Bug fixes](#notable-bug-fixes-25) 196 - [Contributors](#contributors-25) 197- [v20.2](#v202) 198- [v20.1](#v201) 199- [v20.0](#v200) 200 - [Multiple PCI segments support](#multiple-pci-segments-support) 201 - [CPU pinning](#cpu-pinning) 202 - [Improved VFIO support](#improved-vfio-support) 203 - [Safer code](#safer-code) 204 - [Extended documentation](#extended-documentation) 205 - [Notable bug fixes](#notable-bug-fixes-26) 206 - [Contributors](#contributors-26) 207- [v19.0](#v190) 208 - [Improved PTY handling for serial and `virtio-console`](#improved-pty-handling-for-serial-and-virtio-console) 209 - [PCI boot time optimisations](#pci-boot-time-optimisations) 210 - [Improved TDX support](#improved-tdx-support) 211 - [Live migration enhancements](#live-migration-enhancements) 212 - [`virtio-mem` support with `vfio-user`](#virtio-mem-support-with-vfio-user) 213 - [AArch64 for `virtio-iommu`](#aarch64-for-virtio-iommu) 214 - [Notable bug fixes](#notable-bug-fixes-27) 215 - [Contributors](#contributors-27) 216- [v18.0](#v180) 217 - [Experimental User Device (`vfio-user`) support](#experimental-user-device-vfio-user-support) 218 - [Migration support for `vhost-user` devices](#migration-support-for-vhost-user-devices) 219 - [VHDX disk image support](#vhdx-disk-image-support) 220 - [Device pass through on MSHV hypervisor](#device-pass-through-on-mshv-hypervisor) 221 - [AArch64 for support `virtio-mem`](#aarch64-for-support-virtio-mem) 222 - [Live migration on MSHV hypervisor](#live-migration-on-mshv-hypervisor) 223 - [AArch64 CPU topology support](#aarch64-cpu-topology-support) 224 - [Power button support on AArch64](#power-button-support-on-aarch64) 225 - [Notable bug fixes](#notable-bug-fixes-28) 226 - [Contributors](#contributors-28) 227- [v17.0](#v170) 228 - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi) 229 - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend) 230 - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices) 231 - [Improved SGX support](#improved-sgx-support) 232 - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices) 233 - [Notable bug fixes](#notable-bug-fixes-29) 234 - [Contributors](#contributors-29) 235- [v16.0](#v160) 236 - [Improved live migration support](#improved-live-migration-support) 237 - [Improved `vhost-user` support](#improved-vhost-user-support) 238 - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support) 239 - [Notable bug fixes](#notable-bug-fixes-30) 240 - [Removed functionality](#removed-functionality) 241 - [Contributors](#contributors-30) 242- [v15.0](#v150) 243 - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees) 244 - [Network device rate limiting](#network-device-rate-limiting) 245 - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload) 246 - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter) 247 - [Bug fixes](#bug-fixes) 248 - [Deprecations](#deprecations-5) 249 - [Contributors](#contributors-31) 250- [v0.14.1](#v0141) 251- [v0.14.0](#v0140) 252 - [Structured event monitoring](#structured-event-monitoring) 253 - [MSHV improvements](#mshv-improvements) 254 - [Improved aarch64 platform](#improved-aarch64-platform) 255 - [Updated hotplug documentation](#updated-hotplug-documentation) 256 - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console) 257 - [Block device rate limiting](#block-device-rate-limiting) 258 - [Deprecations](#deprecations-6) 259 - [Contributors](#contributors-32) 260- [v0.13.0](#v0130) 261 - [Wider VFIO device support](#wider-vfio-device-support) 262 - [Improved huge page support](#improved-huge-page-support) 263 - [MACvTAP support](#macvtap-support) 264 - [VHD disk image support](#vhd-disk-image-support) 265 - [Improved Virtio device threading](#improved-virtio-device-threading) 266 - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button) 267 - [Contributors](#contributors-33) 268- [v0.12.0](#v0120) 269 - [ARM64 enhancements](#arm64-enhancements) 270 - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning) 271 - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend) 272 - [Enhanced "info" API](#enhanced-info-api) 273 - [Contributors](#contributors-34) 274- [v0.11.0](#v0110) 275 - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block) 276 - [Windows Guest Support](#windows-guest-support) 277 - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation) 278 - [`virtio-mmio` Removal](#virtio-mmio-removal) 279 - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64) 280 - [Improved Linux Boot Time](#improved-linux-boot-time) 281 - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling) 282 - [Default Log Level Changed](#default-log-level-changed) 283 - [New `--balloon` Parameter Added](#new---balloon-parameter-added) 284 - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support) 285 - [Notable Bug Fixes](#notable-bug-fixes-31) 286 - [Contributors](#contributors-35) 287- [v0.10.0](#v0100) 288 - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors) 289 - [Memory Zones](#memory-zones) 290 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements) 291 - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control) 292 - [Notable Bug Fixes](#notable-bug-fixes-32) 293 - [Contributors](#contributors-36) 294- [v0.9.0](#v090) 295 - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support) 296 - [Block and Network Device Statistics](#block-and-network-device-statistics) 297 - [HTTP API Responses](#http-api-responses) 298 - [CPU Topology](#cpu-topology) 299 - [Release Build Optimization](#release-build-optimization) 300 - [Hypervisor Abstraction](#hypervisor-abstraction) 301 - [Snapshot/Restore Improvements](#snapshotrestore-improvements) 302 - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support) 303 - [Enhancements to ARM64 Support](#enhancements-to-arm64-support) 304 - [Intel SGX Support](#intel-sgx-support) 305 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1) 306 - [Notable Bug Fixes](#notable-bug-fixes-33) 307 - [Contributors](#contributors-37) 308- [v0.8.0](#v080) 309 - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support) 310 - [Experimental ARM64 Support](#experimental-arm64-support) 311 - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests) 312 - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices) 313 - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements) 314 - [Notable Bug Fixes](#notable-bug-fixes-34) 315 - [Command Line and API Changes](#command-line-and-api-changes) 316 - [Contributors](#contributors-38) 317- [v0.7.0](#v070) 318 - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug) 319 - [Alternative `libc` Support](#alternative-libc-support) 320 - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends) 321 - [Initial RamFS Support](#initial-ramfs-support) 322 - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem) 323 - [`Seccomp` Sandboxing](#seccomp-sandboxing) 324 - [Updated Distribution Support](#updated-distribution-support) 325 - [Command Line and API Changes](#command-line-and-api-changes-1) 326 - [Contributors](#contributors-39) 327- [v0.6.0](#v060) 328 - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug) 329 - [Shared Filesystem Improvements](#shared-filesystem-improvements) 330 - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading) 331 - [Command Line Interface](#command-line-interface) 332 - [PVH Boot](#pvh-boot) 333 - [Contributors](#contributors-40) 334- [v0.5.1](#v051) 335- [v0.5.0](#v050) 336 - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing) 337 - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization) 338 - [New Interrupt Management Framework](#new-interrupt-management-framework) 339 - [Development Tools](#development-tools) 340 - [Kata Containers Integration](#kata-containers-integration) 341 - [Contributors](#contributors-41) 342- [v0.4.0](#v040) 343 - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition) 344 - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation) 345 - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends) 346 - [Guest pause and resume](#guest-pause-and-resume) 347 - [Userspace IOAPIC by default](#userspace-ioapic-by-default) 348 - [PCI BAR reprogramming](#pci-bar-reprogramming) 349 - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization) 350 - [Contributors](#contributors-42) 351- [v0.3.0](#v030) 352 - [Block device offloading](#block-device-offloading) 353 - [Network device backend](#network-device-backend) 354 - [Virtual sockets](#virtual-sockets) 355 - [HTTP based API](#http-based-api) 356 - [Memory mapped virtio transport](#memory-mapped-virtio-transport) 357 - [Paravirtualized IOMMU](#paravirtualized-iommu) 358 - [Ubuntu 19.10](#ubuntu-1910) 359 - [Large memory guests](#large-memory-guests) 360- [v0.2.0](#v020) 361 - [Network device offloading](#network-device-offloading) 362 - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi) 363 - [Debug I/O port](#debug-io-port) 364 - [Improved direct device assignment](#improved-direct-device-assignment) 365 - [Improved shared filesystem](#improved-shared-filesystem) 366 - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci) 367- [v0.1.0](#v010) 368 - [Shared filesystem](#shared-filesystem) 369 - [Initial direct device assignment support](#initial-direct-device-assignment-support) 370 - [Userspace IOAPIC](#userspace-ioapic) 371 - [Virtual persistent memory](#virtual-persistent-memory) 372 - [Linux kernel bzImage](#linux-kernel-bzimage) 373 - [Console over virtio](#console-over-virtio) 374 - [Unit testing](#unit-testing) 375 - [Integration tests parallelization](#integration-tests-parallelization) 376 377# v46.0 378 379This release has been tracked in [v46.0 380group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+46%22) 381of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 382 383### File-level Locking Support with `--disk` 384 385Now file-level locking is enforced for disk images, provided by users 386with `--disk`. This ensures that only a single Cloud Hypervisor instance 387can obtain write access to a given disk image at any time, preventing 388misconfiguration and avoiding potential data corruption. (#6974) 389 390### Improved Error Reporting with VM Resizing 391 392Instead of returning a generic error `400` (e.g. `BadRequest`), users 393now get a more specific error `429` (e.g. `TooManyRequests`) when a 394pending VM resizing is not completed. This allows users to better handle 395different errors, say retrying the request when applicable. (#7043) 396 397### IPv6 Address Support with `--net` 398 399It is now possible to specify an IPv6 address and mask when creating a 400network interface with `--net`. (#7048) 401 402### Experimental AArch64 Support with the MSHV Hypervisor 403 404It is now possible to start VMs on AArch64 platforms when using MSHV 405hypervisor. (#7055) 406 407### Deprecated SGX Support 408 409The SGX support now is deprecated with a warning message if it being 410used, with the intention to remove its support from our code base in two 411release cycles (e.g. v48.0). (#7090) 412 413### Notable Bug Fixes 414 415* Remove `path` as required for `DiskConfig` from the OpenAPI spec file 416 (#7017) 417* Properly parse PCI capabilities (#7018) 418* Reprogram PCI device BAR when its MSE bit is set (#7063) 419* Update IOMMU mappings of MMIO regions with BAR reprogram for VFIO 420 devices (#7064) 421* Avoid resizing VMs to zero vCPUs (#7086) 422* Fix extended topology enumeration leaf exposed to the guest (#7087) 423 424### Contributors 425 426Many thanks to everyone who has contributed to our release: 427 428* Bingxin Li <bl497@cam.ac.uk> 429* Bo Chen <bchen@crusoe.ai> 430* Fabiano Fidêncio <fidencio@northflank.com> 431* Gregory Anders <ganders@cloudflare.com> 432* Jinank Jain <jinankjain@microsoft.com> 433* Julian Stecklina <julian.stecklina@cyberus-technology.de> 434* Muminul Islam <muislam@microsoft.com> 435* Paolo Bonzini <pbonzini@redhat.com> 436* Philipp Schuster <philipp.schuster@cyberus-technology.de> 437* Rob Bradford <rbradford@rivosinc.com> 438* Ruoqing He <heruoqing@iscas.ac.cn> 439* Thomas Prescher <thomas.prescher@cyberus-technology.de> 440* abm-77 <andrewmiller77@protonmail.com> 441 442# v45.0 443 444This release has been tracked in [v45.0 445group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v44.0%22) 446of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 447 448### Experimental `riscv64` Architecture Support 449 450Cloud Hypervisor now has experimental `riscv64` architecture 451support. Details can be found from the [riscv 452documentation](docs/riscv.md). 453 454### Alphabetically Sorted CLI Options 455 456To improve the readability of CLI options, the output of the `--help` 457now is alphabetically sorted. (#6988) 458 459### Improved Downtime of VM Live Migration 460 461The downtime of VM live migration is reduced via delaying some of the 462tearing down process of the source VM after the destination VM is up and 463running. (#6987) 464 465### Notable Bug Fixes 466 467* Fix seccomp filters related to http-api thread (#6967) 468* Handle cross-page access in the emulator for mshv (#6968) 469 470### Contributors 471 472Many thanks to everyone who has contributed to our release: 473 474* Anatol Belski <anbelski@linux.microsoft.com> 475* Andrew Consroe <aconz2@gmail.com> 476* Bo Chen <bchen@crusoe.ai> 477* Jinank Jain <jinankjain@microsoft.com> 478* Jinrong Liang <cloudliang@tencent.com> 479* Philipp Schuster <philipp.schuster@cyberus-technology.de> 480* Ruoqing He <heruoqing@iscas.ac.cn> 481* Stefan Kober <stefan.kober@cyberus-technology.de> 482* Wei Liu <liuwe@microsoft.com> 483 484# v44.0 485 486This release has been tracked in [v44.0 487group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v44.0%22) 488of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 489 490### Configurable `virtio-iommu` Address Width 491 492The `iommu_address_width` option has been added to `--platform` to allow users 493to limit the `virtio-iommu` address space in the guest. (#6900) 494 495### Notable Performance Improvements 496 497The `VIRTIO_BLK_F_SEG_MAX` feature has been enabled for `virtio-block` devices, 498which brings significant performance improvements on throughput. (#6885) 499 500The `io_uring` entries are no longer forced to use async helper workers, 501delegating the decision to the kernel. This change resolved the issue of having 502excessive amount of worker threads when `io_uring` is being used, which is 503expected to improve performance, such as reducing memory usage and reduce CPU 504contention. 505 506### New Fuzzers 507 508Our continuous fuzzing infrastructure is augmented with two new fuzzers to cover 509x86 instruction emulator and `virtio-vsock`. 510 511### Notable Bug Fixes 512 513* Fix short read and short write that impact QCOW and VHDX support. (#6884) 514* Various bug fixes on VHDX support. (#6890, #6899) 515 516### Contributors 517 518Many thanks to everyone who has contributed to our release: 519 520* Arvind Vasudev <avasudev@crusoeenergy.com> 521* Bo Chen <bo.arvin.chen@gmail.com> 522* Nikolay Edigaryev <edigaryev@gmail.com> 523* Rob Bradford <rbradford@rivosinc.com> 524* Ruoqing He <heruoqing@iscas.ac.cn> 525* Wei Liu <liuwe@microsoft.com> 526 527# v43.0 528 529This release has been tracked in [v43.0 530group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v43.0%22) 531of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 532 533### Live Migration over TCP Connections 534 535Support has been added to enable direct live migration from two hosts via TCP 536connections. This supplements the existing support for migrating over a UNIX 537socket which can then be tunnelled as desired. The documentation has been 538updated. (#6850) 539 540### Notable Performance Improvements 541 542The `VIRTIO_RING_F_INDIRECT_DESC` feature has been enabled for `virtio-block` 543devices. This significantly increases the throughput of the devices with a 544small negative impact on latency. (#6826) 545 546### Notable Bug Fixes 547 548* Cloud Hypervisor now accepts VFIO devices that use I/O PCI BARs on non x86-64 549 architectures. Whether they function depends on the host PCI host bridge 550 support - previously they would be rejected even if the driver did not use 551 these BARs. (#6871) 552* Command line groups were adjusted to ensure that at least one payload 553 parameter was provided if any other VM parameters provided. (#6832) 554 555### Contributors 556 557Many thanks to everyone who has contributed to our release: 558 559* Alyssa Ross <hi@alyssa.is> 560* Andrew Consroe <aconz2@gmail.com> 561* Bo Chen <bo.arvin.chen@gmail.com> 562* Jinrong Liang <cloudliang@tencent.com> 563* Julian Stecklina <julian.stecklina@cyberus-technology.de> 564* Muminul Islam <muislam@microsoft.com> 565* Rob Bradford <rbradford@rivosinc.com> 566* Ruoqing He <heruoqing@iscas.ac.cn> 567* Wojtek Czekalski <wczekalski@me.com> 568 569# v42.0 570 571This release has been tracked in [v42.0 572group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v42.0%22) 573of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 574 575### SVE/SVE2 Support on AArch64 576 577The SVE and SVE2 feature bits are now propagated through to the guest on 578AArch64. (#6678, #6691) 579 580### Notable Bug Fixes 581 582* Reduce latency notification when rate limited (#6672) 583* Fix `virtio-console` resizing (#6704) 584* Fix resizing when console uses TTY (#6754) 585* Avoid deadlock in PCI BAR reprogramming that can occur when adding a new 586 `virtio` device to a VM that has been restored (#6775) 587* Fix console resizing after VM restore (#6748) 588* Fix memory resize error due to incorrect bounds checks (#6736) 589 590### Sponsorships 591 592During this release cycle a new VFIO CI worker has been provided by Crusoe 593Energy and a new ARM64 CI worker has been provided by Ubicloud. 594 595### Contributors 596 597Many thanks to everyone who has contributed to our release: 598 599* Alexandru Matei <alexandru.matei@uipath.com> 600* Alyssa Ross <hi@alyssa.is> 601* Anirudh Rayabharam <anrayabh@microsoft.com> 602* BharatNarasimman <bharatn@microsoft.com> 603* Bo Chen <chen.bo@intel.com> 604* Jinank Jain <jinankjain@microsoft.com> 605* Jonas Scholz <Jonas.Scholz@bbscholz.de> 606* Nuno Das Neves <nudasnev@microsoft.com> 607* Praveen K Paladugu <prapal@linux.microsoft.com> 608* Purna Pavan Chandra <paekkaladevi@microsoft.com> 609* Rob Bradford <rbradford@rivosinc.com> 610* Ruoqing He <heruoqing@iscas.ac.cn> 611* Songqian Li <sionli@tencent.com> 612* Tom Dohrmann <erbse.13@gmx.de> 613* Wei Liu <liuwe@microsoft.com> 614* Wenyu Huang <huangwenyuu@outlook.com> 615* Yuhong Zhong <yz@cs.columbia.edu> 616* wuxinyue <wuxinyue.wxy@antgroup.com> 617 618# v41.0 619 620This release has been tracked in [v41.0 621group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v41.0%22) 622of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 623 624### Experimental "Pvmemcontrol" Support 625 626VMM support has been added for this experimental functionality (requires 627currently out of tree Linux kernel patches) to allow guests to control its 628physical memory properties to allow optimisations and security features. 629(#6318, #6467) 630 631### Sandboxing With Landlock Support 632 633Support for restricting the VMM process using the Linux kernel "Landlock" API 634has been added - this can be used to restrict the files (and the read/write 635permissions) that the VMM process can access. This adds another layer of 636security alongside the existing sycall filters (`seccomp`) - this can be 637enabled with `--landlock` and [fully documentated](docs/landlock.md). (#5170) 638 639### Notable Performance Improvements 640 641* Reduced heap allocations in `virtio-net` via the use of a cache of `Iovec` 642 structures (#6636) 643* Notification suppression ("`EVENT_IDX`") support has been added to 644 `virtio-block` giving a 60% improvement in single queue block throughput and 645 IOPs performance (#6580) 646* Correct size used for `status` field in `virtio-block` state (#6586) 647 648### Notable Bug Fixes 649 650* Avoid panic on out-of-bounds PCI MSI-X access (#6657) 651* Fix undefined behaviour on AArch64 leading to wrong optimisation on KVM API 652 access (#6647) 653* Rust v1.80.0 added use of `fcntl` syscall on debug assertions so this is now 654 included in the virtio-device seccomp filters for tests that use this (#6648) 655* Short reads are now handled correctly in the `virtio-vsock` device (#6621) 656* Fix undefined behaviour on TTY ioctl leading to wrong optimisation (#6568) 657 658### Contributors 659 660Many thanks to everyone who has contributed to our release: 661 662* Alyssa Ross <hi@alyssa.is> 663* Bo Chen <chen.bo@intel.com> 664* Changyuan Lyu <changyuanl@google.com> 665* Jinank Jain <jinankjain@microsoft.com> 666* Julian Stecklina <julian.stecklina@cyberus-technology.de> 667* Muminul Islam <muislam@microsoft.com> 668* Nuno Das Neves <nudasnev@microsoft.com> 669* Praveen K Paladugu <prapal@linux.microsoft.com> 670* Rob Bradford <rbradford@rivosinc.com> 671* Songqian Li <sionli@tencent.com> 672* Wei Liu <liuwe@microsoft.com> 673* Yuanchu Xie <yuanchu@google.com> 674* ihciah <ihciah@gmail.com> 675* wuxinyue <wuxinyue.wxy@antgroup.com> 676 677# v40.0 678 679This release has been tracked in [v40.0 680group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v40.0%22) 681of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 682 683### Support for Restoring File Descriptor Backed Network Devices 684 685It is now possible to pass file descriptors over the HTTP API (and using 686`ch-remote`) when restoring to update the file descriptors for network devices. 687This enables snapshot & restore functionality for guests using `macvtap` or 688other file descriptor backed network devices. (#6286) 689 690### Notable Bug Fixes 691 692* Default values have been removed from required fields in the OpenAPI metadata 693 (#6495) 694* The help syntax of `ch-remote remove-device` has been improved (#6456) 695* A double close of file descriptors has been fixed when using `--serial` 696 (#6486) 697* To prevent loops a limit on the nesting level for QCOW2 backing files has 698 been introduced (#6482) 699* Boot time performance has been improved with multiple cores by avoiding 700 `cpuid` instructions and by seeding the in kernel file descriptor table 701 (#6498, #6478) 702* L1 cache details are more likely to be propagated into the guest (#6523) 703* The default topology for guests now uses multiple cores rather than sockets 704 (#6504) 705 706### Contributors 707 708Many thanks to everyone who has contributed to our release: 709 710* Alexandru Matei <alexandru.matei@uipath.com> 711* Bo Chen <chen.bo@intel.com> 712* Jinank Jain <jinankjain@microsoft.com> 713* Josh Soref <2119212+jsoref@users.noreply.github.com> 714* Muminul Islam <muislam@microsoft.com> 715* Nuno Das Neves <nudasnev@microsoft.com> 716* Omer Faruk Bayram <omer.faruk@sartura.hr> 717* Praveen K Paladugu <prapal@linux.microsoft.com> 718* Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> 719* Rob Bradford <rbradford@rivosinc.com> 720* SamrutGadde <samrut.gadde@gmail.com> 721* Sean Banko <sbanko@crusoe.ai> 722* Songqian Li <sionli@tencent.com> 723* Wei Liu <liuwe@microsoft.com> 724* Yi Wang <foxywang@tencent.com> 725* Yu Li <liyu.yukiteru@bytedance.com> 726* 727 728# v39.0 729 730This release has been tracked in [v39.0 731group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v39.0%22) 732of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 733 734### Variable Sizing of PCI Apertures for Segments 735 736It is now possible to use `--pci-segment` to adjust the aperture size that 737devices 32-bit and 64-bit PCI device BARs will be allocated from. Previously 738the address space was equally distributed across all the segments which may 739leave insufficient space for devices that require a large 32-bit space. With 740this change the weighting per segment can be adjusted. (#6387) 741 742### Direct Booting with bzImages 743 744Support for directly booting Linux from bzImages has been added.(#6200) 745 746### Support for NVIDIA GPUDirect P2P Support 747 748The `x_nv_gpudirect_clique` option was added to `--device` to allow the 749configuration of device P2P support with NVIDIA GPUs. (#6235) 750 751### Guest NMI Injection Support 752 753A new API endpoint and `ch-remote` option added for injecting an NMI into the 754guest. (#6047) 755 756### Notable Bug Fixes 757 758* Workaround for kernel bug affecting guest IRQ masking on AMD (#6353) 759* Correctly cleanup `sigwinch_listener` process (#6208) 760* Graceful shutdown of HTTP API thread (#6248, #6247) 761* Fix `queue_affinity` option in OpenAPI metadata (#6268) 762* Fix documentation to indicate only stream mode is supported by `virtio-vsock` 763 (#6306) 764* Fix `virtio-fs` tag validation (#6358, #6359) 765* Add missing `pvpanic` device to OpenAPI metadata (#6372) 766* Fixes for nested virtualization with VFIO devices (#6110, #6298, #6297, 767 #6319) 768* Fix for backing file for `virtio-mem` regions with snapshot/restore (#6337, 769 #6338) 770* Explicitly mark FDs used for network devices as invalid across 771 snapshot/restore (#6332, #6286) 772* Improve `event-monitor` events around reboot (#6277, #6274) 773* Fix potential deadlock around paused devices during live migration (#6293) 774* Fix panic when running `ch-remote` with no subcommand (#6230) 775* Fix hotplug of `virtio` devices after snapshot/restore and live migration 776 (#6326, #6265) 777 778### Contributors 779 780Many thanks to everyone who has contributed to our release: 781 782* Alexandru Matei <alexandru.matei@uipath.com> 783* Andrew Carp <acarp@crusoeenergy.com> 784* Bo Chen <chen.bo@intel.com> 785* Bouke van der Bijl <i@bou.ke> 786* Chris Webb <chris@arachsys.com> 787* Jinank Jain <jinankjain@microsoft.com> 788* Lucas Jacques <contact@lucasjacques.com> 789* Muminul Islam <muislam@microsoft.com> 790* Nuno Das Neves <nudasnev@microsoft.com> 791* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 792* Rob Bradford <rbradford@rivosinc.com> 793* Ruslan Mstoi <ruslan.mstoi@intel.com> 794* Stefan Nuernberger <stefan.nuernberger@cyberus-technology.de> 795* Thomas Barrett <tbarrett@crusoeenergy.com> 796* Wei Liu <liuwe@microsoft.com> 797* Yi Wang <foxywang@tencent.com> 798 799# v38.0 800 801This release has been tracked in [v38.0 802group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v38.0%22) 803of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 804 805### Group Rate Limiter on Block Devices 806 807Users now can throttle a group of block devices with the new 808`--rate-limiter-group` option. Details can be found from the [I/O 809Throttling documentation](docs/io_throttling.md) 810 811### CPU Pinning Support for Block Device Worker Thread 812 813Users now have the option to pin virt-queue threads for block devices 814to specific host cpus. 815 816### Optimized Boot Time with Parallel Memory Prefault 817 818The boot time with `prefault` option enabled is optimized via parallel 819memory prefault. 820 821### New 'debug-console' Device 822 823A 'debug-console' device is added to provide a user-configurable debug 824port for logging guest information. Details can be found from the [Debug 825IO Ports documentation](docs/debug-port.md). 826 827### Improved VFIO Device Support 828 829All non-emulated MMIO regions of VFIO devices are now mapped to the VFIO 830container, allowing PCIe P2P between all VFIO devices on the same 831VM. This is required for a wide variety of multi-GPU workloads involving 832GPUDirect P2P (DMA between two GPUs), GPUDirect RDMA (DMA between a GPU 833and an IB device). 834 835### Extended CPU Affinity Support 836 837Users now can set the vcpu affinity to a host CPU with index larger 838than 255. 839 840### Notable Bug Fixes 841 842* Enable HTT flag to avoid crashing cpu topology enumeration software 843such as hwloc in the guest (#6146) 844* Fix several security advisories from dependencies (#6134, #6141) 845* Handle non-power-of-two CPU topology properly (#6062) 846* Various bug fixes around `virtio-vsock`(#6080, #6091, #6095) 847* Enable nested virtualization on AMD if supported (#6106) 848* Align VFIO devices PCI BARs naturally (#6196) 849 850### Contributors 851 852Many thanks to everyone who has contributed to our release: 853* Alyssa Ross <hi@alyssa.is> 854* Bo Chen <chen.bo@intel.com> 855* Daniel Farina <daniel@ubicloud.com> 856* Jinank Jain <jinankjain@microsoft.com> 857* Muminul Islam <muislam@microsoft.com> 858* Peteris Rudzusiks <rye@stripe.com> 859* Philipp Schuster <philipp.schuster@cyberus-technology.de> 860* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 861* Rob Bradford <rbradford@rivosinc.com> 862* Ruslan Mstoi <ruslan.mstoi@intel.com> 863* Sean Banko <sbanko@crusoeenergy.com> 864* Thomas Barrett <tbarrett@crusoeenergy.com> 865* Wei Liu <liuwe@microsoft.com> 866* Yi Wang <foxywang@tencent.com> 867* acarp <acarp@crusoeenergy.com> 868 869# v37.0 870 871This release has been tracked in [v37.0 872group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v37.0%22) 873of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 874 875### Long Term Support (LTS) Release 876 877This release is a LTS release. Point releases for bug fixes will be made 878for the next 18 months; live migration and live upgrade will be 879supported between the point releases of the LTS. 880 881### Multiple PCI segments Support for 32-bit VFIO devices 882 883Now VFIO devices with 32-bit memory BARs can be attached to non-zero PCI 884segments on the guest, allowing users to have more 32-bit devices and 885assign such devices to appropriate NUMA nodes for better performance. 886 887### Configurable Named TAP Devices 888 889Named TAP devices now accepts IP configuration from users, such as IP 890and MAC address, as long as the named TAP device is created by Cloud 891Hypervisor (e.g. not existing TAP devices). 892 893### TTY Output from Both Serial Device and Virtio Console 894 895Now legacy serial device and virtio console can be set as TTY mode as 896the same time. This allows users to capture early boot logs with the 897legacy serial device without losing performance benefits of using 898virtio-console, when appropriate kernel configuration is used (such as 899using kernel command-line `console=hvc0 earlyprintk=ttyS0` on x86). 900 901### Faster VM Restoration from Snapshots 902 903The speed of VM restoration from snapshots is improved with a better 904implementation of deserializing JSON files. 905 906### Notable Bug Fixes 907 908* Fix aio backend behavior for block devices when writeback cache 909 disabled (#5930) 910* Fix PvPanic device PCI BAR alignment (#5956) 911* Bug fix to OpenAPI specification file (#5967) 912* Error out early for live migration when TDX is enabled (#6025) 913 914### Contributors 915 916Many thanks to everyone who has contributed to our release: 917 918* Bo Chen <chen.bo@intel.com> 919* Jinank Jain <jinankjain@microsoft.com> 920* Markus Sütter <markus.suetter@secunet.com> 921* Michael Zhao <michael.zhao@arm.com> 922* Muminul Islam <muislam@microsoft.com> 923* Rob Bradford <rbradford@rivosinc.com> 924* Rui Chang <rui.chang@arm.com> 925* Ruslan Mstoi <ruslan.mstoi@intel.com> 926* Thomas Barrett <tbarrett@crusoeenergy.com> 927* Wei Liu <liuwe@microsoft.com> 928* Yi Wang <foxywang@tencent.com> 929* Yong He <alexyonghe@tencent.com> 930 931# v36.0 932 933This release has been tracked in [v36.0 934group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v36.0%22) 935of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 936 937### Command Line Changes 938 939We switched back to use the `clap` crate to create our command line, 940since the `argh` crate is barely maintained. There were several syntax 941changes: 942 943* All `--option value` commands now are `--option=value`. 944* The `--disk DISK1 --disk DISK2` command now is `--disk DISK1 DISK2`. 945* The `-v -v -v`command now is `-vvv`. 946 947Note: the released binary size increased around 0.3M due to this change. 948 949### Enabled Features Reported via API Endpoint and CLI 950 951Now the enabled (Cargo) features of the running Cloud Hypervisor 952instance can be queried via API endpoint (`/vmm.ping`) and CLI 953(`--version -v`). 954 955### NUMA Support for PCI segments 956 957The `--numa` command is augmented with a new option `pci_segment=`, so 958that users can define the relationship between PCI segments and NUMA 959nodes. Examples can be found from the [memory documentation](docs/memory.md) 960 961### CPU Topology Support on AMD Platforms 962 963Now the CPU topology on x86_64 platforms supports multiple vendors. 964 965### Unix Socket Backend for Serial Port 966 967The `--serial` command is augmented with a new option `socket=`, allowing 968users to access the serial port using a Unix socket. 969 970### AIO Backend for Block Devices 971 972An AIO backend is added for `virtio-block` devices to improve block 973device performance when the `io_uring` feature is not supported by the 974host Operating System. 975 976### Documentation Improvements 977 978* New [documentation](docs/coverage.md) for collecting coverage data 979* Various typo fixes 980 981### Notable Bug Fixes 982 983* Fix a deadlock when TDX is enabled (#5845) 984* Only advertise AMX feature bits to guest when the AMX cpu feature is 985 enabled (#5834) 986* Correct default value for vCPU topology on AArch64 (#5893) 987 988### Contributors 989 990Many thanks to everyone who has contributed to our release: 991 992* Anatol Belski <anbelski@linux.microsoft.com> 993* Bo Chen <chen.bo@intel.com> 994* Dario Nieuwenhuis <dirbaio@dirbaio.net> 995* Jianyong Wu <jianyong.wu@arm.com> 996* Jinank Jain <jinankjain@microsoft.com> 997* Muminul Islam <muislam@microsoft.com> 998* Praveen K Paladugu <prapal@linux.microsoft.com> 999* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1000* Rob Bradford <rbradford@rivosinc.com> 1001* Thomas Barrett <tbarrett@crusoeenergy.com> 1002* Wei Liu <liuwe@microsoft.com> 1003* Yi Wang <foxywang@tencent.com> 1004* dom.song <dom.song@amperecomputing.com> 1005 1006# v35.0 1007 1008This release has been tracked in [v35.0 1009group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v35.0%22) 1010of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1011 1012### `virtio-vsock` Support for Linux Guest Kernel v6.3+ 1013 1014Since kernel v6.3, a vsock packet can be included in a single descriptor, 1015instead of being split over two descriptors. Our `virtio-vsock` implementation 1016now support both situations. 1017 1018### User Specified Serial Number for `virtio-block` 1019 1020A new option `serial` is added to the `--block` command that allows users to 1021specify a serial number for block devices which will be guest visible. 1022 1023### vCPU TSC Frequency Included in Migration State 1024 1025This ensures migration works correctly between hosts that have different TSC 1026frequencies if the guest is running with TSC as the source of timekeeping. 1027 1028### Notable Bug Fixes 1029 1030* Disallow concurrent CPU resizing (#5668) 1031* Handle APIC EOI message for MSHV (#5681) 1032* Use wrapping add for memory offset from instruction emulator (#5719) 1033* Add global spell check via the 'typos' GitHub action (#5720) 1034* Ensure probing reads are block size aligned (#5727) 1035* Multiple bug fixes around the latency counter for block devices 1036 (#5712, #5750, #5762, #5763) 1037* Replace unsound `static mut` with `once_cell` (#5772) 1038 1039### Contributors 1040 1041Many thanks to everyone who has contributed to our release: 1042 1043* Alyssa Ross <hi@alyssa.is> 1044* Anatol Belski <anbelski@linux.microsoft.com> 1045* Bo Chen <chen.bo@intel.com> 1046* Christian Blichmann <cblichmann@google.com> 1047* Jianyong Wu <jianyong.wu@arm.com> 1048* Jinank Jain <jinankjain@microsoft.com> 1049* Julian Stecklina <julian.stecklina@cyberus-technology.de> 1050* Omer Faruk Bayram <omer.faruk@sartura.hr> 1051* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1052* Rob Bradford <rbradford@rivosinc.com> 1053* Ruslan Mstoi <ruslan.mstoi@intel.com> 1054* Thomas Barrett <tbarrett@crusoeenergy.com> 1055* Wei Liu <liuwe@microsoft.com> 1056* Yi Wang <foxywang@tencent.com> 1057* zhongbingnan <zhongbingnan@bytedance.com> 1058 1059# v34.0 1060 1061This release has been tracked in [v34.0 1062group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v34.0%22) 1063of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1064 1065### Paravirtualised Panic Device Support 1066 1067A new device has been added that can communicate when the guest kernel has 1068panicked and share those details with the VMM. This is controlled with a new 1069`--pvpanic` command line option and JSON API change equivalent. (#5526) 1070 1071### Improvements to VM Core Dump 1072 1073Requesting to dump the guest memory as core dump will now transparently pause 1074the VM if required; returning to the original state after. (#5604) 1075 1076### QCOW2 Support for Backing Files 1077 1078The support for QCOW2 files has been enhanced to include support for using 1079backing files. (#5573) 1080 1081### Minimum Host Kernel Bump 1082 1083The minimum supported host kernel is now 5.13 in order to incorporate a bug fix 1084for `KVM_FEATURE_ASYNC_PF_INT` functionality. (#5626) 1085 1086### Notable Bug Fixes 1087 1088* The x86 emulator is only compiled in if MSHV is compiled in (the kernel 1089 carries out this job with KVM) (#5561). 1090* A regression has been fixed in VFIO support for devices that use MSI rather 1091 than MSI-X (#5658). 1092* When triggering a VM shutdown or reset via I/O the vCPU thread will be 1093 blocked until that asynchronous event has been received (#5645). 1094* Pausing a VM is now a synchronous action with the request only completing 1095 when all vCPUs are paused (#5611). 1096* Event monitor support now correctly supports concurrent access (#5633). 1097* Bug fixes for the QCOW2 file support (#5573). 1098 1099### Contributors 1100 1101Many thanks to everyone who has contributed to our release: 1102 1103* Alyssa Ross <hi@alyssa.is> 1104* Anatol Belski <anbelski@linux.microsoft.com> 1105* Bo Chen <chen.bo@intel.com> 1106* Changyuan Lyu <changyuanl@google.com> 1107* Christian Blichmann <cblichmann@google.com> 1108* Manish Goregaokar <manishsmail@gmail.com> 1109* Omer Faruk Bayram <omer.faruk@sartura.hr> 1110* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1111* Praveen K Paladugu <prapal@linux.microsoft.com> 1112* Rob Bradford <rbradford@rivosinc.com> 1113* Ruslan Mstoi <ruslan.mstoi@intel.com> 1114* Yi Wang <foxywang@tencent.com> 1115* Yong He <alexyonghe@tencent.com> 1116* Yu Li <liyu.yukiteru@bytedance.com> 1117* dom.song <dom.song@amperecomputing.com> 1118 1119# v33.0 1120 1121This release has been tracked in [v33.0 1122group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v33.0%22) 1123of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1124 1125### D-Bus based API 1126 1127A D-Bus based API has been added as an alternative to the existing REST 1128API. This feature is gated by the `dbus_api` feature. Details can be 1129found in the [API documentation](docs/api.md). 1130 1131### Expose Host CPU Cache Details for AArch64 1132 1133Now the CPU cache information on the host is properly exposed to the 1134guest on AArch64. 1135 1136### Notable Bug Fixes 1137 1138* Report errors explicitly to users when VM failed to boot (#5453) 1139* Fix VFIO on platforms with non-4k page size (#5450, #5469) 1140* Fix TDX initialization (#5454) 1141* Ensure all guest memory regions are page-size aligned (#5496) 1142* Fix seccomp filter lists related to virtio-console, serial and pty 1143 (#5506, #5524) 1144* Populate APIC ID properly (#5512) 1145* Ignore and warn TAP FDs in more situations (#5522) 1146 1147### Contributors 1148 1149Many thanks to everyone who has contributed to our release: 1150 1151* Alyssa Ross <hi@alyssa.is> 1152* Anatol Belski <anbelski@linux.microsoft.com> 1153* Bo Chen <chen.bo@intel.com> 1154* Jianyong Wu <jianyong.wu@arm.com> 1155* Omer Faruk Bayram <omer.faruk@sartura.hr> 1156* Rafael Mendonca <rafaelmendsr@gmail.com> 1157* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1158* Rob Bradford <rbradford@rivosinc.com> 1159* Ruslan Mstoi <ruslan.mstoi@intel.com> 1160* Yu Li <liyu.yukiteru@bytedance.com> 1161* zhongbingnan <zhongbingnan@bytedance.com> 1162 1163# v32.0 1164 1165This release has been tracked in [v32.0 1166group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v32.0%22) 1167of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1168 1169### Increased PCI Segment Limit 1170 1171The maximum number of PCI segments that can be used is now 96 (up from 16). 1172 1173### API Changes 1174 1175* The VmmPingResponse now includes the PID as well as the build details. 1176 (#5348) 1177 1178### Notable Bug Fixes 1179 1180* Ignore and warn TAP FDs sent via the HTTP request body (#5350) 1181* Properly preserve and close valid FDs for TAP devices (#5373) 1182* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360) 1183* Only touch the tty flags if it's being used (#5343) 1184* Fix seccomp filter lists for vhost-user devices (#5361) 1185* The number of vCPUs is capped at the hypervisor maximum (#5357) 1186* Fixes for TTY reset (#5414) 1187* CPU topology fixes on MSHV (#5325) 1188* Seccomp fixes for older distributions (#5397) 1189 1190### Contributors 1191 1192Many thanks to everyone who has contributed to our release: 1193 1194* Alyssa Ross <hi@alyssa.is> 1195* Anatol Belski <anbelski@linux.microsoft.com> 1196* Bo Chen <chen.bo@intel.com> 1197* Hao Xu <howeyxu@tencent.com> 1198* Muminul Islam <muislam@microsoft.com> 1199* Omer Faruk Bayram <omer.faruk@sartura.hr> 1200* Rafael Mendonca <rafaelmendsr@gmail.com> 1201* Rob Bradford <rbradford@rivosinc.com> 1202* Ruslan Mstoi <ruslan.mstoi@intel.com> 1203* Smit Gardhariya <gardhariya.smit@gmail.com> 1204* Wei Liu <liuwe@microsoft.com> 1205 1206# v31.1 1207 1208This is a bug fix release. The following issues have been addressed: 1209 1210* Ignore and warn TAP FDs sent via the HTTP request body (#5350) 1211* Properly preserve and close valid FDs for TAP devices (#5373) 1212* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360) 1213* Only touch the tty flags if it's being used (#5343) 1214* Fix seccomp filter lists for vhost-user devices (#5361) 1215 1216# v31.0 1217 1218This release has been tracked in [v31.0 1219group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v31.0%22) 1220of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1221 1222### Update to Latest `acpi_tables` 1223 1224Adapted to the latest [acpi_tables](https://github.com/rust-vmm/acpi_tables). 1225There has been significant API changes in the crate. 1226 1227### Update Reference Kernel to 6.2 1228 1229Updated the recommended guest kernel version from 6.1.6 to 6.2. 1230 1231### Improvements on Console `SIGWINCH` Handler 1232 1233A separate thread had been created to capture the `SIGWINCH` signal and resize 1234the guest console. Now the thread is skipped if the console is not resizable. 1235 1236Two completely different code paths existed for handling console resizing, one 1237for `tty` and the other for `pty`. That makes the understanding of the console 1238handling code unnecessarily complicated. Now the code paths are unified. Both 1239`tty` and `pty` are supported in single `SIGWINCH` handler. And the new handler 1240can works with kernel versions earlier than v5.5. 1241 1242### Remove Directory Support from `MemoryZoneConfig::file` 1243 1244Setting a directory to `MemoryZoneConfig::file` is no longer supported. 1245 1246Before this change, user can set a directory to `file` of the `--memory-zone` 1247option. In that case, a temporary file will be created as the backing file for 1248the `mmap(2)` operation. This functionality has been unnecessary since we had 1249the native support for hugepages and allocating anonymous shared memory. 1250 1251### Documentation Improvements 1252 1253* Various improvements in API document 1254* Improvements in Doc comments 1255* Updated Slack channel information in README 1256 1257### Notable Bug Fixes 1258 1259* Fixed the offset setting while removing the entire mapping of `vhost-user` FS 1260 client. 1261* Fixed the `ShutdownVmm` and `Shutdown` commands to call the correct API 1262 endpoint. 1263 1264### Contributors 1265 1266Many thanks to everyone who has contributed to our release: 1267 1268* Alyssa Ross <hi@alyssa.is> 1269* Bo Chen <chen.bo@intel.com> 1270* Daniel Farina <daniel@fdr.io> 1271* Dom <peng6662001@163.com> 1272* Hao Xu <howeyxu@tencent.com> 1273* Muminul Islam <muislam@microsoft.com> 1274* Omer Faruk Bayram <omer.faruk@sartura.hr> 1275* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1276* Rob Bradford <rbradford@rivosinc.com> 1277* Ruslan Mstoi <ruslan.mstoi@intel.com> 1278* Smit Gardhariya <gardhariya.smit@gmail.com> 1279* Yang <ailin.yang@intel.com> 1280* Yong He <alexyonghe@tencent.com> 1281 1282# v30.0 1283 1284This release has been tracked in [v30.0 1285group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v30.0%22) 1286of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1287 1288### Command Line Changes for Reduced Binary Size 1289 1290The `clap` crate was replaced by the `argh` crate to create our command 1291line, which reduced our release binary size from 3.6MB to 3.3MB. There 1292were several syntax changes: 1293 1294* All `--option=value` commands now are `--option value`. 1295* The `--disk DISK1 DISK2` command now is `--disk DISK1 --disk DISK2`. 1296* The `-vvv` command now is `-v -v -v` 1297 1298### Basic vfio-user Server Support 1299 1300Our `vfio-user` crate is extended to provide basic server side support 1301with an example of gpio vfio-user device. This crate now is moved to [its 1302own repository](https://github.com/rust-vmm/vfio-user) under the 1303`rust-vmm` organization. 1304 1305### Heap Profiling Support 1306 1307A new building target is added for profiling purposes with examples of 1308heap profiling using `dhat` gated by the `dhat-heap` feature. 1309 1310### Documentation Improvements 1311 1312The documentation on Intel TDX is expanded with details of the building 1313and using [TD-Shim](https://github.com/confidential-containers/td-shim), 1314references to [TDX Linux](https://github.com/intel/tdx-linux), and 1315version information of guest/host kernel/TDVF/TDShim being tested. Also, 1316a new 'heap profiling' documentation is added with improvements on the 1317existing 'profiling' documentation. 1318 1319### Notable Bug Fixes 1320 1321* Close FDs for TAP devices that are provided to VM (#5199, #5206) 1322* Set vcpu thread status properly and signal `exit_evt` upon thread exit (#5211) 1323* Populate CPUID leaf 0x4000_0010 (TSC frequency) (#5178, #5179) 1324* Inform the TPM guest driver upon failed TPM requests on the host (#5151) 1325* Bug fix to OpenAPI specification file (#5186) 1326 1327### Contributors 1328 1329Many thanks to everyone who has contributed to our release: 1330 1331* Anatol Belski <anbelski@linux.microsoft.com> 1332* Anirudh Rayabharam <anrayabh@linux.microsoft.com> 1333* Bo Chen <chen.bo@intel.com> 1334* Jinank Jain <jinankjain@microsoft.com> 1335* Kaihang Zhang <kaihang.zhang@smartx.com> 1336* Michael Zhao <michael.zhao@arm.com> 1337* Muminul Islam <muislam@microsoft.com> 1338* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1339* Praveen K Paladugu <prapal@linux.microsoft.com> 1340* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1341* Rob Bradford <robert.bradford@intel.com> 1342* Ruslan Mstoi <ruslan.mstoi@intel.com> 1343* Sebastien Boeuf <sebastien.boeuf@intel.com> 1344* Wei Liu <liuwe@microsoft.com> 1345* Yong He <alexyonghe@tencent.com> 1346* Yu Li <liyu.yukiteru@bytedance.com> 1347 1348# v28.2 1349This is a bug fix release. The following issues have been addressed: 1350 1351* Fix QCOW2 refcount table size (#5034) 1352* Fix unpause support on MSHV in dual binary (#5037) 1353* Threads inside `virtio` devices are now shutdown on reboot (#5095) 1354 1355# v29.0 1356 1357This release has been tracked in [v29.0 1358group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v29.0%22) 1359of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1360 1361### Release Binary Supports Both MSHV and KVM 1362 1363On `x86-64` the binary included in releases supports both the KVM and MSHV 1364hypervisor with runtime detection to identify the correct hypervisor to use. 1365 1366### Snapshot/Restore and Live Migration Improvements 1367 1368Improvements have been made to the data structures used for both live migration 1369and snapshot/restore. Unfortunately this has broken compatibility with older 1370versions (support for migrating between major versions is not yet officially 1371supported but has worked for some versions.) 1372 1373### Heap Allocation Improvements 1374 1375Improvements have been made to the volume of heap allocations when running with 1376`virtio-block` devices along with a reduction in the peak heap size. 1377 1378### `ch-remote` Improvements 1379 1380Support for "pinging" the VMM and shutting the VMM down have been added to 1381`ch-remote`. 1382 1383### `AArch64` Documentation Integration 1384 1385The documentation for `AArch64` support has been integrated into the main 1386README. 1387 1388### `virtio-block` Counters Enhancement 1389 1390The counters for the `virtio-block` device has extended to include min/mean/max 1391latency counters. 1392 1393### TCP Offload Control 1394 1395The `virtio-net` device has gained support for controlling the enabling of 1396checksum and offloading. This allows the device to be used in environments 1397where the hardware lacks support for the offloading. 1398 1399### Notable Bug Fixes 1400 1401* Update dependencies including a version of `linux-loader` that addresses an 1402 infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6)) 1403* Fix bugs related to `virtio-net` including an integer overflow issue 1404 (#4924, #4949) 1405* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920) 1406* Memory mapped into the guest is now marked as non-dumpable which prevents large core files (#5016) 1407* Fix QCOW2 refcount table size (#5034) 1408* Fix unpause support on MSHV in dual binary (#5037) 1409* Threads inside `virtio` devices are now shutdown on reboot (#5095) 1410 1411### Removals 1412 1413No functionality has been removed in this release. 1414 1415### Deprecations 1416 1417* Support for specifying a directory with `MemoryZoneConfig::file` or 1418 `MemoryConfig::file` has been deprecated. This was originally used for 1419 supporting huge pages or shared memory backing which is now natively supported 1420 (#5085) 1421 1422### Contributors 1423 1424Many thanks to everyone who has contributed to our release: 1425 1426* Bo Chen <chen.bo@intel.com> 1427* Claudio Fontana <claudio.fontana@gmail.com> 1428* Hao Xu <howeyxu@tencent.com> 1429* Henry Wang <Henry.Wang@arm.com> 1430* Jinank Jain <jinankjain@microsoft.com> 1431* Michael Zhao <michael.zhao@arm.com> 1432* Muminul Islam <muislam@microsoft.com> 1433* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1434* Rob Bradford <robert.bradford@intel.com> 1435* Sebastien Boeuf <sebastien.boeuf@intel.com> 1436* Shuaiyi Zhang <zhangsy28@lenovo.com> 1437* Wei Liu <liuwe@microsoft.com> 1438* Yong He <alexyonghe@tencent.com> 1439* Yuji Hagiwara <yuuzi41@gmail.com> 1440 1441 1442# v28.1 1443This is a bug fix release. The following issues have been addressed: 1444 1445* Update dependencies including a version of `linux-loader` that 1446addresses an infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6)) 1447* Fix bugs related to `virtio-net` including an integer overflow issue 1448 (#4924, #4949) 1449* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920) 1450* Improve README and documentation 1451 1452# v28.0 1453 1454This release has been tracked in [v28.0 1455group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v28.0%22) 1456of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1457 1458### Community Engagement (Reminder) 1459 1460Just a reminder that we have a new mailing list to support broader community 1461discussions. Please consider 1462[subscribing](https://lists.cloudhypervisor.org/g/dev/). We plan to use 1463this to announce a regular meeting for those interested in talking about Cloud 1464Hypervisor development. 1465 1466### Long Term Support (LTS) Release 1467 1468This is the first version of Cloud Hypervisor to be released under the LTS 1469release process. Point releases for bug fixes will be made for the next 18 1470months; live migration and live upgrade will be supported between the point 1471releases of the LTS. 1472 1473### Virtualised TPM Support 1474 1475Support for adding an emulated CRB TPM has been added. This has its own [TPM 1476documentation](docs/tpm.md). 1477 1478### Transparent Huge Page Support 1479 1480By default, but controllable through `--memory thp=off` if it possible to back 1481the guest memory with Transparent Huge Pages (no file backing/`shared=off`) 1482then this will be used resulting in improved boot performance. 1483 1484### README Quick Start Improved 1485 1486The README has been refreshed with the quick start guide updated to reflect the 1487different firmware options and to recommend the use of pre-built binaries. 1488 1489### Notable Bug Fixes 1490 1491* Inappropriate Copy-on-Write of pinned pages (e.g. VFIO) leading to higher 1492 memory consumption (#4835) 1493* Multiple `virtio` device bug fixes found through fuzzing (#4859, #4799) 1494* Large QCOW files (> 4TiB) are now supported (#4767) 1495* Support for > 31 vCPUS on aarch64 (#4863) 1496* Multiple fixes to OpenAPI specification file (#4720, #4811) 1497* Programming of the MSI-X table has been optimised leading to faster boot on 1498 newer Linux kernels (#4744) 1499* Error on reboot from race to close TAP devices (#4871) 1500* Non-spec compliant virtio-block read-only support (#4888) 1501 1502### Removals 1503 1504The following functionality has been removed: 1505 1506* Support for non-PVH firmware booting has been removed (#4511) 1507* I/O ports used for older versions of firmware have been removed (#3926) 1508* Deprecated API options for kernel/cmdline/initramfs have been removed (#4737) 1509 1510### Contributors 1511 1512Many thanks to everyone who has contributed to our release: 1513 1514* Anatol Belski <anbelski@linux.microsoft.com> 1515* Bo Chen <chen.bo@intel.com> 1516* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1517* Jianyong Wu <jianyong.wu@arm.com> 1518* Jinank Jain <jinankjain@microsoft.com> 1519* Jinrong Liang <cloudliang@tencent.com> 1520* lv.mengzhao <lv.mengzhao@zte.com.cn> 1521* Michael Zhao <michael.zhao@arm.com> 1522* Muminul Islam <muislam@microsoft.com> 1523* Praveen K Paladugu <prapal@linux.microsoft.com> 1524* Rob Bradford <robert.bradford@intel.com> 1525* Sebastien Boeuf <sebastien.boeuf@intel.com> 1526* Wei Liu <liuwe@microsoft.com> 1527 1528# v27.0 1529 1530This release has been tracked in [v27.0 1531group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v27.0%22) 1532of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1533 1534### Community Engagement 1535 1536A new mailing list has been created to support broader community discussions. 1537Please consider [subscribing](https://lists.cloudhypervisor.org/g/dev/); an 1538announcement of a regular meeting will be announced via this list shortly. 1539 1540### Prebuilt Packages 1541 1542Prebuilt packages are now available. Please see [this 1543document](https://github.com/cloud-hypervisor/obs-packaging/blob/main/README.md) 1544on how to install. These packages also include packages for the different 1545firmware options available. 1546 1547### Network Device MTU Exposed to Guest 1548 1549The MTU for the TAP device associated with a `virtio-net` device is now exposed 1550to the guest. If the user provides a MTU with `--net mtu=..` then that MTU is 1551applied to created TAP interfaces. This functionality is also exposed for 1552`vhost-user-net` devices including those created with the reference backend 1553(#4658, #4676.) 1554 1555### Boot Tracing 1556 1557Support for generating a trace report for the boot time has been added 1558including a script for generating an SVG from that trace (#4659.) 1559 1560### Simplified Build Feature Flags 1561 1562The set of feature flags, for e.g. experimental features, have been simplified: 1563 1564* `msvh` and `kvm` features provide support for those specific hypervisors 1565 (with `kvm` enabled by default), 1566* `tdx` provides support for Intel TDX; and although there is no MSHV support 1567 now it is now possible to compile with the `mshv` feature (#4696,) 1568* `tracing` adds support for boot tracing, 1569* `guest_debug` now covers both support for gdbing a guest (formerly `gdb` 1570 feature) and dumping guest memory. 1571 1572The following feature flags were removed as the functionality was enabled by 1573default: `amx`, `fwdebug`, `cmos` and `common` (#4679, #4632.) 1574 1575### Asynchronous Kernel Loading 1576 1577AArch64 has gained support for loading the guest kernel asynchronously like 1578x86-64. (#4538) 1579 1580### GDB Support for AArch64 1581 1582GDB stub support (accessed through `--gdb` under `guest_debug` feature) is now 1583available on AArch64 as well as as x86-64. 1584 1585### Notable Bug Fixes 1586 1587* This version incorporates a version of `virtio-queue` that addresses an issue 1588 where a rogue guest can potentially DoS the VMM (rust-vmm/vm-virtio#196.) 1589* Improvements around PTY handling for `virtio-console` and serial devices 1590 (#4520, #4533, #4535.) 1591* Improved error handling in virtio devices (#4626, #4605, #4509, #4631, #4697) 1592 1593### Deprecations 1594 1595Deprecated features will be removed in a subsequent release and users should 1596plan to use alternatives. 1597 1598* Booting legacy firmware (compiled without a PVH header) has been deprecated. 1599 All the firmware options (Cloud Hypervisor OVMF and Rust Hypervisor Firmware) 1600 support booting with PVH so support for loading firmware in a legacy mode is no 1601 longer needed. This functionality will be removed in the next release. 1602 1603### Contributors 1604 1605Many thanks to everyone who has contributed to our release: 1606 1607* Anatol Belski <anbelski@linux.microsoft.com> 1608* Bo Chen <chen.bo@intel.com> 1609* James O. D. Hunt <james.o.hunt@intel.com> 1610* Jianyong Wu <jianyong.wu@arm.com> 1611* Markus Napierkowski <markus.napierkowski@cyberus-technology.de> 1612* Michael Zhao <michael.zhao@arm.com> 1613* Nuno Das Neves <nudasnev@microsoft.com> 1614* Rob Bradford <robert.bradford@intel.com> 1615* Sebastien Boeuf <sebastien.boeuf@intel.com> 1616* Smit Gardhariya <sgardhariya@microsoft.com> 1617* Wei Liu <liuwe@microsoft.com> 1618 1619# v26.0 1620 1621### SMBIOS Improvements via `--platform` 1622 1623`--platform` and the appropriate API structure has gained support for supplying 1624OEM strings (primarily used to communicate metadata to systemd in the guest) 1625(#4319, #4446) and support for specifying the UUID (#4389.) 1626 1627### Unified Binary MSHV and KVM Support 1628 1629Support for both the MSHV and KVM hypervisors can be compiled into the same 1630binary with the detection of the hypervisor to use made at runtime. 1631 1632### Notable Bug Fixes 1633 1634* The prefetchable flag is preserved on BARs for VFIO devices (#4353, #4454) 1635* PCI Express capabilities for functionality we do not support are now filtered 1636 out (#4456) 1637* GDB breakpoint support is more reliable (#4354, #4363) 1638* `SIGINT` and `SIGTERM` signals are now handled before the VM has booted 1639 (#4269, #4293) 1640* Multiple API event loop handling bug fixes (#4309, #4362) 1641* Incorrect assumptions in virtio queue numbering were addressed, allowing 1642 the`virtio-fs` driver in OVMF to be used (#4341, #4314) 1643* VHDX file format header fix (#4291) 1644* The same VFIO device cannot be added twice (#4453, #4463) 1645* SMBIOS tables were being incorrectly generated (#4442) 1646 1647### Deprecations 1648 1649Deprecated features will be removed in a subsequent release and users should 1650plan to use alternatives. 1651 1652* The top-level `kernel` and `initramfs` members on the `VmConfig` have been 1653 moved inside a `PayloadConfig` as the `payload` member. The OpenAPI document 1654 has been updated to reflect the change and the old API members continue to 1655 function and are mapped to the new version. The expectation is that these old 1656 versions will be removed in the v28.0 release. 1657 1658### Removals 1659 1660The following functionality has been removed: 1661 1662* The unused `poll_queue` parameter has been removed from `--disk` and 1663 equivalent. This was residual from the removal of the `vhost-user-block` 1664 spawning feature (#4402.) 1665 1666### Contributors 1667 1668Many thanks to everyone who has contributed to our release: 1669 1670* Alyssa Ross <hi@alyssa.is> 1671* Anatol Belski <ab@php.net> 1672* Archana Shinde <archana.m.shinde@intel.com> 1673* Bo Chen <chen.bo@intel.com> 1674* lizhaoxin1 <Lxiaoyouling@163.com> 1675* Maximilian Nitsch <maximilian.nitsch@d3tn.com> 1676* Michael Zhao <michael.zhao@arm.com> 1677* Rob Bradford <robert.bradford@intel.com> 1678* Sebastien Boeuf <sebastien.boeuf@intel.com> 1679* Steven Dake <sdake@lambdal.com> 1680* Wei Liu <liuwe@microsoft.com> 1681 1682# v25.0 1683 1684### `ch-remote` Improvements 1685 1686The `ch-remote` command has gained support for creating the VM from a JSON 1687config and support for booting and deleting the VM from the VMM. 1688 1689### VM "Coredump" Support 1690 1691Under the `guest_debug` feature flag it is now possible to extract the memory 1692of the guest for use in debugging with e.g. the `crash` utility. (#4012) 1693 1694### Notable Bug Fixes 1695 1696* Always restore console mode on exit (#4249, #4248) 1697* Restore vCPUs in numerical order which fixes aarch64 snapshot/restore (#4244) 1698* Don't try and configure `IFF_RUNNING` on TAP devices (#4279) 1699* Propagate configured queue size through to vhost-user backend (#4286) 1700* Always Program vCPU CPUID before running the vCPU to fix running on Linux 1701 5.16 (#4156) 1702* Enable ACPI MADT "Online Capable" flag for hotpluggable vCPUs to fix newer 1703 Linux guest 1704 1705### Removals 1706 1707The following functionality has been removed: 1708 1709* The `mergeable` option from the `virtio-pmem` support has been removed 1710 (#3968) 1711* The `dax` option from the `virtio-fs` support has been removed (#3889) 1712 1713### Contributors 1714 1715Many thanks to everyone who has contributed to our release: 1716 1717* Dylan Bargatze <dbargatz@users.noreply.github.com> 1718* Jinank Jain <jinankjain@microsoft.com> 1719* Michael Zhao <michael.zhao@arm.com> 1720* Rob Bradford <robert.bradford@intel.com> 1721* Sebastien Boeuf <sebastien.boeuf@intel.com> 1722* Wei Liu <liuwe@microsoft.com> 1723* Yi Wang <wang.yi59@zte.com.cn> 1724 1725# v24.0 1726 1727### Bypass Mode for `virtio-iommu` 1728 1729`virtio-iommu` specification describes how a device can be attached by default 1730to a bypass domain. This feature is particularly helpful for booting a VM with 1731guest software which doesn't support `virtio-iommu` but still need to access 1732the device. Now that Cloud Hypervisor supports this feature, it can boot a VM 1733with Rust Hypervisor Firmware or OVMF even if the `virtio-block` device exposing 1734the disk image is placed behind a virtual IOMMU. 1735 1736### Ensure Identifiers Uniqueness 1737 1738Multiple checks have been added to the code to prevent devices with identical 1739identifiers from being created, and therefore avoid unexpected behaviors at boot 1740or whenever a device was hot plugged into the VM. 1741 1742### Sparse Mmap support 1743 1744Sparse mmap support has been added to both VFIO and vfio-user devices. This 1745allows the device regions that are not fully mappable to be partially mapped. 1746And the more a device region can be mapped into the guest address space, the 1747fewer VM exits will be generated when this device is accessed. This directly 1748impacts the performance related to this device. 1749 1750### Expose Platform Serial Number 1751 1752A new `serial_number` option has been added to `--platform`, allowing a user to 1753set a specific serial number for the platform. This number is exposed to the 1754guest through the SMBIOS. 1755 1756### Notable Bug Fixes 1757 1758* Fix loading RAW firmware (#4072) 1759* Reject compressed QCOW images (#4055) 1760* Reject virtio-mem resize if device is not activated (#4003) 1761* Fix potential mmap leaks from VFIO/vfio-user MMIO regions (#4069) 1762* Fix algorithm finding HOB memory resources (#3983) 1763 1764### Notable Improvements 1765 1766* Refactor interrupt handling (#4083) 1767* Load kernel asynchronously (#4022) 1768* Only create ACPI memory manager DSDT when resizable (#4013) 1769 1770### Deprecations 1771 1772Deprecated features will be removed in a subsequent release and users should 1773plan to use alternatives 1774 1775* The `mergeable` option from the `virtio-pmem` support has been deprecated 1776 (#3968) 1777* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 1778 1779### New on the Website 1780 1781A new blog post [Achieving Bare Metal Performance Within a Virtual 1782Machine](https://www.cloudhypervisor.org/blog/achieving-bare-metal-performance-within-a-virtual-machine) 1783has been added to the Cloud Hypervisor website. 1784 1785### Contributors 1786 1787Many thanks to everyone who has contributed to our release: 1788 1789* Anatol Belski <anbelski@linux.microsoft.com> 1790* Bo Chen <chen.bo@intel.com> 1791* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1792* LiHui <andrewli@kubesphere.io> 1793* Maksym Pavlenko <pavlenko.maksym@gmail.com> 1794* Rob Bradford <robert.bradford@intel.com> 1795* Sebastien Boeuf <sebastien.boeuf@intel.com> 1796* Steven Dake <steven.dake@gmail.com> 1797* Vincent Batts <vbatts@hashbangbash.com> 1798* Wei Liu <liuwe@microsoft.com> 1799 1800# v23.1 1801 1802This is a bug fix release. The following issues have been addressed: 1803 1804* Add some missing seccomp rules 1805* Remove `virtio-fs` filesystem entries from config on removal 1806* Do not delete API socket on API server start (#4026) 1807* Reject `virtio-mem` resize if the guest doesn't activate the device 1808* Fix OpenAPI naming of I/O throttling knobs 1809 1810# v23.0 1811 1812### vDPA Support 1813 1814A vDPA device has a datapath that complies with the virtio specification but 1815with a vendor specific control path. The addition of `--vdpa` and the REST API 1816equivalent allows the use of these devices with Cloud Hypervisor. 1817 1818### Updated OS Support list 1819 1820The list of officially supported and tested OS versions has been updated to 1821include Ubuntu "jammy" 22.04 and EOLed versions removed. 1822 1823### `AArch64` Memory Map Improvements 1824 1825The memory map when running on `AArch64` has been improved for the handling of 1826the UEFI region which means that the booted guest OS now has full access to its 1827allocated RAM. (#3938) 1828 1829### `AMX` Support 1830 1831Under a compile time gate of `amx` it is possible compile in support for the 1832`AMX` instruction set extension for guest use. This also requires runtime 1833enabling with `--cpu features=amx`. 1834 1835### Notable Bug Fixes 1836 1837* Generate error when incorrect HTTP method used for some API endpoints (#3887) 1838* CMOS based reset is now available to support rebooting on "jammy" (#3914) 1839* ACPI tables are not produced for memory hotplug when running with 1840 `virtio-mem` (#3883) 1841* `virtio-iommu` backed PCI segments are now comprehensively placed behind the 1842 vIOMMU (#3870) 1843* Seccomp rules have been extended for `virtio-fs` to support direct access 1844 (#3848) 1845 1846### Deprecations 1847 1848Deprecated features will be removed in a subsequent release and users should 1849plan to use alternatives 1850 1851* The `mergeable` option from the `virtio-pmem` support has been deprecated 1852 (#3968) 1853* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 1854 1855### Contributors 1856 1857Many thanks to everyone who has contributed to our release: 1858 1859* Bo Chen <chen.bo@intel.com> 1860* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1861* Henry Wang <Henry.Wang@arm.com> 1862* Jianyong Wu <jianyong.wu@arm.com> 1863* LiHui <andrewli@kubesphere.io> 1864* Michael Zhao <michael.zhao@arm.com> 1865* Rob Bradford <robert.bradford@intel.com> 1866* Sebastien Boeuf <sebastien.boeuf@intel.com> 1867* Wei Liu <liuwe@microsoft.com> 1868* William Douglas <william.douglas@intel.com> 1869* Yi Wang <wang.yi59@zte.com.cn> 1870 1871# v22.1 1872 1873This is a bug fix release. The following issues have been addressed: 1874 1875* VFIO ioctl reordering to fix MSI on AMD platforms (#3827) 1876* Fix `virtio-net` control queue (#3829) 1877 1878# v22.0 1879 1880### GDB Debug Stub Support 1881 1882Cloud Hypervisor can now be used as debug target with GDB. This is controlled 1883by the `gdb` compile time feature and details of how to use it can be found in 1884the [gdb 1885documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/gdb.md). 1886 1887### `virtio-iommu` Backed Segments 1888 1889In order to facilitate hotplug devices that require being behind an IOMMU (e.g. 1890QAT) there is a new option `--platform iommu_segments=<list_of_segments>` that 1891will place all the specified segments behind the IOMMU. 1892 1893### Before Boot Configuration Changes 1894 1895It is now possible to change the VM configuration (e.g. add or remove devices, 1896resize) before the VM is booted. 1897 1898### `virtio-balloon` Free Page Reporting 1899 1900If `--balloon free_page_reporting=on` is used then the guest can report pages 1901that is it not using to the VMM. The VMM will then notify the host OS that 1902those pages are no longer in use and can be freed. This can result in improved 1903memory density. 1904 1905### Support for Direct Kernel Booting with TDX 1906 1907Through the use of `TD-Shim` lightweight firmware it is now possible to 1908directly boot into the kernel with TDX. The [TDX 1909documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/intel_tdx.md#tdshim) 1910has been updated for this usage. 1911 1912### PMU Support for AArch64 1913 1914A PMU is now available on AArch64 for guest performance profiling. This will be 1915exposed automatically if available from the host. 1916 1917### Documentation Under CC-BY-4.0 License 1918 1919The documentation is now licensed under the "Creative Commons Attribution 4.0 1920International" license which is aligned with the project charter under the 1921Linux Foundation. 1922 1923### Deprecation of "Classic" `virtiofsd` 1924 1925The use of the Rust based [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd) 1926is now recommended and we are no longer testing against the C based "classic" 1927version. 1928 1929### Notable Bug Fixes 1930 1931* Can now be used on kernels without `AF_INET` support (#3785) 1932* `virtio-balloon` size is now validated against guest RAM size (#3689) 1933* Ensure that I/O related KVM VM Exits are correctly handled (#3677) 1934* Multiple TAP file descriptors can be used for `virtio-net` device hotplug (#3607) 1935* Minor API improvements and fixes (#3756, #3766, #3647, #3578) 1936* Fix sporadic seccomp violation from glibc memory freeing (#3610, #3609) 1937* Fix Windows 11 on AArch64 due to wider MSI-X register accesses (#3714, #3720) 1938* Ensure `vhost-user` features are correct across migration (#3737) 1939* Improved vCPU topology on AArch64 (#3735, #3733) 1940 1941### Contributors 1942 1943Many thanks to everyone who has contributed to our release: 1944 1945* Akira Moroo <retrage01@gmail.com> 1946* Barret Rhoden <brho@google.com> 1947* Bo Chen <chen.bo@intel.com> 1948* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1949* Feng Ye <yefeng@smartx.com> 1950* Henry Wang <Henry.Wang@arm.com> 1951* Jianyong Wu <jianyong.wu@arm.com> 1952* lizhaoxin1 <Lxiaoyouling@163.com> 1953* Michael Zhao <michael.zhao@arm.com> 1954* Rob Bradford <robert.bradford@intel.com> 1955* Sebastien Boeuf <sebastien.boeuf@intel.com> 1956* Wei Liu <liuwe@microsoft.com> 1957 1958# v21.0 1959 1960### Efficient Local Live Migration (for Live Upgrade) 1961 1962In order to support fast live upgrade of the VMM an optimised path has been 1963added in which the memory for the VM is not compared from source to 1964destination. This is activated by passing `--local` to the `ch-remote 1965send-migration` command. This means that the live upgrade can complete in the 1966order of 50ms vs 3s. (#3566) 1967 1968### Recommended Kernel is Now 5.15 1969 1970Due to an issue in the `virtio-net` code in 5.14 the recommended Linux kernel 1971is now 5.15. (#3530) 1972 1973### Notable Bug fixes 1974 1975* Multiple fixes were made to the OpenAPI YAML file to match the implementation (#3555,#3562) 1976* Avoid live migration deadlock when triggered during the kernel boot (#3585) 1977* Support live migration within firmware (#3586) 1978* Validate the `virtio-net` descriptor chain (#3548) 1979* `direct=on` (`O_DIRECT`) can now be used with a guest that makes unaligned accesses (e.g. firmware) (#3587) 1980 1981### Contributors 1982 1983Many thanks to everyone who has contributed to our release: 1984 1985* Anatol Belski <anbelski@linux.microsoft.com> 1986* Barret Rhoden <brho@google.com> 1987* Bo Chen <chen.bo@intel.com> 1988* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1989* Henry Wang <Henry.Wang@arm.com> 1990* Liang Zhou <zhoul110@chinatelecom.cn> 1991* Michael Zhao <michael.zhao@arm.com> 1992* Muhammad Falak R Wani <falakreyaz@gmail.com> 1993* Muminul Islam <muislam@microsoft.com> 1994* Rob Bradford <robert.bradford@intel.com> 1995* Sebastien Boeuf <sebastien.boeuf@intel.com> 1996* Wei Liu <liuwe@microsoft.com> 1997* Ziye Yang <ziye.yang@intel.com> 1998 1999# v20.2 2000 2001This is a bug fix release. The following issues have been addressed: 2002 2003* Don't error out when setting up the SIGWINCH handler (for console resize) 2004 when this fails due to older kernel (#3456) 2005* Seccomp rules were refined to remove syscalls that are now unused 2006* Fix reboot on older host kernels when SIGWINCH handler was not initialised 2007 (#3496) 2008* Fix virtio-vsock blocking issue (#3497) 2009 2010# v20.1 2011 2012This is a bug fix release. The following issues have been addressed: 2013 2014* Networking performance regression with `virtio-net` (#3450) 2015* Limit file descriptors sent in `vfio-user` support (#3401) 2016* Fully advertise PCI MMIO config regions in ACPI tables (#3432) 2017* Set the TSS and KVM identity maps so they don't overlap with firmware RAM 2018* Correctly update the `DeviceTree` on restore 2019 2020# v20.0 2021 2022### Multiple PCI segments support 2023 2024Cloud Hypervisor is no longer limited to 31 PCI devices. For both `x86_64` and 2025`aarch64` architectures, it is now possible to create up to 16 PCI segments, 2026increasing the total amount of supported PCI devices to 496. 2027 2028### CPU pinning 2029 2030For each vCPU, the user can define a limited set of host CPUs on which it is 2031allowed to run. This can be useful when assigning a 1:1 mapping between host and 2032guest resources, or when running a VM on a specific NUMA node. 2033 2034### Improved VFIO support 2035 2036Based on VFIO region capabilities, all regions can be memory mapped, limiting 2037the amount of triggered VM exits, and therefore increasing the performance of 2038the passthrough device. 2039 2040### Safer code 2041 2042Several sections containing unsafe Rust code have been replaced with safe 2043alternatives, and multiple comments have been added to clarify why the remaining 2044unsafe sections are safe to use. 2045 2046### Extended documentation 2047 2048The documentation related to VFIO has been updated while some new documents have 2049been introduced to cover the usage of `--cpus` parameter as well as how to run 2050Cloud Hypervisor on Intel TDX. 2051 2052### Notable bug fixes 2053 2054* Naturally align PCI BARs on relocation (#3244) 2055* Fix panic in SIGWINCH listener thread when no seccomp filter set (#3338) 2056* Use the tty raw mode implementation from libc (#3344) 2057* Fix the emulation of register D for CMOS/RTC device (#3393) 2058 2059### Contributors 2060 2061Many thanks to everyone who has contributed to our release: 2062 2063* Alyssa Ross <hi@alyssa.is> 2064* Bo Chen <chen.bo@intel.com> 2065* Fabiano Fidêncio <fabiano.fidencio@intel.com> 2066* Michael Zhao <michael.zhao@arm.com> 2067* Muminul Islam <muislam@microsoft.com> 2068* Rob Bradford <robert.bradford@intel.com> 2069* Sebastien Boeuf <sebastien.boeuf@intel.com> 2070* Wei Liu <liuwe@microsoft.com> 2071* Willen Yang <willenyang@gmail.com> 2072* William Douglas <william.douglas@intel.com> 2073* Ziye Yang <ziye.yang@intel.com> 2074 2075# v19.0 2076 2077### Improved PTY handling for serial and `virtio-console` 2078 2079The PTY support for serial has been enhanced with improved buffering when the 2080the PTY is not yet connected to. Using `virtio-console` with PTY now results in 2081the console being resized if the PTY window is also resized. 2082 2083### PCI boot time optimisations 2084 2085Multiple optimisations have been made to the PCI handling resulting in 2086significant improvements in the boot time of the guest. 2087 2088### Improved TDX support 2089 2090When using the latest TDVF firmware the ACPI tables created by the VMM are now 2091exposed via the firmware to the guest. 2092 2093### Live migration enhancements 2094 2095Live migration support has been enhanced to support migration with `virtio-mem` 2096based memory hotplug and the `virtio-balloon` device now supports live 2097migration. 2098 2099### `virtio-mem` support with `vfio-user` 2100 2101The use of `vfio-user` userspaces devices can now be used in conjunction with 2102`virtio-mem` based memory hotplug and unplug. 2103 2104### AArch64 for `virtio-iommu` 2105 2106A paravirtualised IOMMU can now be used on the AArch64 platform. 2107 2108### Notable bug fixes 2109 2110* ACPI hotplugged memory is correctly restored after a live migration or 2111 snapshot/restore (#3165) 2112* Multiple devices from the same IOMMU group can be passed through via VFIO 2113 (#3078 #3113) 2114* Live migration with large blocks of memory was buggy due to an in issue in 2115 the underlying crate (#3157) 2116 2117### Contributors 2118 2119Many thanks to everyone who has contributed to our release: 2120 2121* Alyssa Ross <hi@alyssa.is> 2122* Bo Chen <chen.bo@intel.com> 2123* Henry Wang <Henry.Wang@arm.com> 2124* Hui Zhu <teawater@antfin.com> 2125* Jianyong Wu <jianyong.wu@arm.com> 2126* Li Yu <liyu.yukiteru@bytedance.com> 2127* Michael Zhao <michael.zhao@arm.com> 2128* Muminul Islam <muislam@microsoft.com> 2129* Rob Bradford <robert.bradford@intel.com> 2130* Sebastien Boeuf <sebastien.boeuf@intel.com> 2131* Wei Liu <liuwe@microsoft.com> 2132* William Douglas <william.douglas@intel.com> 2133* Yu Li <liyu.yukiteru@bytedance.com> 2134 2135# v18.0 2136 2137### Experimental User Device (`vfio-user`) support 2138 2139Experimental support for running PCI devices in userspace via `vfio-user` 2140has been included. This allows the use of the SPDK NVMe `vfio-user` controller 2141with Cloud Hypervisor. This is enabled by `--user-device` on the command line. 2142 2143### Migration support for `vhost-user` devices 2144 2145Devices exposed into the VM via `vhost-user` can now be migrated using the live 2146migration support. This requires support from the backend however the commonly 2147used DPDK `vhost-user` backend does support this. 2148 2149### VHDX disk image support 2150 2151Images using the VHDX disk image format can now be used with Cloud Hypervisor. 2152 2153### Device pass through on MSHV hypervisor 2154 2155When running on the MSHV hypervisor it is possible to pass through devices from 2156the host through to the guest (e.g with `--device`) 2157 2158### AArch64 for support `virtio-mem` 2159 2160The reference Linux kernel we recommend for using with Cloud Hypervisor now supports `virtio-mem` on AArch64. 2161 2162### Live migration on MSHV hypervisor 2163 2164Live migration is now supported when running on the MSHV hypervisor including 2165efficient tracking of dirty pages. 2166 2167### AArch64 CPU topology support 2168 2169The CPU topology (as configured through `--cpu topology=`) can now be 2170configured on AArch64 platforms and is conveyed through either ACPI or device 2171tree. 2172 2173### Power button support on AArch64 2174 2175Use of the ACPI power button (e.g `ch-remote --api-socket=<API socket> power-button`) 2176is now supported when running on AArch64. 2177 2178### Notable bug fixes 2179 2180* Using two PTY outputs e.g. `--serial pty --console pty` now works correctly (#3012) 2181* TTY input is now always sent to the correct destination (#3005) 2182* The boot is no longer blocked when using a unattached PTY on the serial console (#3004) 2183* Live migration is now supported on AArch64 (#3049) 2184* Ensure signal handlers are run on the correct thread (#3069) 2185 2186### Contributors 2187 2188Many thanks to everyone who has contributed to our release: 2189 2190* Alyssa Ross <hi@alyssa.is> 2191* Anatol Belski <anbelski@linux.microsoft.com> 2192* Arafatms <arafatms@outlook.com> 2193* Bo Chen <chen.bo@intel.com> 2194* Fazla Mehrab <akm.fazla.mehrab@vt.edu> 2195* Henry Wang <Henry.Wang@arm.com> 2196* Jianyong Wu <jianyong.wu@arm.com> 2197* Jiaqi Gao <jiaqi.gao@intel.com> 2198* Markus Theil <markus.theil@tu-ilmenau.de> 2199* Muminul Islam <muislam@microsoft.com> 2200* Rob Bradford <robert.bradford@intel.com> 2201* Sebastien Boeuf <sebastien.boeuf@intel.com> 2202* Wei Liu <liuwe@microsoft.com> 2203* Yu Li <liyu.yukiteru@bytedance.com> 2204 2205# v17.0 2206 2207### ARM64 NUMA support using ACPI 2208 2209The support for ACPI on ARM64 has been enhanced to include support for 2210specifying a NUMA configuration using the existing control options. 2211 2212### `Seccomp` support for MSHV backend 2213 2214The `seccomp` rules have now been extended to support running against the MSHV 2215hypervisor backend. 2216 2217### Hotplug of `macvtap` devices 2218 2219Hotplug of `macvtap` devices is now supported with the file descriptor for the 2220network device if opened by the user and passed to the VMM. The `ch-remote` 2221tool supports this functionality when adding a network device. 2222 2223### Improved SGX support 2224 2225The SGX support has been updated to match the latest Linux kernel support and 2226now supports SGX provisioning and associating EPC sections to NUMA nodes. 2227 2228### Inflight tracking for `vhost-user` devices 2229 2230Support for handling inflight tracking of I/O requests has been added to the 2231`vhost-user` devices allowing recovery after device reconnection. 2232 2233### Notable bug fixes 2234 2235* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821). 2236* The VMM side of `vhost-user` devices no longer advertise the 2237 `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM 2238(#2833). 2239* On ARM64 VMs can be created with more than 16 vCPUs (#2763). 2240 2241### Contributors 2242 2243Many thanks to everyone who has contributed to our release: 2244 2245* Anatol Belski <anbelski@linux.microsoft.com> 2246* Arafatms <arafatms@outlook.com> 2247* Bo Chen <chen.bo@intel.com> 2248* Fei Li <lifei.shirley@bytedance.com> 2249* Henry Wang <Henry.Wang@arm.com> 2250* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2251* Jianyong Wu <jianyong.wu@arm.com> 2252* Li Hangjing <lihangjing@bytedance.com> 2253* Michael Zhao <michael.zhao@arm.com> 2254* Muminul Islam <muislam@microsoft.com> 2255* Rob Bradford <robert.bradford@intel.com> 2256* Sebastien Boeuf <sebastien.boeuf@intel.com> 2257* Wei Liu <liuwe@microsoft.com> 2258* Yukiteru <wfly1998@sina.com> 2259 2260# v16.0 2261 2262### Improved live migration support 2263 2264The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details. 2265 2266### Improved `vhost-user` support 2267 2268When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch. 2269 2270### ARM64 ACPI and UEFI support 2271 2272Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot. 2273 2274### Notable bug fixes 2275 2276* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578). 2277* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563). 2278* The KVM clock is now correctly handled during live migration / snapshot & restore. 2279 2280### Removed functionality 2281 2282The following formerly deprecated features have been removed: 2283 2284* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2285 binaries has been deprecated. When using direct boot users should configure 2286 their kernel with `CONFIG_PVH=y`. 2287 2288### Contributors 2289 2290Many thanks to everyone who has contributed to our release including some new faces. 2291 2292* Anatol Belski <anbelski@linux.microsoft.com> 2293* Bo Chen <chen.bo@intel.com> 2294* Dayu Liu <liu.dayu@zte.com.cn> 2295* Henry Wang <Henry.Wang@arm.com> 2296* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2297* Jianyong Wu <jianyong.wu@arm.com> 2298* Michael Zhao <michael.zhao@arm.com> 2299* Mikko Ylinen <mikko.ylinen@intel.com> 2300* Muminul Islam <muislam@microsoft.com> 2301* Ren Lei <ren.lei4@zte.com.cn> 2302* Rob Bradford <robert.bradford@intel.com> 2303* Sebastien Boeuf <sebastien.boeuf@intel.com> 2304* Wei Liu <liuwe@microsoft.com> 2305* Yi Wang <wang.yi59@zte.com.cn> 2306 2307# v15.0 2308 2309Highlights for `cloud-hypervisor` version v15.0 include: 2310 2311### Version numbering and stability guarantees 2312 2313This release is the first in a new version numbering scheme to represent that 2314we believe Cloud Hypervisor is maturing and entering a period of stability. 2315With this new release we are beginning our new stability guarantees: 2316 2317* The API (including command line options) will not be removed or changed in a 2318 breaking way without a minimum of 2 releases notice. Where possible warnings 2319 will be given about the use of deprecated functionality and the deprecations 2320 will be documented in the release notes. 2321* Point releases will be made between individual releases where there are 2322 substantial bug fixes or security issues that need to be fixed. 2323 2324Currently the following items are **not** guaranteed across updates: 2325 2326* Snapshot/restore is not supported across different versions 2327* Live migration is not supported across different versions 2328* The following features are considered experimental and may change 2329 substantially between releases: TDX, SGX. 2330 2331### Network device rate limiting 2332 2333Building on our existing support for rate limiting block activity the network 2334device also now supports rate limiting. Full details of the controls are in the 2335[IO throttling documentation.](docs/io_throttling.md) 2336 2337### Support for runtime control of `virtio-net` guest offload 2338 2339The guest is now able to change the offload settings for the `virtio-net` 2340device. As well as providing a useful control this mitigates an issue in the 2341Linux kernel where the guest will attempt to reprogram the offload settings 2342even if they are not advertised as configurable (#2528). 2343 2344### `--api-socket` supports file descriptor parameter 2345 2346The `--api-socket` can now take an `fd=` parameter to specify an existing file 2347descriptor to use. This is particularly beneficial for frameworks that need to 2348programmatically control Cloud Hypervisor. 2349 2350### Bug fixes 2351 2352* A workaround has been put in place to mitigate a Linux kernel issues that 2353 results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277). 2354* PCI BARs are now correctly aligned removing the need for the guest to 2355 reprogram them (#1797,#1798) 2356* Handle TAP interface not being writable within virtio-net (due to the buffer 2357 exhaustion on the host) (#2517) 2358* The recommended Linux kernel is now v5.12.0 as it contains a fix that 2359 prevents snapshot & restore working (#2535) 2360 2361### Deprecations 2362 2363Deprecated features will be removed in a subsequent release and users should plan to use alternatives 2364 2365* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2366 binaries has been deprecated. When using direct boot users should configure 2367 their kernel with `CONFIG_PVH=y`. Will be removed in v16.0. 2368 2369### Contributors 2370 2371Many thanks to everyone who has contributed to our release including some new faces. 2372 2373* Alyssa Ross <hi@alyssa.is> 2374* Anatol Belski <anbelski@linux.microsoft.com> 2375* Bo Chen <chen.bo@intel.com> 2376* Gaelan Steele <gbs@canishe.com> 2377* Jianyong Wu <jianyong.wu@arm.com> 2378* Michael Zhao <michael.zhao@arm.com> 2379* Muminul Islam <muislam@microsoft.com> 2380* Rob Bradford <robert.bradford@intel.com> 2381* Sebastien Boeuf <sebastien.boeuf@intel.com> 2382* Wei Liu <liuwe@microsoft.com> 2383* William Douglas <william.douglas@intel.com> 2384 2385# v0.14.1 2386 2387Bug fix release branched off the v0.14.0 release. The following bugs were fixed 2388in this release: 2389 2390* CPU hotplug on Windows failed due to misreported CPU state information and 2391 the lack of HyperV CPUID bit enabled (#2437, #2449, #2436) 2392* A seccomp rule was missing that was triggered on CPU unplug (#2455) 2393* A bounds check in VIRTIO queue validation was erroneously generating 2394 DescriptorChainTooShort errors in certain circumstances (#2450, #2424) 2395 2396# v0.14.0 2397 2398Highlights for `cloud-hypervisor` version 0.14.0 include: 2399 2400### Structured event monitoring 2401 2402A new option was added to the VMM `--event-monitor` which reports structured 2403events (JSON) over a file or file descriptor at key events in the lifecycle of 2404the VM. The list of events is limited at the moment but will be further 2405extended over subsequent releases. The events exposed form part of the Cloud 2406Hypervisor API surface. 2407 2408### MSHV improvements 2409 2410Basic support has been added for running Windows guests atop the MSHV 2411hypervisor as an alternative to KVM and further improvements have been made to 2412the MSHV support. 2413 2414### Improved aarch64 platform 2415 2416The aarch64 platform has been enhanced with more devices exposed to the running 2417VM including an enhanced serial UART. 2418 2419### Updated hotplug documentation 2420 2421The documentation for the hotplug support has been updated to reflect the use 2422of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as 2423well as documenting hotplug of paravirtualised and VFIO devices. 2424 2425### PTY control for serial and `virtio-console` 2426 2427The `--serial` and `--console` parameters can now direct the console to a PTY 2428allowing programmatic control of the console from another process through the 2429PTY subsystem. 2430 2431### Block device rate limiting 2432 2433The block device performance can now be constrained as part of the VM 2434configuration allowing rate limiting. Full details of the controls are in the 2435[IO throttling documentation.](docs/io_throttling.md) 2436 2437 2438### Deprecations 2439 2440Deprecated features will be removed in a subsequent release and users should plan to use alternatives 2441 2442* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2443 binaries has been deprecated. When using direct boot users should configure 2444 their kernel with `CONFIG_PVH=y`. 2445 2446 2447### Contributors 2448 2449Many thanks to everyone who has contributed to our 0.14.0 release including 2450some new faces. 2451 2452* Bo Chen <chen.bo@intel.com> 2453* Henry Wang <Henry.Wang@arm.com> 2454* Iggy Jackson <iggy@theiggy.com> 2455* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2456* Michael Zhao <michael.zhao@arm.com> 2457* Muminul Islam <muislam@microsoft.com> 2458* Penny Zheng <Penny.Zheng@arm.com> 2459* Rob Bradford <robert.bradford@intel.com> 2460* Sebastien Boeuf <sebastien.boeuf@intel.com> 2461* Vineeth Pillai <viremana@linux.microsoft.com> 2462* Wei Liu <liuwe@microsoft.com> 2463* William Douglas <william.r.douglas@gmail.com> 2464* Zide Chen <zide.chen@intel.com> 2465 2466# v0.13.0 2467 2468Highlights for `cloud-hypervisor` version 0.13.0 include: 2469 2470### Wider VFIO device support 2471 2472It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI 2473devices that do not support MSI or MSI-X and instead rely on INTx interrupts. 2474Most notably this widens the support to most NVIDIA cards with the proprietary 2475drivers. 2476 2477### Improved huge page support 2478 2479Through the addition of `hugepage_size` on `--memory` it is now possible to 2480specify the desired size of the huge pages used when allocating the guest 2481memory. The user is required to ensure they have sufficient pages of the 2482desired size in their pool. 2483 2484### MACvTAP support 2485 2486It is now possible to provide file descriptors using the `fd` parameter to 2487`--net` which point at TAP devices that have already been opened by the user. 2488This aids integration with `libvirt` but also permits the use of MACvTAP 2489support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md). 2490 2491### VHD disk image support 2492 2493It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw 2494disk image with Cloud Hypervisor. 2495 2496### Improved Virtio device threading 2497 2498Device threads are now derived from the main VMM thread which allows more 2499restrictive seccomp filters to be applied to them. The threads also have a 2500predictable name derived from the device id. 2501 2502### Clean shutdown support via synthetic power button 2503 2504It is now possible to request that the guest VM shut itself down by triggering 2505a synthetic ACPI power button press from the VMM. If the guest is listening for 2506such an event (e.g. using systemd) then it will process the event and cleanly 2507shut down. This functionality is exposed through the HTTP API and can be 2508triggered via `ch-remote --api-socket=<API socket> power-button`. 2509 2510### Contributors 2511 2512Many thanks to everyone who has contributed to our 0.13.0 release including 2513some new faces. 2514 2515* Bo Chen <chen.bo@intel.com> 2516* Mikko Ylinen <mikko.ylinen@intel.com> 2517* Muminul Islam <muislam@microsoft.com> 2518* Rob Bradford <robert.bradford@intel.com> 2519* Samuel Ortiz <sameo@linux.intel.com> 2520* Sebastien Boeuf <sebastien.boeuf@intel.com> 2521* Vineeth Pillai <viremana@linux.microsoft.com> 2522* Wei Liu <liuwe@microsoft.com> 2523* William Douglas <william.r.douglas@gmail.com> 2524* Xie Yongji <xieyongji@bytedance.com> 2525 2526# v0.12.0 2527 2528Highlights for `cloud-hypervisor` version 0.12.0 include: 2529 2530### ARM64 enhancements 2531 2532The use of `--watchdog` is now fully supported as is the ability to reboot the 2533VM from within the guest when running Cloud Hypervisor on an ARM64 system. 2534 2535### Removal of `vhost-user-net` and `vhost-user-block` self spawning 2536 2537In order to use `vhost-user-net` or `vhost-user-block` backends the user is now 2538responsible for starting the backend and providing the socket for the VMM to 2539use. This functionality was deprecated in the last release and how now been 2540removed. 2541 2542### Migration of `vhost-user-fs` backend 2543 2544The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is 2545instead hosted in [its own 2546repository](https://gitlab.com/virtio-fs/virtiofsd-rs) 2547 2548### Enhanced "info" API 2549 2550The `vm.info` HTTP API endpoint has been extended to include the details of the 2551devices used by the VM including any VFIO devices used. 2552 2553### Contributors 2554 2555Many thanks to everyone who has contributed to our 0.12.0 release: 2556 2557* Anatol Belski <anbelski@linux.microsoft.com> 2558* Julio Montes <julio.montes@intel.com> 2559* Michael Zhao <michael.zhao@arm.com> 2560* Muminul Islam <muislam@microsoft.com> 2561* Rob Bradford <robert.bradford@intel.com> 2562* Samuel Ortiz <sameo@linux.intel.com> 2563* Wei Liu <liuwe@microsoft.com> 2564 2565# v0.11.0 2566 2567Highlights for `cloud-hypervisor` version 0.11.0 include: 2568 2569### `io_uring` support by default for `virtio-block` 2570 2571Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will 2572be used for a significantly higher performance block device. 2573 2574### Windows Guest Support 2575 2576This is the first release where we officially support Windows running as a 2577guest. Full details of how to setup the image and run Cloud Hypervisor with a 2578Windows guest can be found in the dedicated [Windows 2579documentation](docs/windows.md). 2580 2581### `vhost-user` "Self Spawning" Deprecation 2582 2583Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now 2584deprecated. Users of this functionality will receive a warning and should make 2585adjustments. The functionality will be removed in the next release. 2586 2587### `virtio-mmio` Removal 2588 2589Support for using the `virtio-mmio` transport, rather than using PCI, has been 2590removed. This has been to simplify the code and significantly 2591reduce the testing burden of the project. 2592 2593### Snapshot/Restore support for ARM64 2594 2595When running on the ARM64 architecture snapshot and restore has now been 2596implemented. 2597 2598### Improved Linux Boot Time 2599 2600The time to boot the Linux kernel has been significantly improved by the 2601identifying some areas of delays around PCI bus probing, IOAPIC programming and 2602MPTABLE issues. Full details can be seen in #1728. 2603 2604### `SIGTERM/SIGINT` Interrupt Signal Handling 2605 2606When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will 2607trigger the VMM process to cleanly deallocate resources before exiting. The 2608guest VM will not be cleanly shutdown but the VMM process will clean up its 2609resources. 2610 2611### Default Log Level Changed 2612 2613The default logging level was changed to include warnings which should make it 2614easier to see potential issues. New [logging 2615documentation](docs/logging.md) was also added. 2616 2617### New `--balloon` Parameter Added 2618 2619Control of the setup of `virtio-balloon` has been moved from `--memory` to its 2620own dedicated parameter. This makes it easier to add more balloon specific 2621controls without overloading `--memory`. 2622 2623### Experimental `virtio-watchdog` Support 2624 2625Support for using a new `virtio-watchdog` has been added which can be used to 2626have the VMM reboot the guest if the guest userspace fails to ping the 2627watchdog. This is enabled with `--watchdog` and requires kernel support. 2628 2629### Notable Bug Fixes 2630 2631* MTRR bit was missing from CPUID advertised to guest 2632* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170) 2633* CPU identification string is now exposed to the guest 2634* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as 2635 read only so avoids excessive VM exits (#1795) 2636* PCI device hotplug after an unplug was fixed (#1802) 2637* When using the ACPI method to resize the guest memory the full reserved size 2638 can be used (#1803) 2639* Snapshot and restore followed by a second snapshot and restore now works 2640 correctly 2641* Snapshot and restore of VMs with more than 2GiB in one region now work 2642 correctly 2643 2644### Contributors 2645 2646Many thanks to everyone who has contributed to our 0.11.0 release including some new faces. 2647 2648* Anatol Belski <anbelski@linux.microsoft.com> 2649* Bo Chen <chen.bo@intel.com> 2650* Daniel Verkamp <dverkamp@chromium.org> 2651* Henry Wang <Henry.Wang@arm.com> 2652* Hui Zhu <teawater@antfin.com> 2653* Jiangbo Wu <jiangbo.wu@intel.com> 2654* Josh Soref <jsoref@users.noreply.github.com> 2655* Julio Montes <julio.montes@intel.com> 2656* Michael Zhao <michael.zhao@arm.com> 2657* Muminul Islam <muislam@microsoft.com> 2658* pierwill <19642016+pierwill@users.noreply.github.com> 2659* Praveen Paladugu <prapal@microsoft.com> 2660* Rob Bradford <robert.bradford@intel.com> 2661* Sebastien Boeuf <sebastien.boeuf@intel.com> 2662* Wei Liu <liuwe@microsoft.com> 2663 2664 2665# v0.10.0 2666 2667Highlights for `cloud-hypervisor` version 0.10.0 include: 2668 2669### `virtio-block` Support for Multiple Descriptors 2670 2671Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers. 2672 2673### Memory Zones 2674 2675Support has been added for fine-grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md). 2676 2677### `Seccomp` Sandbox Improvements 2678 2679All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`. 2680 2681### Preliminary KVM HyperV Emulation Control 2682 2683A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices. 2684 2685### Notable Bug Fixes 2686 2687- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596) 2688- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564) 2689- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593) 2690- `--version` shows the version for released binaries (#1669) 2691- Errors generated by worker threads for `virtio` devices are now printed out (#1551) 2692 2693### Contributors 2694 2695Many thanks to everyone who has contributed to our 0.10.0 release including some new faces. 2696 2697* Alyssa Ross <hi@alyssa.is> 2698* Amey Narkhede <ameynarkhede02@gmail.com> 2699* Anatol Belski <ab@php.net> 2700* Bo Chen <chen.bo@intel.com> 2701* Hui Zhu <teawater@antfin.com> 2702* Michael Zhao <michael.zhao@arm.com> 2703* Muminul Islam <muislam@microsoft.com> 2704* Rob Bradford <robert.bradford@intel.com> 2705* Samuel Ortiz <sameo@linux.intel.com> 2706* Sebastien Boeuf <sebastien.boeuf@intel.com> 2707* Wei Liu <liuwe@microsoft.com> 2708 2709# v0.9.0 2710 2711Highlights for `cloud-hypervisor` version 0.9.0 include: 2712 2713### `io_uring` Based Block Device Support 2714 2715If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement. 2716 2717### Block and Network Device Statistics 2718 2719Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM. 2720 2721### HTTP API Responses 2722 2723The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF. 2724 2725### CPU Topology 2726 2727A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core. 2728 2729### Release Build Optimization 2730 2731Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size. 2732 2733### Hypervisor Abstraction 2734 2735A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`. 2736 2737### Snapshot/Restore Improvements 2738 2739Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs. 2740 2741### Virtio Memory Ballooning Support 2742 2743A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest. 2744 2745### Enhancements to ARM64 Support 2746 2747The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting. 2748 2749### Intel SGX Support 2750 2751The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md). 2752 2753### `Seccomp` Sandbox Improvements 2754 2755The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development. 2756 2757### Notable Bug Fixes 2758 2759* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes. 2760* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338) 2761* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543). 2762* `virtio-mmio` based devices are now more widely tested (#275). 2763* Multiple issues have been fixed with virtio device configuration (#1217) 2764* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521) 2765 2766### Contributors 2767 2768Many thanks to everyone who has contributed to our 0.9.0 release including some new faces. 2769 2770* Anatol Belski <ab@php.net> 2771* Bo Chen <chen.bo@intel.com> 2772* Dr. David Alan Gilbert <dgilbert@redhat.com> 2773* Henry Wang <Henry.Wang@arm.com> 2774* Howard Zhang <howard.zhang@arm.com> 2775* Hui Zhu <teawater@antfin.com> 2776* Jianyong Wu <jianyong.wu@arm.com> 2777* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2778* LiYa'nan <oliverliyn@gmail.com> 2779* Michael Zhao <michael.zhao@arm.com> 2780* Muminul Islam <muislam@microsoft.com> 2781* Praveen Paladugu <prapal@microsoft.com> 2782* Ricardo Koller <ricarkol@gmail.com> 2783* Rob Bradford <robert.bradford@intel.com> 2784* Samuel Ortiz <sameo@linux.intel.com> 2785* Sebastien Boeuf <sebastien.boeuf@intel.com> 2786* Stefano Garzarella <sgarzare@redhat.com> 2787* Wei Liu <liuwe@microsoft.com> 2788 2789 2790# v0.8.0 2791 2792Highlights for `cloud-hypervisor` version 0.8.0 include: 2793 2794### Experimental Snapshot and Restore Support 2795 2796This release includes the first version of the snapshot and restore feature. 2797This allows a VM to be paused and then subsequently snapshotted. At a later 2798point that snapshot may be restored into a new running VM identical to the 2799original VM at the point it was paused. 2800 2801This feature can be used for offline migration from one VM host to another, to 2802allow the upgrading or rebooting of the host machine transparently to the guest 2803or for templating the VM. This is an experimental feature and cannot be used on 2804a VM using passthrough (VFIO) devices. Issues with SMP have also been observed 2805(#1176). 2806 2807### Experimental ARM64 Support 2808 2809Included in this release is experimental support for running on ARM64. 2810Currently only `virtio-mmio` devices and a serial port are supported. Full 2811details can be found in the [ARM64 documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/8ab15b9a984a448006f97b1211498c1bb583de3b/docs/arm64.md). 2812 2813### Support for Using 5-level Paging in Guests 2814 2815If the host supports it the guest is now enabled for 5-level paging (aka LA57). 2816This works when booting the Linux kernel with a vmlinux, bzImage or firmware 2817based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not 2818work due to current limitations in the PVH boot process. 2819 2820### Virtio Device Interrupt Suppression for Network Devices 2821 2822With `virtio-net` and `vhost-user-net` devices the guest can suppress 2823interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This 2824can lead to an improvement in performance by reducing the number of interrupts 2825the guest must service. 2826 2827### `vhost_user_fs` Improvements 2828 2829The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`. 2830 2831 2832### Notable Bug Fixes 2833 2834* VMs that have not yet been booted can now be deleted (#1110). 2835* By creating the `tap` device ahead of creating the VM it is not required to 2836 run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273). 2837* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to 2838 the specification and synchronizes to the underlying filesystem as required 2839 based on guest feature negotiation. This avoids potential data loss (#399, 2840 #1216). 2841* When booting with a large number of vCPUs then the ACPI table would be 2842 overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the 2843 `MPTABLE` will no longer be generated (#1132). 2844* Shutting down VMs that have been paused is now supported (#816). 2845* Created socket files are deleted on shutdown (#1083). 2846* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds 2847 (#751). 2848 2849### Command Line and API Changes 2850 2851This is non exhaustive list of HTTP API and command line changes: 2852 2853* All user visible socket parameters are now consistently called `socket` 2854 rather than `sock` in some cases. 2855* The `ch-remote` tool now shows any error message generated by the VMM 2856* The `wce` parameter has been removed from `--disk` as the feature is always 2857 offered for negotiation. 2858* `--net` has gained a `host_mac` option that allows the setting of the MAC 2859 address for the `tap` device on the host. 2860 2861### Contributors 2862 2863Many thanks to everyone who has contributed to our 0.8.0 release including some new faces. 2864 2865* Anatol Belski <ab@php.net> 2866* Arron Wang <arron.wang@intel.com> 2867* Bo Chen <chen.bo@intel.com> 2868* Dr. David Alan Gilbert <dgilbert@redhat.com> 2869* Henry Wang <Henry.Wang@arm.com> 2870* Hui Zhu <teawater@antfin.com> 2871* LiYa'nan <oliverliyn@gmail.com> 2872* Michael Zhao <michael.zhao@arm.com> 2873* Rob Bradford <robert.bradford@intel.com> 2874* Samuel Ortiz <sameo@linux.intel.com> 2875* Sebastien Boeuf <sebastien.boeuf@intel.com> 2876* Sergio Lopez <slp@redhat.com> 2877 2878# v0.7.0 2879 2880Highlights for `cloud-hypervisor` version 0.7.0 include: 2881 2882### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug 2883 2884Further to our effort to support modifying a running guest we now support 2885hotplug and unplug of the following virtio backed devices: block, network, 2886pmem, virtio-fs and vsock. This functionality is available on the (default) PCI 2887based transport and is exposed through the HTTP API. The `ch-remote` utility 2888provides a CLI for adding or removing these device types after the VM has 2889booted. User can use the `id` parameter on the devices to choose names for 2890devices to ease their removal. 2891 2892### Alternative `libc` Support 2893 2894Cloud Hypervisor can now be compiled with the `musl` C library and this release 2895contains a static binary compiled using that toolchain. 2896 2897### Multithreaded Multi Queued `vhost-user` Backends 2898 2899The `vhost-user` backends for network and block support that are shipped by 2900Cloud Hypervisor have been enhanced to support multiple threads and queues to 2901improve throughput. These backends are used automatically if `vhost_user=true` 2902is passed when the devices are created. 2903 2904### Initial RamFS Support 2905 2906By passing the `--initramfs` command line option the user can specify a file to 2907be loaded into the guest memory to be used as the kernel initial filesystem. 2908This is usually used to allow the loading of drivers needed to be able to 2909access the real root filesystem but it can also be used standalone for a very 2910minimal image. 2911 2912### Alternative Memory Hotplug: `virtio-mem` 2913 2914As well as supporting ACPI based hotplug Cloud Hypervisor now supports using 2915the `virtio-mem` hotplug alternative. This can be controlled by the 2916`hotplug_method` parameter on the `--memory` command line option. It currently 2917requires kernel patches to be able to support it. 2918 2919### `Seccomp` Sandboxing 2920 2921Cloud Hypervisor now has support for restricting the system calls that the 2922process can use via the `seccomp` security API. This on by default and is 2923controlled by the `--seccomp` command line option. 2924 2925### Updated Distribution Support 2926 2927With the release of Ubuntu 20.04 we have added that to the list of supported 2928distributions and is part of our regular testing programme. 2929 2930### Command Line and API Changes 2931 2932This is non exhaustive list of HTTP API and command line changes 2933 2934* New `id` fields added for devices to allow them to be named to ease removal. 2935 If no name is specified the VMM chooses one. 2936* Use `--memory`'s `shared` and `hugepages` controls for determining backing 2937 memory instead of providing a path. 2938* The `--vsock` parameter only takes one device as the Linux kernel only 2939 supports a single Vsock device. The REST API has removed the vector for this 2940 option and replaced it with a single optional field. 2941* There is enhanced validation of the command line and API provided 2942 configurations to ensure that the provided options are compatible e.g. that 2943 shared memory is in use if any attempt is made to used a `vhost-user` backed 2944 device. 2945* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and 2946 `add-vsock` subcommands. For removal `remove-device` is used. The REST API 2947 has appropriate new HTTP endpoints too. 2948* Specifying a `size` with `--pmem` is no longer required and instead the size 2949 will be obtained from the file. A `discard_writes` option has also been added 2950 to provide the equivalent of a read-only file. 2951* The parameters to `--block-backend` have been changed to more closely align 2952 with those used by `--disk`. 2953 2954### Contributors 2955 2956Many thanks to everyone who has contributed to our 0.7.0 release including some new faces. 2957 2958* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 2959* Bo Chen <chen.bo@intel.com> 2960* Cathy Zhang <cathy.zhang@intel.com> 2961* Damjan Georgievski <gdamjan@gmail.com> 2962* Dean Sheather <dean@coder.com> 2963* Eryu Guan <eguan@linux.alibaba.com> 2964* Hui Zhu <teawater@antfin.com> 2965* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2966* Martin Xu <martin.xu@intel.com> 2967* Muminul Islam <muislam@microsoft.com> 2968* Rob Bradford <robert.bradford@intel.com> 2969* Samuel Ortiz <sameo@linux.intel.com> 2970* Sebastien Boeuf <sebastien.boeuf@intel.com> 2971* Sergio Lopez <slp@redhat.com> 2972* Yang Zhong <yang.zhong@intel.com> 2973* Yi Sun <yi.y.sun@linux.intel.com> 2974 2975# v0.6.0 2976 2977Highlights for `cloud-hypervisor` version 0.6.0 include: 2978 2979### Directly Assigned Devices Hotplug 2980 2981We continued our efforts around supporting dynamically changing the guest 2982resources. After adding support for CPU and memory hotplug, Cloud Hypervisor 2983now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`) 2984devices into an already running guest. This closes the features gap for 2985providing a complete Kata Containers workloads support with Cloud Hypervisor. 2986 2987### Shared Filesystem Improvements 2988 2989We enhanced our shared filesystem support through many `virtio-fs` improvements. 2990By adding support for DAX, parallel processing of multiple requests, `FS_IO`, 2991`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we 2992improved our filesystem sharing performance, but also made it more stable and 2993compatible with other `virtio-fs` implementations. 2994 2995### Block and Networking IO Self Offloading 2996 2997When choosing to offload the paravirtualized block and networking I/O to an 2998external process (through the `vhost-user` protocol), Cloud Hypervisor now 2999automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends 3000into their own, separate processes. 3001This provides a seamless paravirtualized I/O user experience for those who want 3002to run their guest I/O into separate executions contexts. 3003 3004### Command Line Interface 3005 3006More and more Cloud Hypervisor services are exposed through the 3007[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only 3008accessible via relatively cumbersome HTTP calls. In order to abstract 3009those calls into a more user friendly tool, we created a Cloud Hypervisor 3010Command Line Interface (CLI) called `ch-remote`. The `ch-remote` binary 3011is created with each build and available e.g. at 3012`cloud-hypervisor/target/debug/ch-remote` when doing a debug build. 3013 3014Please check `ch-remote --help` for a complete description of all available 3015commands. 3016 3017### PVH Boot 3018 3019In addition to the traditional Linux boot protocol, Cloud Hypervisor now 3020supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html). 3021 3022### Contributors 3023 3024With the 0.6.0 release, we are welcoming a few new contributors. Many thanks 3025to them and to everyone that contributed to this release: 3026 3027* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 3028* Arron Wang <arron.wang@intel.com> 3029* Bin Liu <liubin0329@gmail.com> 3030* Bo Chen <chen.bo@intel.com> 3031* Cathy Zhang <cathy.zhang@intel.com> 3032* Eryu Guan <eguan@linux.alibaba.com> 3033* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 3034* Liu Bo <bo.liu@linux.alibaba.com> 3035* Qiu Wenbo <qiuwenbo@phytium.com.cn> 3036* Rob Bradford <robert.bradford@intel.com> 3037* Samuel Ortiz <sameo@linux.intel.com> 3038* Sebastien Boeuf <sebastien.boeuf@intel.com> 3039* Sergio Lopez <slp@redhat.com> 3040 3041# v0.5.1 3042 3043This is a bugfix release branched off v0.5.0. It contains the following fixes: 3044 3045* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez 3046* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf 3047* Fixed error reporting for resize command - Samuel Ortiz 3048* Double reboot workaround (#783) - Rob Bradford 3049* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford 3050 3051# v0.5.0 3052 3053Highlights for `cloud-hypervisor` version 0.5.0 include: 3054 3055### Virtual Machine Dynamic Resizing 3056 3057With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and 3058memory hot plug as well. This allows to dynamically resize Cloud Hypervisor 3059guests which is needed for e.g. Kubernetes related use cases. 3060The memory hot plug implementation is based on the same framework as the CPU hot 3061plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest. 3062 3063Next on our VM resizing roadmap is the PCI devices hotplug feature. 3064 3065### Multi-Queue, Multi-Threaded Paravirtualization 3066 3067We enhanced our virtio networking and block support by having both devices use 3068multiple I/O queues handled by multiple threads. This improves our default 3069paravirtualized networking and block devices throughput. 3070 3071### New Interrupt Management Framework 3072 3073We improved our interrupt management implementation by introducing an Interrupt 3074Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device) 3075crates discussions. This move made the code significantly cleaner, and allowed 3076us to remove several KVM related dependencies from crates like the PCI and 3077virtio ones. 3078 3079### Development Tools 3080 3081In order to provide a better developer experience, we worked on improving our 3082build, development and testing tools. 3083Somehow similar to the excellent 3084[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool), 3085we now provide a [dev_cli script](scripts/dev_cli.sh). 3086 3087With this new tool, our users and contributors will be able to build and test 3088Cloud Hypervisor through a containerized environment. 3089 3090### Kata Containers Integration 3091 3092We spent some significant time and efforts debugging and fixing our integration 3093with the [Kata Containers](https://github.com/kata-containers) project. Cloud 3094Hypervisor is now a fully supported Kata Containers hypervisor, and is 3095integrated into the project's CI. 3096 3097### Contributors 3098 3099Many thanks to everyone that contributed to the 0.5.0 release: 3100 3101* Bo Chen <chen.bo@intel.com> 3102* Cathy Zhang <cathy.zhang@intel.com> 3103* Qiu Wenbo <qiuwenbo@phytium.com.cn> 3104* Rob Bradford <robert.bradford@intel.com> 3105* Samuel Ortiz <sameo@linux.intel.com> 3106* Sebastien Boeuf <sebastien.boeuf@intel.com> 3107* Sergio Lopez <slp@redhat.com> 3108* Yang Zhong <yang.zhong@intel.com> 3109 3110# v0.4.0 3111 3112Highlights for `cloud-hypervisor` version 0.4.0 include: 3113 3114### Dynamic virtual CPUs addition 3115 3116As a way to vertically scale Cloud Hypervisor guests, we now support dynamically 3117adding virtual CPUs to the guests, a mechanism also known as CPU hot plug. 3118Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs 3119to an already running guest and the high level operations for that process are 3120documented [here](docs/hotplug.md) 3121 3122During the next release cycles we are planning to extend Cloud Hypervisor 3123hot plug framework to other resources, namely PCI devices and memory. 3124 3125### Programmatic firmware tables generation 3126 3127As part of the CPU hot plug feature enablement, and as a requirement for hot 3128plugging other resources like devices or RAM, we added support for 3129programmatically generating the needed ACPI tables. Through a dedicated 3130`acpi-tables` crate, we now have a flexible and clean way of generating those 3131tables based on the VMM device model and topology. 3132 3133### Filesystem and block devices vhost-user backends 3134 3135Our objective of running all Cloud Hypervisor paravirtualized I/O to a 3136vhost-user based framework is getting closer as we've added Rust based 3137implementations for vhost-user-blk and virtiofs backends. Together with the 3138vhost-user-net backend that came with the 0.3.0 release, this will form the 3139default Cloud Hypervisor I/O architecture. 3140 3141### Guest pause and resume 3142 3143As an initial requirement for enabling live migration, we added support for 3144pausing and resuming any VMM components. As an intermediate step towards live 3145migration, the upcoming guest snapshotting feature will be based on the pause 3146and resume capabilities. 3147 3148### Userspace IOAPIC by default 3149 3150As a way to simplify our device manager implementation, but also in order to 3151stay away from privileged rings as often as possible, any device that relies on 3152pin based interrupts will be using the userspace IOAPIC implementation by 3153default. 3154 3155### PCI BAR reprogramming 3156 3157In order to allow for a more flexible device model, and also support guests 3158that would want to move PCI devices, we added support for PCI devices BAR 3159reprogramming. 3160 3161### New `cloud-hypervisor` organization 3162 3163As we wanted to be more flexible on how we manage the Cloud Hypervisor project, 3164we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor). 3165Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) 3166project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux) 3167and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 3168repositories. We may also use it to host any rust-vmm that we'd need to 3169temporarily fork. 3170Thanks to GitHub's seamless repository redirections, the move is completely 3171transparent to all Cloud Hypervisor contributors, users and followers. 3172 3173### Contributors 3174 3175Many thanks to everyone that contributed to the 0.4.0 release: 3176 3177* Cathy Zhang <cathy.zhang@intel.com> 3178* Emin Ghuliev <drmint80@gmail.com> 3179* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 3180* Qiu Wenbo <qiuwenbo@phytium.com.cn> 3181* Rob Bradford <robert.bradford@intel.com> 3182* Samuel Ortiz <sameo@linux.intel.com> 3183* Sebastien Boeuf <sebastien.boeuf@intel.com> 3184* Sergio Lopez <slp@redhat.com> 3185* Wu Zongyong <wuzongyong@linux.alibaba.com> 3186 3187# v0.3.0 3188 3189Highlights for `cloud-hypervisor` version 0.3.0 include: 3190 3191### Block device offloading 3192 3193We continue to work on offloading paravirtualized I/O to external processes, 3194and we added support for 3195[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends. 3196This enables `cloud-hypervisor` users to plug a `vhost-user` based block device 3197like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage 3198backend. 3199 3200### Network device backend 3201 3202The previous release provided support for 3203[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we 3204also provide a TAP based vhost-user-net backend, implemented in Rust. Together 3205with the vhost-user-net device implementation, this will eventually become the 3206Cloud Hypervisor default paravirtualized networking architecture. 3207 3208### Virtual sockets 3209 3210In order to more efficiently and securely communicate between host and guest, 3211we added an hybrid implementation of the 3212[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address 3213family over virtio. Credits go to the 3214[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md) 3215project as our implementation is a copy of theirs. 3216 3217### HTTP based API 3218 3219In anticipation of the need to support asynchronous operations to Cloud 3220Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP 3221based API to the VMM. The API will be more extensively documented during the 3222next release cycle. 3223 3224### Memory mapped virtio transport 3225 3226In order to support potential PCI-free use cases, we added support for the 3227[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002) 3228transport layer. This will allow us to support simple, minimal guest 3229configurations that do not require a PCI bus emulation. 3230 3231### Paravirtualized IOMMU 3232 3233As we want to improve our nested guests support, we added support for exposing 3234a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows 3235for a safer nested virtio and directly assigned devices support. 3236 3237To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor 3238users to be able to specify if devices had to be handled through this virtual 3239IOMMU or not. In particular, the `--disk` option now expects disk paths to be 3240prefixed with a `path=` string, and supports an optional `iommu=[on|off]` 3241setting. 3242 3243### Ubuntu 19.10 3244 3245With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware), 3246we can now support the latest 3247[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images. 3248 3249### Large memory guests 3250 3251After simplifying and changing our guest address space handling, we can now 3252support guests with large amount of memory (more than 64GB). 3253 3254# v0.2.0 3255 3256Highlights for `cloud-hypervisor` version 0.2.0 include: 3257 3258### Network device offloading 3259 3260As part of our general effort to offload paravirtualized I/O to external 3261processes, we added support for 3262[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This 3263enables `cloud-hypervisor` users to plug a `vhost-user` based networking device 3264(e.g. [DPDK](https://www.dpdk.org)) into the VMM as their virtio network backend. 3265 3266### Minimal hardware-reduced ACPI 3267 3268In order to properly implement and guest reset and shutdown, we implemented 3269a minimal version of the hardware-reduced ACPI specification. Together with 3270a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to 3271cleanly reboot and shutdown. 3272 3273The ACPI implementation is a `cloud-hypervisor` build time option that is 3274enabled by default. 3275 3276### Debug I/O port 3277 3278Based on the Firecracker idea of using a dedicated I/O port to measure guest 3279boot times, we added support for logging guest events through the 3280[0x80](https://web.archive.org/web/20211028033025/https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html) 3281PC debug port. This allows, among other things, for granular guest boot time 3282measurements. See our [debug port documentation](docs/debug-port.md) for more 3283details. 3284 3285### Improved direct device assignment 3286 3287We fixed a major performance issue with our initial VFIO implementation: When 3288enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads 3289were (in many cases) not cached. After correctly tagging the guest memory from 3290`cloud-hypervisor` we're now able to reach the expected performance from 3291directly assigned devices. 3292 3293### Improved shared filesystem 3294 3295We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt) 3296support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system. 3297This provides better shared filesystem IO performance with a smaller guest 3298memory footprint. 3299 3300### Ubuntu bionic based CI 3301 3302Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 3303improvements, we are now able to boot Ubuntu bionic images. We added those to 3304our CI pipeline. 3305 3306# v0.1.0 3307 3308Highlights for `cloud-hypervisor` version 0.1.0 include: 3309 3310### Shared filesystem 3311 3312We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file 3313system, allowing for an efficient and reliable way of sharing a filesystem 3314between the host and the `cloud-hypervisor` guest. 3315 3316See our [filesystem sharing](docs/fs.md) documentation for more details on how 3317to use virtio-fs with `cloud-hypervisor`. 3318 3319### Initial direct device assignment support 3320 3321VFIO (Virtual Function I/O) is a kernel framework that exposes direct device 3322access to userspace. `cloud-hypervisor` uses VFIO to directly assign host 3323physical devices into its guest. 3324 3325See our [VFIO](docs/vfio.md) documentation for more detail on how to directly 3326assign host devices to `cloud-hypervisor` guests. 3327 3328### Userspace IOAPIC 3329 3330`cloud-hypervisor` supports a so-called split IRQ chip implementation by 3331implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC). 3332By moving part of the IRQ chip implementation from kernel space to user space, 3333the IRQ chip emulation does not always run in a fully privileged mode. 3334 3335### Virtual persistent memory 3336 3337The `virtio-pmem` implementation emulates a virtual persistent memory device 3338that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device 3339allows to bypass the guest page cache and improve the guest memory footprint. 3340 3341### Linux kernel bzImage 3342 3343The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from 3344`bzImage` kernel images, which is usually the format that Linux distributions 3345use to ship their kernels. For example, this allows for booting from the host 3346distribution kernel image. 3347 3348### Console over virtio 3349 3350`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although 3351using this device as a guest console can potentially cut some early boot 3352messages, it can reduce the guest boot time and provides a complete console 3353implementation. 3354 3355The `virtio-console` device is enabled by default for the guest console. 3356Switching back to the legacy serial port is done by selecting 3357`--serial tty --console off` from the command line. 3358 3359### Unit testing 3360 3361We now run all unit tests from all our crates directly from our CI. 3362 3363### Integration tests parallelization 3364 3365The CI cycle run time has been significantly reduced by refactoring our 3366integration tests; allowing them to all be run in parallel. 3367