1- [v45.0](#v450) 2 - [Experimental `riscv64` Architecture Support](#experimental-riscv64-architecture-support) 3 - [Alphabetically Sorted CLI Options](#alphabetically-sorted-cli-options) 4 - [Improved Downtime from VM Live Migration](#improved-downtime-from-vm-live-migration) 5 - [Notable Bug Fixes](#notable-bug-fixes) 6 - [Contributors](#contributors) 7- [v44.0](#v440) 8 - [Configurable `virtio-iommu` Address Width](#configurable-virtio-iommu-address-width) 9 - [Notable Performance Improvements](#notable-performance-improvements) 10 - [New Fuzzers](#new-fuzzers) 11 - [Notable Bug Fixes](#notable-bug-fixes-1) 12 - [Contributors](#contributors-1) 13- [v43.0](#v430) 14 - [Live Migration over TCP Connections](#live-migration-over-tcp-connections) 15 - [Notable Performance Improvements](#notable-performance-improvements-1) 16 - [Notable Bug Fixes](#notable-bug-fixes-2) 17 - [Contributors](#contributors-2) 18- [v42.0](#v420) 19 - [SVE/SVE2 Support on AArch64](#svesve2-support-on-aarch64) 20 - [Notable Bug Fixes](#notable-bug-fixes-3) 21 - [Sponsorships](#sponsorships) 22 - [Contributors](#contributors-3) 23- [v41.0](#v410) 24 - [Experimental "Pvmemcontrol" Support](#experimental-pvmemcontrol-support) 25 - [Sandboxing With Landlock Support](#sandboxing-with-landlock-support) 26 - [Notable Performance Improvements](#notable-performance-improvements-2) 27 - [Notable Bug Fixes](#notable-bug-fixes-4) 28 - [Contributors](#contributors-4) 29- [v40.0](#v400) 30 - [Support for Restoring File Descriptor Backed Network Devices](#support-for-restoring-file-descriptor-backed-network-devices) 31 - [Notable Bug Fixes](#notable-bug-fixes-5) 32 - [Contributors](#contributors-5) 33- [v39.0](#v390) 34 - [Variable Sizing of PCI Apertures for Segments](#variable-sizing-of-pci-apertures-for-segments) 35 - [Direct Booting with bzImages](#direct-booting-with-bzimages) 36 - [Support for NVIDIA GPUDirect P2P Support](#support-for-nvidia-gpudirect-p2p-support) 37 - [Guest NMI Injection Support](#guest-nmi-injection-support) 38 - [Notable Bug Fixes](#notable-bug-fixes-6) 39 - [Contributors](#contributors-6) 40- [v38.0](#v380) 41 - [Group Rate Limiter on Block Devices](#group-rate-limiter-on-block-devices) 42 - [CPU Pinning Support for Block Device Worker Thread](#cpu-pinning-support-for-block-device-worker-thread) 43 - [Optimized Boot Time with Parallel Memory Prefault](#optimized-boot-time-with-parallel-memory-prefault) 44 - [New 'debug-console' Device](#new-debug-console-device) 45 - [Improved VFIO Device Support](#improved-vfio-device-support) 46 - [Extended CPU Affinity Support](#extended-cpu-affinity-support) 47 - [Notable Bug Fixes](#notable-bug-fixes-7) 48 - [Contributors](#contributors-7) 49- [v37.0](#v370) 50 - [Long Term Support (LTS) Release](#long-term-support-lts-release) 51 - [Multiple PCI segments Support for 32-bit VFIO devices](#multiple-pci-segments-support-for-32-bit-vfio-devices) 52 - [Configurable Named TAP Devices](#configurable-named-tap-devices) 53 - [TTY Output from Both Serial Device and Virtio Console](#tty-output-from-both-serial-device-and-virtio-console) 54 - [Faster VM Restoration from Snapshots](#faster-vm-restoration-from-snapshots) 55 - [Notable Bug Fixes](#notable-bug-fixes-8) 56 - [Contributors](#contributors-8) 57- [v36.0](#v360) 58 - [Command Line Changes](#command-line-changes) 59 - [Enabled Features Reported via API Endpoint and CLI](#enabled-features-reported-via-api-endpoint-and-cli) 60 - [NUMA Support for PCI segments](#numa-support-for-pci-segments) 61 - [CPU Topology Support on AMD Platforms](#cpu-topology-support-on-amd-platforms) 62 - [Unix Socket Backend for Serial Port](#unix-socket-backend-for-serial-port) 63 - [AIO Backend for Block Devices](#aio-backend-for-block-devices) 64 - [Documentation Improvements](#documentation-improvements) 65 - [Notable Bug Fixes](#notable-bug-fixes-9) 66 - [Contributors](#contributors-9) 67- [v35.0](#v350) 68 - [`virtio-vsock` Support for Linux Guest Kernel v6.3+](#virtio-vsock-support-for-linux-guest-kernel-v63) 69 - [User Specified Serial Number for `virtio-block`](#user-specified-serial-number-for-virtio-block) 70 - [vCPU TSC Frequency Included in Migration State](#vcpu-tsc-frequency-included-in-migration-state) 71 - [Notable Bug Fixes](#notable-bug-fixes-10) 72 - [Contributors](#contributors-10) 73- [v34.0](#v340) 74 - [Paravirtualised Panic Device Support](#paravirtualised-panic-device-support) 75 - [Improvements to VM Core Dump](#improvements-to-vm-core-dump) 76 - [QCOW2 Support for Backing Files](#qcow2-support-for-backing-files) 77 - [Minimum Host Kernel Bump](#minimum-host-kernel-bump) 78 - [Notable Bug Fixes](#notable-bug-fixes-11) 79 - [Contributors](#contributors-11) 80- [v33.0](#v330) 81 - [D-Bus based API](#d-bus-based-api) 82 - [Expose Host CPU Cache Details for AArch64](#expose-host-cpu-cache-details-for-aarch64) 83 - [Notable Bug Fixes](#notable-bug-fixes-12) 84 - [Contributors](#contributors-12) 85- [v32.0](#v320) 86 - [Increased PCI Segment Limit](#increased-pci-segment-limit) 87 - [API Changes](#api-changes) 88 - [Notable Bug Fixes](#notable-bug-fixes-13) 89 - [Contributors](#contributors-13) 90- [v31.1](#v311) 91- [v31.0](#v310) 92 - [Update to Latest `acpi_tables`](#update-to-latest-acpi_tables) 93 - [Update Reference Kernel to 6.2](#update-reference-kernel-to-62) 94 - [Improvements on Console `SIGWINCH` Handler](#improvements-on-console-sigwinch-handler) 95 - [Remove Directory Support from `MemoryZoneConfig::file`](#remove-directory-support-from-memoryzoneconfigfile) 96 - [Documentation Improvements](#documentation-improvements-1) 97 - [Notable Bug Fixes](#notable-bug-fixes-14) 98 - [Contributors](#contributors-14) 99- [v30.0](#v300) 100 - [Command Line Changes for Reduced Binary Size](#command-line-changes-for-reduced-binary-size) 101 - [Basic vfio-user Server Support](#basic-vfio-user-server-support) 102 - [Heap Profiling Support](#heap-profiling-support) 103 - [Documentation Improvements](#documentation-improvements-2) 104 - [Notable Bug Fixes](#notable-bug-fixes-15) 105 - [Contributors](#contributors-15) 106- [v28.2](#v282) 107- [v29.0](#v290) 108 - [Release Binary Supports Both MSHV and KVM](#release-binary-supports-both-mshv-and-kvm) 109 - [Snapshot/Restore and Live Migration Improvements](#snapshotrestore-and-live-migration-improvements) 110 - [Heap Allocation Improvements](#heap-allocation-improvements) 111 - [`ch-remote` Improvements](#ch-remote-improvements) 112 - [`AArch64` Documentation Integration](#aarch64-documentation-integration) 113 - [`virtio-block` Counters Enhancement](#virtio-block-counters-enhancement) 114 - [TCP Offload Control](#tcp-offload-control) 115 - [Notable Bug Fixes](#notable-bug-fixes-16) 116 - [Removals](#removals) 117 - [Deprecations](#deprecations) 118 - [Contributors](#contributors-16) 119- [v28.1](#v281) 120- [v28.0](#v280) 121 - [Community Engagement (Reminder)](#community-engagement-reminder) 122 - [Long Term Support (LTS) Release](#long-term-support-lts-release-1) 123 - [Virtualised TPM Support](#virtualised-tpm-support) 124 - [Transparent Huge Page Support](#transparent-huge-page-support) 125 - [README Quick Start Improved](#readme-quick-start-improved) 126 - [Notable Bug Fixes](#notable-bug-fixes-17) 127 - [Removals](#removals-1) 128 - [Contributors](#contributors-17) 129- [v27.0](#v270) 130 - [Community Engagement](#community-engagement) 131 - [Prebuilt Packages](#prebuilt-packages) 132 - [Network Device MTU Exposed to Guest](#network-device-mtu-exposed-to-guest) 133 - [Boot Tracing](#boot-tracing) 134 - [Simplified Build Feature Flags](#simplified-build-feature-flags) 135 - [Asynchronous Kernel Loading](#asynchronous-kernel-loading) 136 - [GDB Support for AArch64](#gdb-support-for-aarch64) 137 - [Notable Bug Fixes](#notable-bug-fixes-18) 138 - [Deprecations](#deprecations-1) 139 - [Contributors](#contributors-18) 140- [v26.0](#v260) 141 - [SMBIOS Improvements via `--platform`](#smbios-improvements-via---platform) 142 - [Unified Binary MSHV and KVM Support](#unified-binary-mshv-and-kvm-support) 143 - [Notable Bug Fixes](#notable-bug-fixes-19) 144 - [Deprecations](#deprecations-2) 145 - [Removals](#removals-2) 146 - [Contributors](#contributors-19) 147- [v25.0](#v250) 148 - [`ch-remote` Improvements](#ch-remote-improvements-1) 149 - [VM "Coredump" Support](#vm-coredump-support) 150 - [Notable Bug Fixes](#notable-bug-fixes-20) 151 - [Removals](#removals-3) 152 - [Contributors](#contributors-20) 153- [v24.0](#v240) 154 - [Bypass Mode for `virtio-iommu`](#bypass-mode-for-virtio-iommu) 155 - [Ensure Identifiers Uniqueness](#ensure-identifiers-uniqueness) 156 - [Sparse Mmap support](#sparse-mmap-support) 157 - [Expose Platform Serial Number](#expose-platform-serial-number) 158 - [Notable Bug Fixes](#notable-bug-fixes-21) 159 - [Notable Improvements](#notable-improvements) 160 - [Deprecations](#deprecations-3) 161 - [New on the Website](#new-on-the-website) 162 - [Contributors](#contributors-21) 163- [v23.1](#v231) 164- [v23.0](#v230) 165 - [vDPA Support](#vdpa-support) 166 - [Updated OS Support list](#updated-os-support-list) 167 - [`AArch64` Memory Map Improvements](#aarch64-memory-map-improvements) 168 - [`AMX` Support](#amx-support) 169 - [Notable Bug Fixes](#notable-bug-fixes-22) 170 - [Deprecations](#deprecations-4) 171 - [Contributors](#contributors-22) 172- [v22.1](#v221) 173- [v22.0](#v220) 174 - [GDB Debug Stub Support](#gdb-debug-stub-support) 175 - [`virtio-iommu` Backed Segments](#virtio-iommu-backed-segments) 176 - [Before Boot Configuration Changes](#before-boot-configuration-changes) 177 - [`virtio-balloon` Free Page Reporting](#virtio-balloon-free-page-reporting) 178 - [Support for Direct Kernel Booting with TDX](#support-for-direct-kernel-booting-with-tdx) 179 - [PMU Support for AArch64](#pmu-support-for-aarch64) 180 - [Documentation Under CC-BY-4.0 License](#documentation-under-cc-by-40-license) 181 - [Deprecation of "Classic" `virtiofsd`](#deprecation-of-classic-virtiofsd) 182 - [Notable Bug Fixes](#notable-bug-fixes-23) 183 - [Contributors](#contributors-23) 184- [v21.0](#v210) 185 - [Efficient Local Live Migration (for Live Upgrade)](#efficient-local-live-migration-for-live-upgrade) 186 - [Recommended Kernel is Now 5.15](#recommended-kernel-is-now-515) 187 - [Notable Bug fixes](#notable-bug-fixes-24) 188 - [Contributors](#contributors-24) 189- [v20.2](#v202) 190- [v20.1](#v201) 191- [v20.0](#v200) 192 - [Multiple PCI segments support](#multiple-pci-segments-support) 193 - [CPU pinning](#cpu-pinning) 194 - [Improved VFIO support](#improved-vfio-support) 195 - [Safer code](#safer-code) 196 - [Extended documentation](#extended-documentation) 197 - [Notable bug fixes](#notable-bug-fixes-25) 198 - [Contributors](#contributors-25) 199- [v19.0](#v190) 200 - [Improved PTY handling for serial and `virtio-console`](#improved-pty-handling-for-serial-and-virtio-console) 201 - [PCI boot time optimisations](#pci-boot-time-optimisations) 202 - [Improved TDX support](#improved-tdx-support) 203 - [Live migration enhancements](#live-migration-enhancements) 204 - [`virtio-mem` support with `vfio-user`](#virtio-mem-support-with-vfio-user) 205 - [AArch64 for `virtio-iommu`](#aarch64-for-virtio-iommu) 206 - [Notable bug fixes](#notable-bug-fixes-26) 207 - [Contributors](#contributors-26) 208- [v18.0](#v180) 209 - [Experimental User Device (`vfio-user`) support](#experimental-user-device-vfio-user-support) 210 - [Migration support for `vhost-user` devices](#migration-support-for-vhost-user-devices) 211 - [VHDX disk image support](#vhdx-disk-image-support) 212 - [Device pass through on MSHV hypervisor](#device-pass-through-on-mshv-hypervisor) 213 - [AArch64 for support `virtio-mem`](#aarch64-for-support-virtio-mem) 214 - [Live migration on MSHV hypervisor](#live-migration-on-mshv-hypervisor) 215 - [AArch64 CPU topology support](#aarch64-cpu-topology-support) 216 - [Power button support on AArch64](#power-button-support-on-aarch64) 217 - [Notable bug fixes](#notable-bug-fixes-27) 218 - [Contributors](#contributors-27) 219- [v17.0](#v170) 220 - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi) 221 - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend) 222 - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices) 223 - [Improved SGX support](#improved-sgx-support) 224 - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices) 225 - [Notable bug fixes](#notable-bug-fixes-28) 226 - [Contributors](#contributors-28) 227- [v16.0](#v160) 228 - [Improved live migration support](#improved-live-migration-support) 229 - [Improved `vhost-user` support](#improved-vhost-user-support) 230 - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support) 231 - [Notable bug fixes](#notable-bug-fixes-29) 232 - [Removed functionality](#removed-functionality) 233 - [Contributors](#contributors-29) 234- [v15.0](#v150) 235 - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees) 236 - [Network device rate limiting](#network-device-rate-limiting) 237 - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload) 238 - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter) 239 - [Bug fixes](#bug-fixes) 240 - [Deprecations](#deprecations-5) 241 - [Contributors](#contributors-30) 242- [v0.14.1](#v0141) 243- [v0.14.0](#v0140) 244 - [Structured event monitoring](#structured-event-monitoring) 245 - [MSHV improvements](#mshv-improvements) 246 - [Improved aarch64 platform](#improved-aarch64-platform) 247 - [Updated hotplug documentation](#updated-hotplug-documentation) 248 - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console) 249 - [Block device rate limiting](#block-device-rate-limiting) 250 - [Deprecations](#deprecations-6) 251 - [Contributors](#contributors-31) 252- [v0.13.0](#v0130) 253 - [Wider VFIO device support](#wider-vfio-device-support) 254 - [Improved huge page support](#improved-huge-page-support) 255 - [MACvTAP support](#macvtap-support) 256 - [VHD disk image support](#vhd-disk-image-support) 257 - [Improved Virtio device threading](#improved-virtio-device-threading) 258 - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button) 259 - [Contributors](#contributors-32) 260- [v0.12.0](#v0120) 261 - [ARM64 enhancements](#arm64-enhancements) 262 - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning) 263 - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend) 264 - [Enhanced "info" API](#enhanced-info-api) 265 - [Contributors](#contributors-33) 266- [v0.11.0](#v0110) 267 - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block) 268 - [Windows Guest Support](#windows-guest-support) 269 - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation) 270 - [`virtio-mmio` Removal](#virtio-mmio-removal) 271 - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64) 272 - [Improved Linux Boot Time](#improved-linux-boot-time) 273 - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling) 274 - [Default Log Level Changed](#default-log-level-changed) 275 - [New `--balloon` Parameter Added](#new---balloon-parameter-added) 276 - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support) 277 - [Notable Bug Fixes](#notable-bug-fixes-30) 278 - [Contributors](#contributors-34) 279- [v0.10.0](#v0100) 280 - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors) 281 - [Memory Zones](#memory-zones) 282 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements) 283 - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control) 284 - [Notable Bug Fixes](#notable-bug-fixes-31) 285 - [Contributors](#contributors-35) 286- [v0.9.0](#v090) 287 - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support) 288 - [Block and Network Device Statistics](#block-and-network-device-statistics) 289 - [HTTP API Responses](#http-api-responses) 290 - [CPU Topology](#cpu-topology) 291 - [Release Build Optimization](#release-build-optimization) 292 - [Hypervisor Abstraction](#hypervisor-abstraction) 293 - [Snapshot/Restore Improvements](#snapshotrestore-improvements) 294 - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support) 295 - [Enhancements to ARM64 Support](#enhancements-to-arm64-support) 296 - [Intel SGX Support](#intel-sgx-support) 297 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1) 298 - [Notable Bug Fixes](#notable-bug-fixes-32) 299 - [Contributors](#contributors-36) 300- [v0.8.0](#v080) 301 - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support) 302 - [Experimental ARM64 Support](#experimental-arm64-support) 303 - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests) 304 - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices) 305 - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements) 306 - [Notable Bug Fixes](#notable-bug-fixes-33) 307 - [Command Line and API Changes](#command-line-and-api-changes) 308 - [Contributors](#contributors-37) 309- [v0.7.0](#v070) 310 - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug) 311 - [Alternative `libc` Support](#alternative-libc-support) 312 - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends) 313 - [Initial RamFS Support](#initial-ramfs-support) 314 - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem) 315 - [`Seccomp` Sandboxing](#seccomp-sandboxing) 316 - [Updated Distribution Support](#updated-distribution-support) 317 - [Command Line and API Changes](#command-line-and-api-changes-1) 318 - [Contributors](#contributors-38) 319- [v0.6.0](#v060) 320 - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug) 321 - [Shared Filesystem Improvements](#shared-filesystem-improvements) 322 - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading) 323 - [Command Line Interface](#command-line-interface) 324 - [PVH Boot](#pvh-boot) 325 - [Contributors](#contributors-39) 326- [v0.5.1](#v051) 327- [v0.5.0](#v050) 328 - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing) 329 - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization) 330 - [New Interrupt Management Framework](#new-interrupt-management-framework) 331 - [Development Tools](#development-tools) 332 - [Kata Containers Integration](#kata-containers-integration) 333 - [Contributors](#contributors-40) 334- [v0.4.0](#v040) 335 - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition) 336 - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation) 337 - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends) 338 - [Guest pause and resume](#guest-pause-and-resume) 339 - [Userspace IOAPIC by default](#userspace-ioapic-by-default) 340 - [PCI BAR reprogramming](#pci-bar-reprogramming) 341 - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization) 342 - [Contributors](#contributors-41) 343- [v0.3.0](#v030) 344 - [Block device offloading](#block-device-offloading) 345 - [Network device backend](#network-device-backend) 346 - [Virtual sockets](#virtual-sockets) 347 - [HTTP based API](#http-based-api) 348 - [Memory mapped virtio transport](#memory-mapped-virtio-transport) 349 - [Paravirtualized IOMMU](#paravirtualized-iommu) 350 - [Ubuntu 19.10](#ubuntu-1910) 351 - [Large memory guests](#large-memory-guests) 352- [v0.2.0](#v020) 353 - [Network device offloading](#network-device-offloading) 354 - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi) 355 - [Debug I/O port](#debug-io-port) 356 - [Improved direct device assignment](#improved-direct-device-assignment) 357 - [Improved shared filesystem](#improved-shared-filesystem) 358 - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci) 359- [v0.1.0](#v010) 360 - [Shared filesystem](#shared-filesystem) 361 - [Initial direct device assignment support](#initial-direct-device-assignment-support) 362 - [Userspace IOAPIC](#userspace-ioapic) 363 - [Virtual persistent memory](#virtual-persistent-memory) 364 - [Linux kernel bzImage](#linux-kernel-bzimage) 365 - [Console over virtio](#console-over-virtio) 366 - [Unit testing](#unit-testing) 367 - [Integration tests parallelization](#integration-tests-parallelization) 368 369# v45.0 370 371This release has been tracked in [v45.0 372group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v44.0%22) 373of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 374 375### Experimental `riscv64` Architecture Support 376 377Cloud Hypervisor now has experimental `riscv64` architecture 378support. Details can be found from the [riscv 379documentation](docs/riscv.md). 380 381### Alphabetically Sorted CLI Options 382 383To improve the readability of CLI options, the output of the `--help` 384now is alphabetically sorted. (#6988) 385 386### Improved Downtime of VM Live Migration 387 388The downtime of VM live migration is reduced via delaying some of the 389tearing down process of the source VM after the destination VM is up and 390running. (#6987) 391 392### Notable Bug Fixes 393 394* Fix seccomp filters related to http-api thread (#6967) 395* Handle cross-page access in the emulator for mshv (#6968) 396 397### Contributors 398 399Many thanks to everyone who has contributed to our release: 400 401* Anatol Belski <anbelski@linux.microsoft.com> 402* Andrew Consroe <aconz2@gmail.com> 403* Bo Chen <bchen@crusoe.ai> 404* Jinank Jain <jinankjain@microsoft.com> 405* Jinrong Liang <cloudliang@tencent.com> 406* Philipp Schuster <philipp.schuster@cyberus-technology.de> 407* Ruoqing He <heruoqing@iscas.ac.cn> 408* Stefan Kober <stefan.kober@cyberus-technology.de> 409* Wei Liu <liuwe@microsoft.com> 410 411# v44.0 412 413This release has been tracked in [v44.0 414group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v44.0%22) 415of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 416 417### Configurable `virtio-iommu` Address Width 418 419The `iommu_address_width` option has been added to `--platform` to allow users 420to limit the `virtio-iommu` address space in the guest. (#6900) 421 422### Notable Performance Improvements 423 424The `VIRTIO_BLK_F_SEG_MAX` feature has been enabled for `virtio-block` devices, 425which brings significant performance improvements on throughput. (#6885) 426 427The `io_uring` entries are no longer forced to use async helper workers, 428delegating the decision to the kernel. This change resolved the issue of having 429excessive amount of worker threads when `io_uring` is being used, which is 430expected to improve performance, such as reducing memory usage and reduce CPU 431contention. 432 433### New Fuzzers 434 435Our continuous fuzzing infrastructure is augmented with two new fuzzers to cover 436x86 instruction emulator and `virtio-vsock`. 437 438### Notable Bug Fixes 439 440* Fix short read and short write that impact QCOW and VHDX support. (#6884) 441* Various bug fixes on VHDX support. (#6890, #6899) 442 443### Contributors 444 445Many thanks to everyone who has contributed to our release: 446 447* Arvind Vasudev <avasudev@crusoeenergy.com> 448* Bo Chen <bo.arvin.chen@gmail.com> 449* Nikolay Edigaryev <edigaryev@gmail.com> 450* Rob Bradford <rbradford@rivosinc.com> 451* Ruoqing He <heruoqing@iscas.ac.cn> 452* Wei Liu <liuwe@microsoft.com> 453 454# v43.0 455 456This release has been tracked in [v43.0 457group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v43.0%22) 458of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 459 460### Live Migration over TCP Connections 461 462Support has been added to enable direct live migration from two hosts via TCP 463connections. This supplements the existing support for migrating over a UNIX 464socket which can then be tunnelled as desired. The documentation has been 465updated. (#6850) 466 467### Notable Performance Improvements 468 469The `VIRTIO_RING_F_INDIRECT_DESC` feature has been enabled for `virtio-block` 470devices. This significantly increases the throughput of the devices with a 471small negative impact on latency. (#6826) 472 473### Notable Bug Fixes 474 475* Cloud Hypervisor now accepts VFIO devices that use I/O PCI BARs on non x86-64 476 architectures. Whether they function depends on the host PCI host bridge 477 support - previously they would be rejected even if the driver did not use 478 these BARs. (#6871) 479* Command line groups were adjusted to ensure that at least one payload 480 parameter was provided if any other VM parameters provided. (#6832) 481 482### Contributors 483 484Many thanks to everyone who has contributed to our release: 485 486* Alyssa Ross <hi@alyssa.is> 487* Andrew Consroe <aconz2@gmail.com> 488* Bo Chen <bo.arvin.chen@gmail.com> 489* Jinrong Liang <cloudliang@tencent.com> 490* Julian Stecklina <julian.stecklina@cyberus-technology.de> 491* Muminul Islam <muislam@microsoft.com> 492* Rob Bradford <rbradford@rivosinc.com> 493* Ruoqing He <heruoqing@iscas.ac.cn> 494* Wojtek Czekalski <wczekalski@me.com> 495 496# v42.0 497 498This release has been tracked in [v42.0 499group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v42.0%22) 500of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 501 502### SVE/SVE2 Support on AArch64 503 504The SVE and SVE2 feature bits are now propagated through to the guest on 505AArch64. (#6678, #6691) 506 507### Notable Bug Fixes 508 509* Reduce latency notification when rate limited (#6672) 510* Fix `virtio-console` resizing (#6704) 511* Fix resizing when console uses TTY (#6754) 512* Avoid deadlock in PCI BAR reprogramming that can occur when adding a new 513 `virtio` device to a VM that has been restored (#6775) 514* Fix console resizing after VM restore (#6748) 515* Fix memory resize error due to incorrect bounds checks (#6736) 516 517### Sponsorships 518 519During this release cycle a new VFIO CI worker has been provided by Crusoe 520Energy and a new ARM64 CI worker has been provided by Ubicloud. 521 522### Contributors 523 524Many thanks to everyone who has contributed to our release: 525 526* Alexandru Matei <alexandru.matei@uipath.com> 527* Alyssa Ross <hi@alyssa.is> 528* Anirudh Rayabharam <anrayabh@microsoft.com> 529* BharatNarasimman <bharatn@microsoft.com> 530* Bo Chen <chen.bo@intel.com> 531* Jinank Jain <jinankjain@microsoft.com> 532* Jonas Scholz <Jonas.Scholz@bbscholz.de> 533* Nuno Das Neves <nudasnev@microsoft.com> 534* Praveen K Paladugu <prapal@linux.microsoft.com> 535* Purna Pavan Chandra <paekkaladevi@microsoft.com> 536* Rob Bradford <rbradford@rivosinc.com> 537* Ruoqing He <heruoqing@iscas.ac.cn> 538* Songqian Li <sionli@tencent.com> 539* Tom Dohrmann <erbse.13@gmx.de> 540* Wei Liu <liuwe@microsoft.com> 541* Wenyu Huang <huangwenyuu@outlook.com> 542* Yuhong Zhong <yz@cs.columbia.edu> 543* wuxinyue <wuxinyue.wxy@antgroup.com> 544 545# v41.0 546 547This release has been tracked in [v41.0 548group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v41.0%22) 549of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 550 551### Experimental "Pvmemcontrol" Support 552 553VMM support has been added for this experimental functionality (requires 554currently out of tree Linux kernel patches) to allow guests to control its 555physical memory properties to allow optimisations and security features. 556(#6318, #6467) 557 558### Sandboxing With Landlock Support 559 560Support for restricting the VMM process using the Linux kernel "Landlock" API 561has been added - this can be used to restrict the files (and the read/write 562permissions) that the VMM process can access. This adds another layer of 563security alongside the existing sycall filters (`seccomp`) - this can be 564enabled with `--landlock` and [fully documentated](docs/landlock.md). (#5170) 565 566### Notable Performance Improvements 567 568* Reduced heap allocations in `virtio-net` via the use of a cache of `Iovec` 569 structures (#6636) 570* Notification suppression ("`EVENT_IDX`") support has been added to 571 `virtio-block` giving a 60% improvement in single queue block throughput and 572 IOPs performance (#6580) 573* Correct size used for `status` field in `virtio-block` state (#6586) 574 575### Notable Bug Fixes 576 577* Avoid panic on out-of-bounds PCI MSI-X access (#6657) 578* Fix undefined behaviour on AArch64 leading to wrong optimisation on KVM API 579 access (#6647) 580* Rust v1.80.0 added use of `fcntl` syscall on debug assertions so this is now 581 included in the virtio-device seccomp filters for tests that use this (#6648) 582* Short reads are now handled correctly in the `virtio-vsock` device (#6621) 583* Fix undefined behaviour on TTY ioctl leading to wrong optimisation (#6568) 584 585### Contributors 586 587Many thanks to everyone who has contributed to our release: 588 589* Alyssa Ross <hi@alyssa.is> 590* Bo Chen <chen.bo@intel.com> 591* Changyuan Lyu <changyuanl@google.com> 592* Jinank Jain <jinankjain@microsoft.com> 593* Julian Stecklina <julian.stecklina@cyberus-technology.de> 594* Muminul Islam <muislam@microsoft.com> 595* Nuno Das Neves <nudasnev@microsoft.com> 596* Praveen K Paladugu <prapal@linux.microsoft.com> 597* Rob Bradford <rbradford@rivosinc.com> 598* Songqian Li <sionli@tencent.com> 599* Wei Liu <liuwe@microsoft.com> 600* Yuanchu Xie <yuanchu@google.com> 601* ihciah <ihciah@gmail.com> 602* wuxinyue <wuxinyue.wxy@antgroup.com> 603 604# v40.0 605 606This release has been tracked in [v40.0 607group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v40.0%22) 608of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 609 610### Support for Restoring File Descriptor Backed Network Devices 611 612It is now possible to pass file descriptors over the HTTP API (and using 613`ch-remote`) when restoring to update the file descriptors for network devices. 614This enables snapshot & restore functionality for guests using `macvtap` or 615other file descriptor backed network devices. (#6286) 616 617### Notable Bug Fixes 618 619* Default values have been removed from required fields in the OpenAPI metadata 620 (#6495) 621* The help syntax of `ch-remote remove-device` has been improved (#6456) 622* A double close of file descriptors has been fixed when using `--serial` 623 (#6486) 624* To prevent loops a limit on the nesting level for QCOW2 backing files has 625 been introduced (#6482) 626* Boot time performance has been improved with multiple cores by avoiding 627 `cpuid` instructions and by seeding the in kernel file descriptor table 628 (#6498, #6478) 629* L1 cache details are more likely to be propagated into the guest (#6523) 630* The default topology for guests now uses multiple cores rather than sockets 631 (#6504) 632 633### Contributors 634 635Many thanks to everyone who has contributed to our release: 636 637* Alexandru Matei <alexandru.matei@uipath.com> 638* Bo Chen <chen.bo@intel.com> 639* Jinank Jain <jinankjain@microsoft.com> 640* Josh Soref <2119212+jsoref@users.noreply.github.com> 641* Muminul Islam <muislam@microsoft.com> 642* Nuno Das Neves <nudasnev@microsoft.com> 643* Omer Faruk Bayram <omer.faruk@sartura.hr> 644* Praveen K Paladugu <prapal@linux.microsoft.com> 645* Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> 646* Rob Bradford <rbradford@rivosinc.com> 647* SamrutGadde <samrut.gadde@gmail.com> 648* Sean Banko <sbanko@crusoe.ai> 649* Songqian Li <sionli@tencent.com> 650* Wei Liu <liuwe@microsoft.com> 651* Yi Wang <foxywang@tencent.com> 652* Yu Li <liyu.yukiteru@bytedance.com> 653* 654 655# v39.0 656 657This release has been tracked in [v39.0 658group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v39.0%22) 659of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 660 661### Variable Sizing of PCI Apertures for Segments 662 663It is now possible to use `--pci-segment` to adjust the aperture size that 664devices 32-bit and 64-bit PCI device BARs will be allocated from. Previously 665the address space was equally distributed across all the segments which may 666leave insufficient space for devices that require a large 32-bit space. With 667this change the weighting per segment can be adjusted. (#6387) 668 669### Direct Booting with bzImages 670 671Support for directly booting Linux from bzImages has been added.(#6200) 672 673### Support for NVIDIA GPUDirect P2P Support 674 675The `x_nv_gpudirect_clique` option was added to `--device` to allow the 676configuration of device P2P support with NVIDIA GPUs. (#6235) 677 678### Guest NMI Injection Support 679 680A new API endpoint and `ch-remote` option added for injecting an NMI into the 681guest. (#6047) 682 683### Notable Bug Fixes 684 685* Workaround for kernel bug affecting guest IRQ masking on AMD (#6353) 686* Correctly cleanup `sigwinch_listener` process (#6208) 687* Graceful shutdown of HTTP API thread (#6248, #6247) 688* Fix `queue_affinity` option in OpenAPI metadata (#6268) 689* Fix documentation to indicate only stream mode is supported by `virtio-vsock` 690 (#6306) 691* Fix `virtio-fs` tag validation (#6358, #6359) 692* Add missing `pvpanic` device to OpenAPI metadata (#6372) 693* Fixes for nested virtualization with VFIO devices (#6110, #6298, #6297, 694 #6319) 695* Fix for backing file for `virtio-mem` regions with snapshot/restore (#6337, 696 #6338) 697* Explicitly mark FDs used for network devices as invalid across 698 snapshot/restore (#6332, #6286) 699* Improve `event-monitor` events around reboot (#6277, #6274) 700* Fix potential deadlock around paused devices during live migration (#6293) 701* Fix panic when running `ch-remote` with no subcommand (#6230) 702* Fix hotplug of `virtio` devices after snapshot/restore and live migration 703 (#6326, #6265) 704 705### Contributors 706 707Many thanks to everyone who has contributed to our release: 708 709* Alexandru Matei <alexandru.matei@uipath.com> 710* Andrew Carp <acarp@crusoeenergy.com> 711* Bo Chen <chen.bo@intel.com> 712* Bouke van der Bijl <i@bou.ke> 713* Chris Webb <chris@arachsys.com> 714* Jinank Jain <jinankjain@microsoft.com> 715* Lucas Jacques <contact@lucasjacques.com> 716* Muminul Islam <muislam@microsoft.com> 717* Nuno Das Neves <nudasnev@microsoft.com> 718* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 719* Rob Bradford <rbradford@rivosinc.com> 720* Ruslan Mstoi <ruslan.mstoi@intel.com> 721* Stefan Nuernberger <stefan.nuernberger@cyberus-technology.de> 722* Thomas Barrett <tbarrett@crusoeenergy.com> 723* Wei Liu <liuwe@microsoft.com> 724* Yi Wang <foxywang@tencent.com> 725 726# v38.0 727 728This release has been tracked in [v38.0 729group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v38.0%22) 730of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 731 732### Group Rate Limiter on Block Devices 733 734Users now can throttle a group of block devices with the new 735`--rate-limiter-group` option. Details can be found from the [I/O 736Throttling documentation](docs/io_throttling.md) 737 738### CPU Pinning Support for Block Device Worker Thread 739 740Users now have the option to pin virt-queue threads for block devices 741to specific host cpus. 742 743### Optimized Boot Time with Parallel Memory Prefault 744 745The boot time with `prefault` option enabled is optimized via parallel 746memory prefault. 747 748### New 'debug-console' Device 749 750A 'debug-console' device is added to provide a user-configurable debug 751port for logging guest information. Details can be found from the [Debug 752IO Ports documentation](docs/debug-port.md). 753 754### Improved VFIO Device Support 755 756All non-emulated MMIO regions of VFIO devices are now mapped to the VFIO 757container, allowing PCIe P2P between all VFIO devices on the same 758VM. This is required for a wide variety of multi-GPU workloads involving 759GPUDirect P2P (DMA between two GPUs), GPUDirect RDMA (DMA between a GPU 760and an IB device). 761 762### Extended CPU Affinity Support 763 764Users now can set the vcpu affinity to a host CPU with index larger 765than 255. 766 767### Notable Bug Fixes 768 769* Enable HTT flag to avoid crashing cpu topology enumeration software 770such as hwloc in the guest (#6146) 771* Fix several security advisories from dependencies (#6134, #6141) 772* Handle non-power-of-two CPU topology properly (#6062) 773* Various bug fixes around `virtio-vsock`(#6080, #6091, #6095) 774* Enable nested virtualization on AMD if supported (#6106) 775* Align VFIO devices PCI BARs naturally (#6196) 776 777### Contributors 778 779Many thanks to everyone who has contributed to our release: 780* Alyssa Ross <hi@alyssa.is> 781* Bo Chen <chen.bo@intel.com> 782* Daniel Farina <daniel@ubicloud.com> 783* Jinank Jain <jinankjain@microsoft.com> 784* Muminul Islam <muislam@microsoft.com> 785* Peteris Rudzusiks <rye@stripe.com> 786* Philipp Schuster <philipp.schuster@cyberus-technology.de> 787* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 788* Rob Bradford <rbradford@rivosinc.com> 789* Ruslan Mstoi <ruslan.mstoi@intel.com> 790* Sean Banko <sbanko@crusoeenergy.com> 791* Thomas Barrett <tbarrett@crusoeenergy.com> 792* Wei Liu <liuwe@microsoft.com> 793* Yi Wang <foxywang@tencent.com> 794* acarp <acarp@crusoeenergy.com> 795 796# v37.0 797 798This release has been tracked in [v37.0 799group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v37.0%22) 800of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 801 802### Long Term Support (LTS) Release 803 804This release is a LTS release. Point releases for bug fixes will be made 805for the next 18 months; live migration and live upgrade will be 806supported between the point releases of the LTS. 807 808### Multiple PCI segments Support for 32-bit VFIO devices 809 810Now VFIO devices with 32-bit memory BARs can be attached to non-zero PCI 811segments on the guest, allowing users to have more 32-bit devices and 812assign such devices to appropriate NUMA nodes for better performance. 813 814### Configurable Named TAP Devices 815 816Named TAP devices now accepts IP configuration from users, such as IP 817and MAC address, as long as the named TAP device is created by Cloud 818Hypervisor (e.g. not existing TAP devices). 819 820### TTY Output from Both Serial Device and Virtio Console 821 822Now legacy serial device and virtio console can be set as TTY mode as 823the same time. This allows users to capture early boot logs with the 824legacy serial device without losing performance benefits of using 825virtio-console, when appropriate kernel configuration is used (such as 826using kernel command-line `console=hvc0 earlyprintk=ttyS0` on x86). 827 828### Faster VM Restoration from Snapshots 829 830The speed of VM restoration from snapshots is improved with a better 831implementation of deserializing JSON files. 832 833### Notable Bug Fixes 834 835* Fix aio backend behavior for block devices when writeback cache 836 disabled (#5930) 837* Fix PvPanic device PCI BAR alignment (#5956) 838* Bug fix to OpenAPI specification file (#5967) 839* Error out early for live migration when TDX is enabled (#6025) 840 841### Contributors 842 843Many thanks to everyone who has contributed to our release: 844 845* Bo Chen <chen.bo@intel.com> 846* Jinank Jain <jinankjain@microsoft.com> 847* Markus Sütter <markus.suetter@secunet.com> 848* Michael Zhao <michael.zhao@arm.com> 849* Muminul Islam <muislam@microsoft.com> 850* Rob Bradford <rbradford@rivosinc.com> 851* Rui Chang <rui.chang@arm.com> 852* Ruslan Mstoi <ruslan.mstoi@intel.com> 853* Thomas Barrett <tbarrett@crusoeenergy.com> 854* Wei Liu <liuwe@microsoft.com> 855* Yi Wang <foxywang@tencent.com> 856* Yong He <alexyonghe@tencent.com> 857 858# v36.0 859 860This release has been tracked in [v36.0 861group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v36.0%22) 862of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 863 864### Command Line Changes 865 866We switched back to use the `clap` crate to create our command line, 867since the `argh` crate is barely maintained. There were several syntax 868changes: 869 870* All `--option value` commands now are `--option=value`. 871* The `--disk DISK1 --disk DISK2` command now is `--disk DISK1 DISK2`. 872* The `-v -v -v`command now is `-vvv`. 873 874Note: the released binary size increased around 0.3M due to this change. 875 876### Enabled Features Reported via API Endpoint and CLI 877 878Now the enabled (Cargo) features of the running Cloud Hypervisor 879instance can be queried via API endpoint (`/vmm.ping`) and CLI 880(`--version -v`). 881 882### NUMA Support for PCI segments 883 884The `--numa` command is augmented with a new option `pci_segment=`, so 885that users can define the relationship between PCI segments and NUMA 886nodes. Examples can be found from the [memory documentation](docs/memory.md) 887 888### CPU Topology Support on AMD Platforms 889 890Now the CPU topology on x86_64 platforms supports multiple vendors. 891 892### Unix Socket Backend for Serial Port 893 894The `--serial` command is augmented with a new option `socket=`, allowing 895users to access the serial port using a Unix socket. 896 897### AIO Backend for Block Devices 898 899An AIO backend is added for `virtio-block` devices to improve block 900device performance when the `io_uring` feature is not supported by the 901host Operating System. 902 903### Documentation Improvements 904 905* New [documentation](docs/coverage.md) for collecting coverage data 906* Various typo fixes 907 908### Notable Bug Fixes 909 910* Fix a deadlock when TDX is enabled (#5845) 911* Only advertise AMX feature bits to guest when the AMX cpu feature is 912 enabled (#5834) 913* Correct default value for vCPU topology on AArch64 (#5893) 914 915### Contributors 916 917Many thanks to everyone who has contributed to our release: 918 919* Anatol Belski <anbelski@linux.microsoft.com> 920* Bo Chen <chen.bo@intel.com> 921* Dario Nieuwenhuis <dirbaio@dirbaio.net> 922* Jianyong Wu <jianyong.wu@arm.com> 923* Jinank Jain <jinankjain@microsoft.com> 924* Muminul Islam <muislam@microsoft.com> 925* Praveen K Paladugu <prapal@linux.microsoft.com> 926* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 927* Rob Bradford <rbradford@rivosinc.com> 928* Thomas Barrett <tbarrett@crusoeenergy.com> 929* Wei Liu <liuwe@microsoft.com> 930* Yi Wang <foxywang@tencent.com> 931* dom.song <dom.song@amperecomputing.com> 932 933# v35.0 934 935This release has been tracked in [v35.0 936group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v35.0%22) 937of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 938 939### `virtio-vsock` Support for Linux Guest Kernel v6.3+ 940 941Since kernel v6.3, a vsock packet can be included in a single descriptor, 942instead of being split over two descriptors. Our `virtio-vsock` implementation 943now support both situations. 944 945### User Specified Serial Number for `virtio-block` 946 947A new option `serial` is added to the `--block` command that allows users to 948specify a serial number for block devices which will be guest visible. 949 950### vCPU TSC Frequency Included in Migration State 951 952This ensures migration works correctly between hosts that have different TSC 953frequencies if the guest is running with TSC as the source of timekeeping. 954 955### Notable Bug Fixes 956 957* Disallow concurrent CPU resizing (#5668) 958* Handle APIC EOI message for MSHV (#5681) 959* Use wrapping add for memory offset from instruction emulator (#5719) 960* Add global spell check via the 'typos' GitHub action (#5720) 961* Ensure probing reads are block size aligned (#5727) 962* Multiple bug fixes around the latency counter for block devices 963 (#5712, #5750, #5762, #5763) 964* Replace unsound `static mut` with `once_cell` (#5772) 965 966### Contributors 967 968Many thanks to everyone who has contributed to our release: 969 970* Alyssa Ross <hi@alyssa.is> 971* Anatol Belski <anbelski@linux.microsoft.com> 972* Bo Chen <chen.bo@intel.com> 973* Christian Blichmann <cblichmann@google.com> 974* Jianyong Wu <jianyong.wu@arm.com> 975* Jinank Jain <jinankjain@microsoft.com> 976* Julian Stecklina <julian.stecklina@cyberus-technology.de> 977* Omer Faruk Bayram <omer.faruk@sartura.hr> 978* Philipp Schuster <philipp.schuster@cyberus-technology.de> 979* Rob Bradford <rbradford@rivosinc.com> 980* Ruslan Mstoi <ruslan.mstoi@intel.com> 981* Thomas Barrett <tbarrett@crusoeenergy.com> 982* Wei Liu <liuwe@microsoft.com> 983* Yi Wang <foxywang@tencent.com> 984* zhongbingnan <zhongbingnan@bytedance.com> 985 986# v34.0 987 988This release has been tracked in [v34.0 989group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v34.0%22) 990of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 991 992### Paravirtualised Panic Device Support 993 994A new device has been added that can communicate when the guest kernel has 995panicked and share those details with the VMM. This is controlled with a new 996`--pvpanic` command line option and JSON API change equivalent. (#5526) 997 998### Improvements to VM Core Dump 999 1000Requesting to dump the guest memory as core dump will now transparently pause 1001the VM if required; returning to the original state after. (#5604) 1002 1003### QCOW2 Support for Backing Files 1004 1005The support for QCOW2 files has been enhanced to include support for using 1006backing files. (#5573) 1007 1008### Minimum Host Kernel Bump 1009 1010The minimum supported host kernel is now 5.13 in order to incorporate a bug fix 1011for `KVM_FEATURE_ASYNC_PF_INT` functionality. (#5626) 1012 1013### Notable Bug Fixes 1014 1015* The x86 emulator is only compiled in if MSHV is compiled in (the kernel 1016 carries out this job with KVM) (#5561). 1017* A regression has been fixed in VFIO support for devices that use MSI rather 1018 than MSI-X (#5658). 1019* When triggering a VM shutdown or reset via I/O the vCPU thread will be 1020 blocked until that asynchronous event has been received (#5645). 1021* Pausing a VM is now a synchronous action with the request only completing 1022 when all vCPUs are paused (#5611). 1023* Event monitor support now correctly supports concurrent access (#5633). 1024* Bug fixes for the QCOW2 file support (#5573). 1025 1026### Contributors 1027 1028Many thanks to everyone who has contributed to our release: 1029 1030* Alyssa Ross <hi@alyssa.is> 1031* Anatol Belski <anbelski@linux.microsoft.com> 1032* Bo Chen <chen.bo@intel.com> 1033* Changyuan Lyu <changyuanl@google.com> 1034* Christian Blichmann <cblichmann@google.com> 1035* Manish Goregaokar <manishsmail@gmail.com> 1036* Omer Faruk Bayram <omer.faruk@sartura.hr> 1037* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1038* Praveen K Paladugu <prapal@linux.microsoft.com> 1039* Rob Bradford <rbradford@rivosinc.com> 1040* Ruslan Mstoi <ruslan.mstoi@intel.com> 1041* Yi Wang <foxywang@tencent.com> 1042* Yong He <alexyonghe@tencent.com> 1043* Yu Li <liyu.yukiteru@bytedance.com> 1044* dom.song <dom.song@amperecomputing.com> 1045 1046# v33.0 1047 1048This release has been tracked in [v33.0 1049group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v33.0%22) 1050of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1051 1052### D-Bus based API 1053 1054A D-Bus based API has been added as an alternative to the existing REST 1055API. This feature is gated by the `dbus_api` feature. Details can be 1056found in the [API documentation](docs/api.md). 1057 1058### Expose Host CPU Cache Details for AArch64 1059 1060Now the CPU cache information on the host is properly exposed to the 1061guest on AArch64. 1062 1063### Notable Bug Fixes 1064 1065* Report errors explicitly to users when VM failed to boot (#5453) 1066* Fix VFIO on platforms with non-4k page size (#5450, #5469) 1067* Fix TDX initialization (#5454) 1068* Ensure all guest memory regions are page-size aligned (#5496) 1069* Fix seccomp filter lists related to virtio-console, serial and pty 1070 (#5506, #5524) 1071* Populate APIC ID properly (#5512) 1072* Ignore and warn TAP FDs in more situations (#5522) 1073 1074### Contributors 1075 1076Many thanks to everyone who has contributed to our release: 1077 1078* Alyssa Ross <hi@alyssa.is> 1079* Anatol Belski <anbelski@linux.microsoft.com> 1080* Bo Chen <chen.bo@intel.com> 1081* Jianyong Wu <jianyong.wu@arm.com> 1082* Omer Faruk Bayram <omer.faruk@sartura.hr> 1083* Rafael Mendonca <rafaelmendsr@gmail.com> 1084* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1085* Rob Bradford <rbradford@rivosinc.com> 1086* Ruslan Mstoi <ruslan.mstoi@intel.com> 1087* Yu Li <liyu.yukiteru@bytedance.com> 1088* zhongbingnan <zhongbingnan@bytedance.com> 1089 1090# v32.0 1091 1092This release has been tracked in [v32.0 1093group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v32.0%22) 1094of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1095 1096### Increased PCI Segment Limit 1097 1098The maximum number of PCI segments that can be used is now 96 (up from 16). 1099 1100### API Changes 1101 1102* The VmmPingResponse now includes the PID as well as the build details. 1103 (#5348) 1104 1105### Notable Bug Fixes 1106 1107* Ignore and warn TAP FDs sent via the HTTP request body (#5350) 1108* Properly preserve and close valid FDs for TAP devices (#5373) 1109* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360) 1110* Only touch the tty flags if it's being used (#5343) 1111* Fix seccomp filter lists for vhost-user devices (#5361) 1112* The number of vCPUs is capped at the hypervisor maximum (#5357) 1113* Fixes for TTY reset (#5414) 1114* CPU topology fixes on MSHV (#5325) 1115* Seccomp fixes for older distributions (#5397) 1116 1117### Contributors 1118 1119Many thanks to everyone who has contributed to our release: 1120 1121* Alyssa Ross <hi@alyssa.is> 1122* Anatol Belski <anbelski@linux.microsoft.com> 1123* Bo Chen <chen.bo@intel.com> 1124* Hao Xu <howeyxu@tencent.com> 1125* Muminul Islam <muislam@microsoft.com> 1126* Omer Faruk Bayram <omer.faruk@sartura.hr> 1127* Rafael Mendonca <rafaelmendsr@gmail.com> 1128* Rob Bradford <rbradford@rivosinc.com> 1129* Ruslan Mstoi <ruslan.mstoi@intel.com> 1130* Smit Gardhariya <gardhariya.smit@gmail.com> 1131* Wei Liu <liuwe@microsoft.com> 1132 1133# v31.1 1134 1135This is a bug fix release. The following issues have been addressed: 1136 1137* Ignore and warn TAP FDs sent via the HTTP request body (#5350) 1138* Properly preserve and close valid FDs for TAP devices (#5373) 1139* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360) 1140* Only touch the tty flags if it's being used (#5343) 1141* Fix seccomp filter lists for vhost-user devices (#5361) 1142 1143# v31.0 1144 1145This release has been tracked in [v31.0 1146group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v31.0%22) 1147of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1148 1149### Update to Latest `acpi_tables` 1150 1151Adapted to the latest [acpi_tables](https://github.com/rust-vmm/acpi_tables). 1152There has been significant API changes in the crate. 1153 1154### Update Reference Kernel to 6.2 1155 1156Updated the recommended guest kernel version from 6.1.6 to 6.2. 1157 1158### Improvements on Console `SIGWINCH` Handler 1159 1160A separate thread had been created to capture the `SIGWINCH` signal and resize 1161the guest console. Now the thread is skipped if the console is not resizable. 1162 1163Two completely different code paths existed for handling console resizing, one 1164for `tty` and the other for `pty`. That makes the understanding of the console 1165handling code unnecessarily complicated. Now the code paths are unified. Both 1166`tty` and `pty` are supported in single `SIGWINCH` handler. And the new handler 1167can works with kernel versions earlier than v5.5. 1168 1169### Remove Directory Support from `MemoryZoneConfig::file` 1170 1171Setting a directory to `MemoryZoneConfig::file` is no longer supported. 1172 1173Before this change, user can set a directory to `file` of the `--memory-zone` 1174option. In that case, a temporary file will be created as the backing file for 1175the `mmap(2)` operation. This functionality has been unnecessary since we had 1176the native support for hugepages and allocating anonymous shared memory. 1177 1178### Documentation Improvements 1179 1180* Various improvements in API document 1181* Improvements in Doc comments 1182* Updated Slack channel information in README 1183 1184### Notable Bug Fixes 1185 1186* Fixed the offset setting while removing the entire mapping of `vhost-user` FS 1187 client. 1188* Fixed the `ShutdownVmm` and `Shutdown` commands to call the correct API 1189 endpoint. 1190 1191### Contributors 1192 1193Many thanks to everyone who has contributed to our release: 1194 1195* Alyssa Ross <hi@alyssa.is> 1196* Bo Chen <chen.bo@intel.com> 1197* Daniel Farina <daniel@fdr.io> 1198* Dom <peng6662001@163.com> 1199* Hao Xu <howeyxu@tencent.com> 1200* Muminul Islam <muislam@microsoft.com> 1201* Omer Faruk Bayram <omer.faruk@sartura.hr> 1202* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1203* Rob Bradford <rbradford@rivosinc.com> 1204* Ruslan Mstoi <ruslan.mstoi@intel.com> 1205* Smit Gardhariya <gardhariya.smit@gmail.com> 1206* Yang <ailin.yang@intel.com> 1207* Yong He <alexyonghe@tencent.com> 1208 1209# v30.0 1210 1211This release has been tracked in [v30.0 1212group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v30.0%22) 1213of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1214 1215### Command Line Changes for Reduced Binary Size 1216 1217The `clap` crate was replaced by the `argh` crate to create our command 1218line, which reduced our release binary size from 3.6MB to 3.3MB. There 1219were several syntax changes: 1220 1221* All `--option=value` commands now are `--option value`. 1222* The `--disk DISK1 DISK2` command now is `--disk DISK1 --disk DISK2`. 1223* The `-vvv` command now is `-v -v -v` 1224 1225### Basic vfio-user Server Support 1226 1227Our `vfio-user` crate is extended to provide basic server side support 1228with an example of gpio vfio-user device. This crate now is moved to [its 1229own repository](https://github.com/rust-vmm/vfio-user) under the 1230`rust-vmm` organization. 1231 1232### Heap Profiling Support 1233 1234A new building target is added for profiling purposes with examples of 1235heap profiling using `dhat` gated by the `dhat-heap` feature. 1236 1237### Documentation Improvements 1238 1239The documentation on Intel TDX is expanded with details of the building 1240and using [TD-Shim](https://github.com/confidential-containers/td-shim), 1241references to [TDX Linux](https://github.com/intel/tdx-linux), and 1242version information of guest/host kernel/TDVF/TDShim being tested. Also, 1243a new 'heap profiling' documentation is added with improvements on the 1244existing 'profiling' documentation. 1245 1246### Notable Bug Fixes 1247 1248* Close FDs for TAP devices that are provided to VM (#5199, #5206) 1249* Set vcpu thread status properly and signal `exit_evt` upon thread exit (#5211) 1250* Populate CPUID leaf 0x4000_0010 (TSC frequency) (#5178, #5179) 1251* Inform the TPM guest driver upon failed TPM requests on the host (#5151) 1252* Bug fix to OpenAPI specification file (#5186) 1253 1254### Contributors 1255 1256Many thanks to everyone who has contributed to our release: 1257 1258* Anatol Belski <anbelski@linux.microsoft.com> 1259* Anirudh Rayabharam <anrayabh@linux.microsoft.com> 1260* Bo Chen <chen.bo@intel.com> 1261* Jinank Jain <jinankjain@microsoft.com> 1262* Kaihang Zhang <kaihang.zhang@smartx.com> 1263* Michael Zhao <michael.zhao@arm.com> 1264* Muminul Islam <muislam@microsoft.com> 1265* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1266* Praveen K Paladugu <prapal@linux.microsoft.com> 1267* Ravi kumar Veeramally <ravikumar.veeramally@intel.com> 1268* Rob Bradford <robert.bradford@intel.com> 1269* Ruslan Mstoi <ruslan.mstoi@intel.com> 1270* Sebastien Boeuf <sebastien.boeuf@intel.com> 1271* Wei Liu <liuwe@microsoft.com> 1272* Yong He <alexyonghe@tencent.com> 1273* Yu Li <liyu.yukiteru@bytedance.com> 1274 1275# v28.2 1276This is a bug fix release. The following issues have been addressed: 1277 1278* Fix QCOW2 refcount table size (#5034) 1279* Fix unpause support on MSHV in dual binary (#5037) 1280* Threads inside `virtio` devices are now shutdown on reboot (#5095) 1281 1282# v29.0 1283 1284This release has been tracked in [v29.0 1285group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v29.0%22) 1286of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1287 1288### Release Binary Supports Both MSHV and KVM 1289 1290On `x86-64` the binary included in releases supports both the KVM and MSHV 1291hypervisor with runtime detection to identify the correct hypervisor to use. 1292 1293### Snapshot/Restore and Live Migration Improvements 1294 1295Improvements have been made to the data structures used for both live migration 1296and snapshot/restore. Unfortunately this has broken compatibility with older 1297versions (support for migrating between major versions is not yet officially 1298supported but has worked for some versions.) 1299 1300### Heap Allocation Improvements 1301 1302Improvements have been made to the volume of heap allocations when running with 1303`virtio-block` devices along with a reduction in the peak heap size. 1304 1305### `ch-remote` Improvements 1306 1307Support for "pinging" the VMM and shutting the VMM down have been added to 1308`ch-remote`. 1309 1310### `AArch64` Documentation Integration 1311 1312The documentation for `AArch64` support has been integrated into the main 1313README. 1314 1315### `virtio-block` Counters Enhancement 1316 1317The counters for the `virtio-block` device has extended to include min/mean/max 1318latency counters. 1319 1320### TCP Offload Control 1321 1322The `virtio-net` device has gained support for controlling the enabling of 1323checksum and offloading. This allows the device to be used in environments 1324where the hardware lacks support for the offloading. 1325 1326### Notable Bug Fixes 1327 1328* Update dependencies including a version of `linux-loader` that addresses an 1329 infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6)) 1330* Fix bugs related to `virtio-net` including an integer overflow issue 1331 (#4924, #4949) 1332* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920) 1333* Memory mapped into the guest is now marked as non-dumpable which prevents large core files (#5016) 1334* Fix QCOW2 refcount table size (#5034) 1335* Fix unpause support on MSHV in dual binary (#5037) 1336* Threads inside `virtio` devices are now shutdown on reboot (#5095) 1337 1338### Removals 1339 1340No functionality has been removed in this release. 1341 1342### Deprecations 1343 1344* Support for specifying a directory with `MemoryZoneConfig::file` or 1345 `MemoryConfig::file` has been deprecated. This was originally used for 1346 supporting huge pages or shared memory backing which is now natively supported 1347 (#5085) 1348 1349### Contributors 1350 1351Many thanks to everyone who has contributed to our release: 1352 1353* Bo Chen <chen.bo@intel.com> 1354* Claudio Fontana <claudio.fontana@gmail.com> 1355* Hao Xu <howeyxu@tencent.com> 1356* Henry Wang <Henry.Wang@arm.com> 1357* Jinank Jain <jinankjain@microsoft.com> 1358* Michael Zhao <michael.zhao@arm.com> 1359* Muminul Islam <muislam@microsoft.com> 1360* Philipp Schuster <philipp.schuster@cyberus-technology.de> 1361* Rob Bradford <robert.bradford@intel.com> 1362* Sebastien Boeuf <sebastien.boeuf@intel.com> 1363* Shuaiyi Zhang <zhangsy28@lenovo.com> 1364* Wei Liu <liuwe@microsoft.com> 1365* Yong He <alexyonghe@tencent.com> 1366* Yuji Hagiwara <yuuzi41@gmail.com> 1367 1368 1369# v28.1 1370This is a bug fix release. The following issues have been addressed: 1371 1372* Update dependencies including a version of `linux-loader` that 1373addresses an infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6)) 1374* Fix bugs related to `virtio-net` including an integer overflow issue 1375 (#4924, #4949) 1376* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920) 1377* Improve README and documentation 1378 1379# v28.0 1380 1381This release has been tracked in [v28.0 1382group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v28.0%22) 1383of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1384 1385### Community Engagement (Reminder) 1386 1387Just a reminder that we have a new mailing list to support broader community 1388discussions. Please consider 1389[subscribing](https://lists.cloudhypervisor.org/g/dev/). We plan to use 1390this to announce a regular meeting for those interested in talking about Cloud 1391Hypervisor development. 1392 1393### Long Term Support (LTS) Release 1394 1395This is the first version of Cloud Hypervisor to be released under the LTS 1396release process. Point releases for bug fixes will be made for the next 18 1397months; live migration and live upgrade will be supported between the point 1398releases of the LTS. 1399 1400### Virtualised TPM Support 1401 1402Support for adding an emulated CRB TPM has been added. This has its own [TPM 1403documentation](docs/tpm.md). 1404 1405### Transparent Huge Page Support 1406 1407By default, but controllable through `--memory thp=off` if it possible to back 1408the guest memory with Transparent Huge Pages (no file backing/`shared=off`) 1409then this will be used resulting in improved boot performance. 1410 1411### README Quick Start Improved 1412 1413The README has been refreshed with the quick start guide updated to reflect the 1414different firmware options and to recommend the use of pre-built binaries. 1415 1416### Notable Bug Fixes 1417 1418* Inappropriate Copy-on-Write of pinned pages (e.g. VFIO) leading to higher 1419 memory consumption (#4835) 1420* Multiple `virtio` device bug fixes found through fuzzing (#4859, #4799) 1421* Large QCOW files (> 4TiB) are now supported (#4767) 1422* Support for > 31 vCPUS on aarch64 (#4863) 1423* Multiple fixes to OpenAPI specification file (#4720, #4811) 1424* Programming of the MSI-X table has been optimised leading to faster boot on 1425 newer Linux kernels (#4744) 1426* Error on reboot from race to close TAP devices (#4871) 1427* Non-spec compliant virtio-block read-only support (#4888) 1428 1429### Removals 1430 1431The following functionality has been removed: 1432 1433* Support for non-PVH firmware booting has been removed (#4511) 1434* I/O ports used for older versions of firmware have been removed (#3926) 1435* Deprecated API options for kernel/cmdline/initramfs have been removed (#4737) 1436 1437### Contributors 1438 1439Many thanks to everyone who has contributed to our release: 1440 1441* Anatol Belski <anbelski@linux.microsoft.com> 1442* Bo Chen <chen.bo@intel.com> 1443* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1444* Jianyong Wu <jianyong.wu@arm.com> 1445* Jinank Jain <jinankjain@microsoft.com> 1446* Jinrong Liang <cloudliang@tencent.com> 1447* lv.mengzhao <lv.mengzhao@zte.com.cn> 1448* Michael Zhao <michael.zhao@arm.com> 1449* Muminul Islam <muislam@microsoft.com> 1450* Praveen K Paladugu <prapal@linux.microsoft.com> 1451* Rob Bradford <robert.bradford@intel.com> 1452* Sebastien Boeuf <sebastien.boeuf@intel.com> 1453* Wei Liu <liuwe@microsoft.com> 1454 1455# v27.0 1456 1457This release has been tracked in [v27.0 1458group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v27.0%22) 1459of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/). 1460 1461### Community Engagement 1462 1463A new mailing list has been created to support broader community discussions. 1464Please consider [subscribing](https://lists.cloudhypervisor.org/g/dev/); an 1465announcement of a regular meeting will be announced via this list shortly. 1466 1467### Prebuilt Packages 1468 1469Prebuilt packages are now available. Please see [this 1470document](https://github.com/cloud-hypervisor/obs-packaging/blob/main/README.md) 1471on how to install. These packages also include packages for the different 1472firmware options available. 1473 1474### Network Device MTU Exposed to Guest 1475 1476The MTU for the TAP device associated with a `virtio-net` device is now exposed 1477to the guest. If the user provides a MTU with `--net mtu=..` then that MTU is 1478applied to created TAP interfaces. This functionality is also exposed for 1479`vhost-user-net` devices including those created with the reference backend 1480(#4658, #4676.) 1481 1482### Boot Tracing 1483 1484Support for generating a trace report for the boot time has been added 1485including a script for generating an SVG from that trace (#4659.) 1486 1487### Simplified Build Feature Flags 1488 1489The set of feature flags, for e.g. experimental features, have been simplified: 1490 1491* `msvh` and `kvm` features provide support for those specific hypervisors 1492 (with `kvm` enabled by default), 1493* `tdx` provides support for Intel TDX; and although there is no MSHV support 1494 now it is now possible to compile with the `mshv` feature (#4696,) 1495* `tracing` adds support for boot tracing, 1496* `guest_debug` now covers both support for gdbing a guest (formerly `gdb` 1497 feature) and dumping guest memory. 1498 1499The following feature flags were removed as the functionality was enabled by 1500default: `amx`, `fwdebug`, `cmos` and `common` (#4679, #4632.) 1501 1502### Asynchronous Kernel Loading 1503 1504AArch64 has gained support for loading the guest kernel asynchronously like 1505x86-64. (#4538) 1506 1507### GDB Support for AArch64 1508 1509GDB stub support (accessed through `--gdb` under `guest_debug` feature) is now 1510available on AArch64 as well as as x86-64. 1511 1512### Notable Bug Fixes 1513 1514* This version incorporates a version of `virtio-queue` that addresses an issue 1515 where a rogue guest can potentially DoS the VMM (rust-vmm/vm-virtio#196.) 1516* Improvements around PTY handling for `virtio-console` and serial devices 1517 (#4520, #4533, #4535.) 1518* Improved error handling in virtio devices (#4626, #4605, #4509, #4631, #4697) 1519 1520### Deprecations 1521 1522Deprecated features will be removed in a subsequent release and users should 1523plan to use alternatives. 1524 1525* Booting legacy firmware (compiled without a PVH header) has been deprecated. 1526 All the firmware options (Cloud Hypervisor OVMF and Rust Hypervisor Firmware) 1527 support booting with PVH so support for loading firmware in a legacy mode is no 1528 longer needed. This functionality will be removed in the next release. 1529 1530### Contributors 1531 1532Many thanks to everyone who has contributed to our release: 1533 1534* Anatol Belski <anbelski@linux.microsoft.com> 1535* Bo Chen <chen.bo@intel.com> 1536* James O. D. Hunt <james.o.hunt@intel.com> 1537* Jianyong Wu <jianyong.wu@arm.com> 1538* Markus Napierkowski <markus.napierkowski@cyberus-technology.de> 1539* Michael Zhao <michael.zhao@arm.com> 1540* Nuno Das Neves <nudasnev@microsoft.com> 1541* Rob Bradford <robert.bradford@intel.com> 1542* Sebastien Boeuf <sebastien.boeuf@intel.com> 1543* Smit Gardhariya <sgardhariya@microsoft.com> 1544* Wei Liu <liuwe@microsoft.com> 1545 1546# v26.0 1547 1548### SMBIOS Improvements via `--platform` 1549 1550`--platform` and the appropriate API structure has gained support for supplying 1551OEM strings (primarily used to communicate metadata to systemd in the guest) 1552(#4319, #4446) and support for specifying the UUID (#4389.) 1553 1554### Unified Binary MSHV and KVM Support 1555 1556Support for both the MSHV and KVM hypervisors can be compiled into the same 1557binary with the detection of the hypervisor to use made at runtime. 1558 1559### Notable Bug Fixes 1560 1561* The prefetchable flag is preserved on BARs for VFIO devices (#4353, #4454) 1562* PCI Express capabilities for functionality we do not support are now filtered 1563 out (#4456) 1564* GDB breakpoint support is more reliable (#4354, #4363) 1565* `SIGINT` and `SIGTERM` signals are now handled before the VM has booted 1566 (#4269, #4293) 1567* Multiple API event loop handling bug fixes (#4309, #4362) 1568* Incorrect assumptions in virtio queue numbering were addressed, allowing 1569 the`virtio-fs` driver in OVMF to be used (#4341, #4314) 1570* VHDX file format header fix (#4291) 1571* The same VFIO device cannot be added twice (#4453, #4463) 1572* SMBIOS tables were being incorrectly generated (#4442) 1573 1574### Deprecations 1575 1576Deprecated features will be removed in a subsequent release and users should 1577plan to use alternatives. 1578 1579* The top-level `kernel` and `initramfs` members on the `VmConfig` have been 1580 moved inside a `PayloadConfig` as the `payload` member. The OpenAPI document 1581 has been updated to reflect the change and the old API members continue to 1582 function and are mapped to the new version. The expectation is that these old 1583 versions will be removed in the v28.0 release. 1584 1585### Removals 1586 1587The following functionality has been removed: 1588 1589* The unused `poll_queue` parameter has been removed from `--disk` and 1590 equivalent. This was residual from the removal of the `vhost-user-block` 1591 spawning feature (#4402.) 1592 1593### Contributors 1594 1595Many thanks to everyone who has contributed to our release: 1596 1597* Alyssa Ross <hi@alyssa.is> 1598* Anatol Belski <ab@php.net> 1599* Archana Shinde <archana.m.shinde@intel.com> 1600* Bo Chen <chen.bo@intel.com> 1601* lizhaoxin1 <Lxiaoyouling@163.com> 1602* Maximilian Nitsch <maximilian.nitsch@d3tn.com> 1603* Michael Zhao <michael.zhao@arm.com> 1604* Rob Bradford <robert.bradford@intel.com> 1605* Sebastien Boeuf <sebastien.boeuf@intel.com> 1606* Steven Dake <sdake@lambdal.com> 1607* Wei Liu <liuwe@microsoft.com> 1608 1609# v25.0 1610 1611### `ch-remote` Improvements 1612 1613The `ch-remote` command has gained support for creating the VM from a JSON 1614config and support for booting and deleting the VM from the VMM. 1615 1616### VM "Coredump" Support 1617 1618Under the `guest_debug` feature flag it is now possible to extract the memory 1619of the guest for use in debugging with e.g. the `crash` utility. (#4012) 1620 1621### Notable Bug Fixes 1622 1623* Always restore console mode on exit (#4249, #4248) 1624* Restore vCPUs in numerical order which fixes aarch64 snapshot/restore (#4244) 1625* Don't try and configure `IFF_RUNNING` on TAP devices (#4279) 1626* Propagate configured queue size through to vhost-user backend (#4286) 1627* Always Program vCPU CPUID before running the vCPU to fix running on Linux 1628 5.16 (#4156) 1629* Enable ACPI MADT "Online Capable" flag for hotpluggable vCPUs to fix newer 1630 Linux guest 1631 1632### Removals 1633 1634The following functionality has been removed: 1635 1636* The `mergeable` option from the `virtio-pmem` support has been removed 1637 (#3968) 1638* The `dax` option from the `virtio-fs` support has been removed (#3889) 1639 1640### Contributors 1641 1642Many thanks to everyone who has contributed to our release: 1643 1644* Dylan Bargatze <dbargatz@users.noreply.github.com> 1645* Jinank Jain <jinankjain@microsoft.com> 1646* Michael Zhao <michael.zhao@arm.com> 1647* Rob Bradford <robert.bradford@intel.com> 1648* Sebastien Boeuf <sebastien.boeuf@intel.com> 1649* Wei Liu <liuwe@microsoft.com> 1650* Yi Wang <wang.yi59@zte.com.cn> 1651 1652# v24.0 1653 1654### Bypass Mode for `virtio-iommu` 1655 1656`virtio-iommu` specification describes how a device can be attached by default 1657to a bypass domain. This feature is particularly helpful for booting a VM with 1658guest software which doesn't support `virtio-iommu` but still need to access 1659the device. Now that Cloud Hypervisor supports this feature, it can boot a VM 1660with Rust Hypervisor Firmware or OVMF even if the `virtio-block` device exposing 1661the disk image is placed behind a virtual IOMMU. 1662 1663### Ensure Identifiers Uniqueness 1664 1665Multiple checks have been added to the code to prevent devices with identical 1666identifiers from being created, and therefore avoid unexpected behaviors at boot 1667or whenever a device was hot plugged into the VM. 1668 1669### Sparse Mmap support 1670 1671Sparse mmap support has been added to both VFIO and vfio-user devices. This 1672allows the device regions that are not fully mappable to be partially mapped. 1673And the more a device region can be mapped into the guest address space, the 1674fewer VM exits will be generated when this device is accessed. This directly 1675impacts the performance related to this device. 1676 1677### Expose Platform Serial Number 1678 1679A new `serial_number` option has been added to `--platform`, allowing a user to 1680set a specific serial number for the platform. This number is exposed to the 1681guest through the SMBIOS. 1682 1683### Notable Bug Fixes 1684 1685* Fix loading RAW firmware (#4072) 1686* Reject compressed QCOW images (#4055) 1687* Reject virtio-mem resize if device is not activated (#4003) 1688* Fix potential mmap leaks from VFIO/vfio-user MMIO regions (#4069) 1689* Fix algorithm finding HOB memory resources (#3983) 1690 1691### Notable Improvements 1692 1693* Refactor interrupt handling (#4083) 1694* Load kernel asynchronously (#4022) 1695* Only create ACPI memory manager DSDT when resizable (#4013) 1696 1697### Deprecations 1698 1699Deprecated features will be removed in a subsequent release and users should 1700plan to use alternatives 1701 1702* The `mergeable` option from the `virtio-pmem` support has been deprecated 1703 (#3968) 1704* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 1705 1706### New on the Website 1707 1708A new blog post [Achieving Bare Metal Performance Within a Virtual 1709Machine](https://www.cloudhypervisor.org/blog/achieving-bare-metal-performance-within-a-virtual-machine) 1710has been added to the Cloud Hypervisor website. 1711 1712### Contributors 1713 1714Many thanks to everyone who has contributed to our release: 1715 1716* Anatol Belski <anbelski@linux.microsoft.com> 1717* Bo Chen <chen.bo@intel.com> 1718* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1719* LiHui <andrewli@kubesphere.io> 1720* Maksym Pavlenko <pavlenko.maksym@gmail.com> 1721* Rob Bradford <robert.bradford@intel.com> 1722* Sebastien Boeuf <sebastien.boeuf@intel.com> 1723* Steven Dake <steven.dake@gmail.com> 1724* Vincent Batts <vbatts@hashbangbash.com> 1725* Wei Liu <liuwe@microsoft.com> 1726 1727# v23.1 1728 1729This is a bug fix release. The following issues have been addressed: 1730 1731* Add some missing seccomp rules 1732* Remove `virtio-fs` filesystem entries from config on removal 1733* Do not delete API socket on API server start (#4026) 1734* Reject `virtio-mem` resize if the guest doesn't activate the device 1735* Fix OpenAPI naming of I/O throttling knobs 1736 1737# v23.0 1738 1739### vDPA Support 1740 1741A vDPA device has a datapath that complies with the virtio specification but 1742with a vendor specific control path. The addition of `--vdpa` and the REST API 1743equivalent allows the use of these devices with Cloud Hypervisor. 1744 1745### Updated OS Support list 1746 1747The list of officially supported and tested OS versions has been updated to 1748include Ubuntu "jammy" 22.04 and EOLed versions removed. 1749 1750### `AArch64` Memory Map Improvements 1751 1752The memory map when running on `AArch64` has been improved for the handling of 1753the UEFI region which means that the booted guest OS now has full access to its 1754allocated RAM. (#3938) 1755 1756### `AMX` Support 1757 1758Under a compile time gate of `amx` it is possible compile in support for the 1759`AMX` instruction set extension for guest use. This also requires runtime 1760enabling with `--cpu features=amx`. 1761 1762### Notable Bug Fixes 1763 1764* Generate error when incorrect HTTP method used for some API endpoints (#3887) 1765* CMOS based reset is now available to support rebooting on "jammy" (#3914) 1766* ACPI tables are not produced for memory hotplug when running with 1767 `virtio-mem` (#3883) 1768* `virtio-iommu` backed PCI segments are now comprehensively placed behind the 1769 vIOMMU (#3870) 1770* Seccomp rules have been extended for `virtio-fs` to support direct access 1771 (#3848) 1772 1773### Deprecations 1774 1775Deprecated features will be removed in a subsequent release and users should 1776plan to use alternatives 1777 1778* The `mergeable` option from the `virtio-pmem` support has been deprecated 1779 (#3968) 1780* The `dax` option from the `virtio-fs` support has been deprecated (#3889) 1781 1782### Contributors 1783 1784Many thanks to everyone who has contributed to our release: 1785 1786* Bo Chen <chen.bo@intel.com> 1787* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1788* Henry Wang <Henry.Wang@arm.com> 1789* Jianyong Wu <jianyong.wu@arm.com> 1790* LiHui <andrewli@kubesphere.io> 1791* Michael Zhao <michael.zhao@arm.com> 1792* Rob Bradford <robert.bradford@intel.com> 1793* Sebastien Boeuf <sebastien.boeuf@intel.com> 1794* Wei Liu <liuwe@microsoft.com> 1795* William Douglas <william.douglas@intel.com> 1796* Yi Wang <wang.yi59@zte.com.cn> 1797 1798# v22.1 1799 1800This is a bug fix release. The following issues have been addressed: 1801 1802* VFIO ioctl reordering to fix MSI on AMD platforms (#3827) 1803* Fix `virtio-net` control queue (#3829) 1804 1805# v22.0 1806 1807### GDB Debug Stub Support 1808 1809Cloud Hypervisor can now be used as debug target with GDB. This is controlled 1810by the `gdb` compile time feature and details of how to use it can be found in 1811the [gdb 1812documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/gdb.md). 1813 1814### `virtio-iommu` Backed Segments 1815 1816In order to facilitate hotplug devices that require being behind an IOMMU (e.g. 1817QAT) there is a new option `--platform iommu_segments=<list_of_segments>` that 1818will place all the specified segments behind the IOMMU. 1819 1820### Before Boot Configuration Changes 1821 1822It is now possible to change the VM configuration (e.g. add or remove devices, 1823resize) before the VM is booted. 1824 1825### `virtio-balloon` Free Page Reporting 1826 1827If `--balloon free_page_reporting=on` is used then the guest can report pages 1828that is it not using to the VMM. The VMM will then notify the host OS that 1829those pages are no longer in use and can be freed. This can result in improved 1830memory density. 1831 1832### Support for Direct Kernel Booting with TDX 1833 1834Through the use of `TD-Shim` lightweight firmware it is now possible to 1835directly boot into the kernel with TDX. The [TDX 1836documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/intel_tdx.md#tdshim) 1837has been updated for this usage. 1838 1839### PMU Support for AArch64 1840 1841A PMU is now available on AArch64 for guest performance profiling. This will be 1842exposed automatically if available from the host. 1843 1844### Documentation Under CC-BY-4.0 License 1845 1846The documentation is now licensed under the "Creative Commons Attribution 4.0 1847International" license which is aligned with the project charter under the 1848Linux Foundation. 1849 1850### Deprecation of "Classic" `virtiofsd` 1851 1852The use of the Rust based [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd) 1853is now recommended and we are no longer testing against the C based "classic" 1854version. 1855 1856### Notable Bug Fixes 1857 1858* Can now be used on kernels without `AF_INET` support (#3785) 1859* `virtio-balloon` size is now validated against guest RAM size (#3689) 1860* Ensure that I/O related KVM VM Exits are correctly handled (#3677) 1861* Multiple TAP file descriptors can be used for `virtio-net` device hotplug (#3607) 1862* Minor API improvements and fixes (#3756, #3766, #3647, #3578) 1863* Fix sporadic seccomp violation from glibc memory freeing (#3610, #3609) 1864* Fix Windows 11 on AArch64 due to wider MSI-X register accesses (#3714, #3720) 1865* Ensure `vhost-user` features are correct across migration (#3737) 1866* Improved vCPU topology on AArch64 (#3735, #3733) 1867 1868### Contributors 1869 1870Many thanks to everyone who has contributed to our release: 1871 1872* Akira Moroo <retrage01@gmail.com> 1873* Barret Rhoden <brho@google.com> 1874* Bo Chen <chen.bo@intel.com> 1875* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1876* Feng Ye <yefeng@smartx.com> 1877* Henry Wang <Henry.Wang@arm.com> 1878* Jianyong Wu <jianyong.wu@arm.com> 1879* lizhaoxin1 <Lxiaoyouling@163.com> 1880* Michael Zhao <michael.zhao@arm.com> 1881* Rob Bradford <robert.bradford@intel.com> 1882* Sebastien Boeuf <sebastien.boeuf@intel.com> 1883* Wei Liu <liuwe@microsoft.com> 1884 1885# v21.0 1886 1887### Efficient Local Live Migration (for Live Upgrade) 1888 1889In order to support fast live upgrade of the VMM an optimised path has been 1890added in which the memory for the VM is not compared from source to 1891destination. This is activated by passing `--local` to the `ch-remote 1892send-migration` command. This means that the live upgrade can complete in the 1893order of 50ms vs 3s. (#3566) 1894 1895### Recommended Kernel is Now 5.15 1896 1897Due to an issue in the `virtio-net` code in 5.14 the recommended Linux kernel 1898is now 5.15. (#3530) 1899 1900### Notable Bug fixes 1901 1902* Multiple fixes were made to the OpenAPI YAML file to match the implementation (#3555,#3562) 1903* Avoid live migration deadlock when triggered during the kernel boot (#3585) 1904* Support live migration within firmware (#3586) 1905* Validate the `virtio-net` descriptor chain (#3548) 1906* `direct=on` (`O_DIRECT`) can now be used with a guest that makes unaligned accesses (e.g. firmware) (#3587) 1907 1908### Contributors 1909 1910Many thanks to everyone who has contributed to our release: 1911 1912* Anatol Belski <anbelski@linux.microsoft.com> 1913* Barret Rhoden <brho@google.com> 1914* Bo Chen <chen.bo@intel.com> 1915* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1916* Henry Wang <Henry.Wang@arm.com> 1917* Liang Zhou <zhoul110@chinatelecom.cn> 1918* Michael Zhao <michael.zhao@arm.com> 1919* Muhammad Falak R Wani <falakreyaz@gmail.com> 1920* Muminul Islam <muislam@microsoft.com> 1921* Rob Bradford <robert.bradford@intel.com> 1922* Sebastien Boeuf <sebastien.boeuf@intel.com> 1923* Wei Liu <liuwe@microsoft.com> 1924* Ziye Yang <ziye.yang@intel.com> 1925 1926# v20.2 1927 1928This is a bug fix release. The following issues have been addressed: 1929 1930* Don't error out when setting up the SIGWINCH handler (for console resize) 1931 when this fails due to older kernel (#3456) 1932* Seccomp rules were refined to remove syscalls that are now unused 1933* Fix reboot on older host kernels when SIGWINCH handler was not initialised 1934 (#3496) 1935* Fix virtio-vsock blocking issue (#3497) 1936 1937# v20.1 1938 1939This is a bug fix release. The following issues have been addressed: 1940 1941* Networking performance regression with `virtio-net` (#3450) 1942* Limit file descriptors sent in `vfio-user` support (#3401) 1943* Fully advertise PCI MMIO config regions in ACPI tables (#3432) 1944* Set the TSS and KVM identity maps so they don't overlap with firmware RAM 1945* Correctly update the `DeviceTree` on restore 1946 1947# v20.0 1948 1949### Multiple PCI segments support 1950 1951Cloud Hypervisor is no longer limited to 31 PCI devices. For both `x86_64` and 1952`aarch64` architectures, it is now possible to create up to 16 PCI segments, 1953increasing the total amount of supported PCI devices to 496. 1954 1955### CPU pinning 1956 1957For each vCPU, the user can define a limited set of host CPUs on which it is 1958allowed to run. This can be useful when assigning a 1:1 mapping between host and 1959guest resources, or when running a VM on a specific NUMA node. 1960 1961### Improved VFIO support 1962 1963Based on VFIO region capabilities, all regions can be memory mapped, limiting 1964the amount of triggered VM exits, and therefore increasing the performance of 1965the passthrough device. 1966 1967### Safer code 1968 1969Several sections containing unsafe Rust code have been replaced with safe 1970alternatives, and multiple comments have been added to clarify why the remaining 1971unsafe sections are safe to use. 1972 1973### Extended documentation 1974 1975The documentation related to VFIO has been updated while some new documents have 1976been introduced to cover the usage of `--cpus` parameter as well as how to run 1977Cloud Hypervisor on Intel TDX. 1978 1979### Notable bug fixes 1980 1981* Naturally align PCI BARs on relocation (#3244) 1982* Fix panic in SIGWINCH listener thread when no seccomp filter set (#3338) 1983* Use the tty raw mode implementation from libc (#3344) 1984* Fix the emulation of register D for CMOS/RTC device (#3393) 1985 1986### Contributors 1987 1988Many thanks to everyone who has contributed to our release: 1989 1990* Alyssa Ross <hi@alyssa.is> 1991* Bo Chen <chen.bo@intel.com> 1992* Fabiano Fidêncio <fabiano.fidencio@intel.com> 1993* Michael Zhao <michael.zhao@arm.com> 1994* Muminul Islam <muislam@microsoft.com> 1995* Rob Bradford <robert.bradford@intel.com> 1996* Sebastien Boeuf <sebastien.boeuf@intel.com> 1997* Wei Liu <liuwe@microsoft.com> 1998* Willen Yang <willenyang@gmail.com> 1999* William Douglas <william.douglas@intel.com> 2000* Ziye Yang <ziye.yang@intel.com> 2001 2002# v19.0 2003 2004### Improved PTY handling for serial and `virtio-console` 2005 2006The PTY support for serial has been enhanced with improved buffering when the 2007the PTY is not yet connected to. Using `virtio-console` with PTY now results in 2008the console being resized if the PTY window is also resized. 2009 2010### PCI boot time optimisations 2011 2012Multiple optimisations have been made to the PCI handling resulting in 2013significant improvements in the boot time of the guest. 2014 2015### Improved TDX support 2016 2017When using the latest TDVF firmware the ACPI tables created by the VMM are now 2018exposed via the firmware to the guest. 2019 2020### Live migration enhancements 2021 2022Live migration support has been enhanced to support migration with `virtio-mem` 2023based memory hotplug and the `virtio-balloon` device now supports live 2024migration. 2025 2026### `virtio-mem` support with `vfio-user` 2027 2028The use of `vfio-user` userspaces devices can now be used in conjunction with 2029`virtio-mem` based memory hotplug and unplug. 2030 2031### AArch64 for `virtio-iommu` 2032 2033A paravirtualised IOMMU can now be used on the AArch64 platform. 2034 2035### Notable bug fixes 2036 2037* ACPI hotplugged memory is correctly restored after a live migration or 2038 snapshot/restore (#3165) 2039* Multiple devices from the same IOMMU group can be passed through via VFIO 2040 (#3078 #3113) 2041* Live migration with large blocks of memory was buggy due to an in issue in 2042 the underlying crate (#3157) 2043 2044### Contributors 2045 2046Many thanks to everyone who has contributed to our release: 2047 2048* Alyssa Ross <hi@alyssa.is> 2049* Bo Chen <chen.bo@intel.com> 2050* Henry Wang <Henry.Wang@arm.com> 2051* Hui Zhu <teawater@antfin.com> 2052* Jianyong Wu <jianyong.wu@arm.com> 2053* Li Yu <liyu.yukiteru@bytedance.com> 2054* Michael Zhao <michael.zhao@arm.com> 2055* Muminul Islam <muislam@microsoft.com> 2056* Rob Bradford <robert.bradford@intel.com> 2057* Sebastien Boeuf <sebastien.boeuf@intel.com> 2058* Wei Liu <liuwe@microsoft.com> 2059* William Douglas <william.douglas@intel.com> 2060* Yu Li <liyu.yukiteru@bytedance.com> 2061 2062# v18.0 2063 2064### Experimental User Device (`vfio-user`) support 2065 2066Experimental support for running PCI devices in userspace via `vfio-user` 2067has been included. This allows the use of the SPDK NVMe `vfio-user` controller 2068with Cloud Hypervisor. This is enabled by `--user-device` on the command line. 2069 2070### Migration support for `vhost-user` devices 2071 2072Devices exposed into the VM via `vhost-user` can now be migrated using the live 2073migration support. This requires support from the backend however the commonly 2074used DPDK `vhost-user` backend does support this. 2075 2076### VHDX disk image support 2077 2078Images using the VHDX disk image format can now be used with Cloud Hypervisor. 2079 2080### Device pass through on MSHV hypervisor 2081 2082When running on the MSHV hypervisor it is possible to pass through devices from 2083the host through to the guest (e.g with `--device`) 2084 2085### AArch64 for support `virtio-mem` 2086 2087The reference Linux kernel we recommend for using with Cloud Hypervisor now supports `virtio-mem` on AArch64. 2088 2089### Live migration on MSHV hypervisor 2090 2091Live migration is now supported when running on the MSHV hypervisor including 2092efficient tracking of dirty pages. 2093 2094### AArch64 CPU topology support 2095 2096The CPU topology (as configured through `--cpu topology=`) can now be 2097configured on AArch64 platforms and is conveyed through either ACPI or device 2098tree. 2099 2100### Power button support on AArch64 2101 2102Use of the ACPI power button (e.g `ch-remote --api-socket=<API socket> power-button`) 2103is now supported when running on AArch64. 2104 2105### Notable bug fixes 2106 2107* Using two PTY outputs e.g. `--serial pty --console pty` now works correctly (#3012) 2108* TTY input is now always sent to the correct destination (#3005) 2109* The boot is no longer blocked when using a unattached PTY on the serial console (#3004) 2110* Live migration is now supported on AArch64 (#3049) 2111* Ensure signal handlers are run on the correct thread (#3069) 2112 2113### Contributors 2114 2115Many thanks to everyone who has contributed to our release: 2116 2117* Alyssa Ross <hi@alyssa.is> 2118* Anatol Belski <anbelski@linux.microsoft.com> 2119* Arafatms <arafatms@outlook.com> 2120* Bo Chen <chen.bo@intel.com> 2121* Fazla Mehrab <akm.fazla.mehrab@vt.edu> 2122* Henry Wang <Henry.Wang@arm.com> 2123* Jianyong Wu <jianyong.wu@arm.com> 2124* Jiaqi Gao <jiaqi.gao@intel.com> 2125* Markus Theil <markus.theil@tu-ilmenau.de> 2126* Muminul Islam <muislam@microsoft.com> 2127* Rob Bradford <robert.bradford@intel.com> 2128* Sebastien Boeuf <sebastien.boeuf@intel.com> 2129* Wei Liu <liuwe@microsoft.com> 2130* Yu Li <liyu.yukiteru@bytedance.com> 2131 2132# v17.0 2133 2134### ARM64 NUMA support using ACPI 2135 2136The support for ACPI on ARM64 has been enhanced to include support for 2137specifying a NUMA configuration using the existing control options. 2138 2139### `Seccomp` support for MSHV backend 2140 2141The `seccomp` rules have now been extended to support running against the MSHV 2142hypervisor backend. 2143 2144### Hotplug of `macvtap` devices 2145 2146Hotplug of `macvtap` devices is now supported with the file descriptor for the 2147network device if opened by the user and passed to the VMM. The `ch-remote` 2148tool supports this functionality when adding a network device. 2149 2150### Improved SGX support 2151 2152The SGX support has been updated to match the latest Linux kernel support and 2153now supports SGX provisioning and associating EPC sections to NUMA nodes. 2154 2155### Inflight tracking for `vhost-user` devices 2156 2157Support for handling inflight tracking of I/O requests has been added to the 2158`vhost-user` devices allowing recovery after device reconnection. 2159 2160### Notable bug fixes 2161 2162* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821). 2163* The VMM side of `vhost-user` devices no longer advertise the 2164 `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM 2165(#2833). 2166* On ARM64 VMs can be created with more than 16 vCPUs (#2763). 2167 2168### Contributors 2169 2170Many thanks to everyone who has contributed to our release: 2171 2172* Anatol Belski <anbelski@linux.microsoft.com> 2173* Arafatms <arafatms@outlook.com> 2174* Bo Chen <chen.bo@intel.com> 2175* Fei Li <lifei.shirley@bytedance.com> 2176* Henry Wang <Henry.Wang@arm.com> 2177* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2178* Jianyong Wu <jianyong.wu@arm.com> 2179* Li Hangjing <lihangjing@bytedance.com> 2180* Michael Zhao <michael.zhao@arm.com> 2181* Muminul Islam <muislam@microsoft.com> 2182* Rob Bradford <robert.bradford@intel.com> 2183* Sebastien Boeuf <sebastien.boeuf@intel.com> 2184* Wei Liu <liuwe@microsoft.com> 2185* Yukiteru <wfly1998@sina.com> 2186 2187# v16.0 2188 2189### Improved live migration support 2190 2191The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details. 2192 2193### Improved `vhost-user` support 2194 2195When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch. 2196 2197### ARM64 ACPI and UEFI support 2198 2199Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot. 2200 2201### Notable bug fixes 2202 2203* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578). 2204* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563). 2205* The KVM clock is now correctly handled during live migration / snapshot & restore. 2206 2207### Removed functionality 2208 2209The following formerly deprecated features have been removed: 2210 2211* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2212 binaries has been deprecated. When using direct boot users should configure 2213 their kernel with `CONFIG_PVH=y`. 2214 2215### Contributors 2216 2217Many thanks to everyone who has contributed to our release including some new faces. 2218 2219* Anatol Belski <anbelski@linux.microsoft.com> 2220* Bo Chen <chen.bo@intel.com> 2221* Dayu Liu <liu.dayu@zte.com.cn> 2222* Henry Wang <Henry.Wang@arm.com> 2223* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2224* Jianyong Wu <jianyong.wu@arm.com> 2225* Michael Zhao <michael.zhao@arm.com> 2226* Mikko Ylinen <mikko.ylinen@intel.com> 2227* Muminul Islam <muislam@microsoft.com> 2228* Ren Lei <ren.lei4@zte.com.cn> 2229* Rob Bradford <robert.bradford@intel.com> 2230* Sebastien Boeuf <sebastien.boeuf@intel.com> 2231* Wei Liu <liuwe@microsoft.com> 2232* Yi Wang <wang.yi59@zte.com.cn> 2233 2234# v15.0 2235 2236Highlights for `cloud-hypervisor` version v15.0 include: 2237 2238### Version numbering and stability guarantees 2239 2240This release is the first in a new version numbering scheme to represent that 2241we believe Cloud Hypervisor is maturing and entering a period of stability. 2242With this new release we are beginning our new stability guarantees: 2243 2244* The API (including command line options) will not be removed or changed in a 2245 breaking way without a minimum of 2 releases notice. Where possible warnings 2246 will be given about the use of deprecated functionality and the deprecations 2247 will be documented in the release notes. 2248* Point releases will be made between individual releases where there are 2249 substantial bug fixes or security issues that need to be fixed. 2250 2251Currently the following items are **not** guaranteed across updates: 2252 2253* Snapshot/restore is not supported across different versions 2254* Live migration is not supported across different versions 2255* The following features are considered experimental and may change 2256 substantially between releases: TDX, SGX. 2257 2258### Network device rate limiting 2259 2260Building on our existing support for rate limiting block activity the network 2261device also now supports rate limiting. Full details of the controls are in the 2262[IO throttling documentation.](docs/io_throttling.md) 2263 2264### Support for runtime control of `virtio-net` guest offload 2265 2266The guest is now able to change the offload settings for the `virtio-net` 2267device. As well as providing a useful control this mitigates an issue in the 2268Linux kernel where the guest will attempt to reprogram the offload settings 2269even if they are not advertised as configurable (#2528). 2270 2271### `--api-socket` supports file descriptor parameter 2272 2273The `--api-socket` can now take an `fd=` parameter to specify an existing file 2274descriptor to use. This is particularly beneficial for frameworks that need to 2275programmatically control Cloud Hypervisor. 2276 2277### Bug fixes 2278 2279* A workaround has been put in place to mitigate a Linux kernel issues that 2280 results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277). 2281* PCI BARs are now correctly aligned removing the need for the guest to 2282 reprogram them (#1797,#1798) 2283* Handle TAP interface not being writable within virtio-net (due to the buffer 2284 exhaustion on the host) (#2517) 2285* The recommended Linux kernel is now v5.12.0 as it contains a fix that 2286 prevents snapshot & restore working (#2535) 2287 2288### Deprecations 2289 2290Deprecated features will be removed in a subsequent release and users should plan to use alternatives 2291 2292* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2293 binaries has been deprecated. When using direct boot users should configure 2294 their kernel with `CONFIG_PVH=y`. Will be removed in v16.0. 2295 2296### Contributors 2297 2298Many thanks to everyone who has contributed to our release including some new faces. 2299 2300* Alyssa Ross <hi@alyssa.is> 2301* Anatol Belski <anbelski@linux.microsoft.com> 2302* Bo Chen <chen.bo@intel.com> 2303* Gaelan Steele <gbs@canishe.com> 2304* Jianyong Wu <jianyong.wu@arm.com> 2305* Michael Zhao <michael.zhao@arm.com> 2306* Muminul Islam <muislam@microsoft.com> 2307* Rob Bradford <robert.bradford@intel.com> 2308* Sebastien Boeuf <sebastien.boeuf@intel.com> 2309* Wei Liu <liuwe@microsoft.com> 2310* William Douglas <william.douglas@intel.com> 2311 2312# v0.14.1 2313 2314Bug fix release branched off the v0.14.0 release. The following bugs were fixed 2315in this release: 2316 2317* CPU hotplug on Windows failed due to misreported CPU state information and 2318 the lack of HyperV CPUID bit enabled (#2437, #2449, #2436) 2319* A seccomp rule was missing that was triggered on CPU unplug (#2455) 2320* A bounds check in VIRTIO queue validation was erroneously generating 2321 DescriptorChainTooShort errors in certain circumstances (#2450, #2424) 2322 2323# v0.14.0 2324 2325Highlights for `cloud-hypervisor` version 0.14.0 include: 2326 2327### Structured event monitoring 2328 2329A new option was added to the VMM `--event-monitor` which reports structured 2330events (JSON) over a file or file descriptor at key events in the lifecycle of 2331the VM. The list of events is limited at the moment but will be further 2332extended over subsequent releases. The events exposed form part of the Cloud 2333Hypervisor API surface. 2334 2335### MSHV improvements 2336 2337Basic support has been added for running Windows guests atop the MSHV 2338hypervisor as an alternative to KVM and further improvements have been made to 2339the MSHV support. 2340 2341### Improved aarch64 platform 2342 2343The aarch64 platform has been enhanced with more devices exposed to the running 2344VM including an enhanced serial UART. 2345 2346### Updated hotplug documentation 2347 2348The documentation for the hotplug support has been updated to reflect the use 2349of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as 2350well as documenting hotplug of paravirtualised and VFIO devices. 2351 2352### PTY control for serial and `virtio-console` 2353 2354The `--serial` and `--console` parameters can now direct the console to a PTY 2355allowing programmatic control of the console from another process through the 2356PTY subsystem. 2357 2358### Block device rate limiting 2359 2360The block device performance can now be constrained as part of the VM 2361configuration allowing rate limiting. Full details of the controls are in the 2362[IO throttling documentation.](docs/io_throttling.md) 2363 2364 2365### Deprecations 2366 2367Deprecated features will be removed in a subsequent release and users should plan to use alternatives 2368 2369* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 2370 binaries has been deprecated. When using direct boot users should configure 2371 their kernel with `CONFIG_PVH=y`. 2372 2373 2374### Contributors 2375 2376Many thanks to everyone who has contributed to our 0.14.0 release including 2377some new faces. 2378 2379* Bo Chen <chen.bo@intel.com> 2380* Henry Wang <Henry.Wang@arm.com> 2381* Iggy Jackson <iggy@theiggy.com> 2382* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 2383* Michael Zhao <michael.zhao@arm.com> 2384* Muminul Islam <muislam@microsoft.com> 2385* Penny Zheng <Penny.Zheng@arm.com> 2386* Rob Bradford <robert.bradford@intel.com> 2387* Sebastien Boeuf <sebastien.boeuf@intel.com> 2388* Vineeth Pillai <viremana@linux.microsoft.com> 2389* Wei Liu <liuwe@microsoft.com> 2390* William Douglas <william.r.douglas@gmail.com> 2391* Zide Chen <zide.chen@intel.com> 2392 2393# v0.13.0 2394 2395Highlights for `cloud-hypervisor` version 0.13.0 include: 2396 2397### Wider VFIO device support 2398 2399It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI 2400devices that do not support MSI or MSI-X and instead rely on INTx interrupts. 2401Most notably this widens the support to most NVIDIA cards with the proprietary 2402drivers. 2403 2404### Improved huge page support 2405 2406Through the addition of `hugepage_size` on `--memory` it is now possible to 2407specify the desired size of the huge pages used when allocating the guest 2408memory. The user is required to ensure they have sufficient pages of the 2409desired size in their pool. 2410 2411### MACvTAP support 2412 2413It is now possible to provide file descriptors using the `fd` parameter to 2414`--net` which point at TAP devices that have already been opened by the user. 2415This aids integration with `libvirt` but also permits the use of MACvTAP 2416support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md). 2417 2418### VHD disk image support 2419 2420It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw 2421disk image with Cloud Hypervisor. 2422 2423### Improved Virtio device threading 2424 2425Device threads are now derived from the main VMM thread which allows more 2426restrictive seccomp filters to be applied to them. The threads also have a 2427predictable name derived from the device id. 2428 2429### Clean shutdown support via synthetic power button 2430 2431It is now possible to request that the guest VM shut itself down by triggering 2432a synthetic ACPI power button press from the VMM. If the guest is listening for 2433such an event (e.g. using systemd) then it will process the event and cleanly 2434shut down. This functionality is exposed through the HTTP API and can be 2435triggered via `ch-remote --api-socket=<API socket> power-button`. 2436 2437### Contributors 2438 2439Many thanks to everyone who has contributed to our 0.13.0 release including 2440some new faces. 2441 2442* Bo Chen <chen.bo@intel.com> 2443* Mikko Ylinen <mikko.ylinen@intel.com> 2444* Muminul Islam <muislam@microsoft.com> 2445* Rob Bradford <robert.bradford@intel.com> 2446* Samuel Ortiz <sameo@linux.intel.com> 2447* Sebastien Boeuf <sebastien.boeuf@intel.com> 2448* Vineeth Pillai <viremana@linux.microsoft.com> 2449* Wei Liu <liuwe@microsoft.com> 2450* William Douglas <william.r.douglas@gmail.com> 2451* Xie Yongji <xieyongji@bytedance.com> 2452 2453# v0.12.0 2454 2455Highlights for `cloud-hypervisor` version 0.12.0 include: 2456 2457### ARM64 enhancements 2458 2459The use of `--watchdog` is now fully supported as is the ability to reboot the 2460VM from within the guest when running Cloud Hypervisor on an ARM64 system. 2461 2462### Removal of `vhost-user-net` and `vhost-user-block` self spawning 2463 2464In order to use `vhost-user-net` or `vhost-user-block` backends the user is now 2465responsible for starting the backend and providing the socket for the VMM to 2466use. This functionality was deprecated in the last release and how now been 2467removed. 2468 2469### Migration of `vhost-user-fs` backend 2470 2471The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is 2472instead hosted in [its own 2473repository](https://gitlab.com/virtio-fs/virtiofsd-rs) 2474 2475### Enhanced "info" API 2476 2477The `vm.info` HTTP API endpoint has been extended to include the details of the 2478devices used by the VM including any VFIO devices used. 2479 2480### Contributors 2481 2482Many thanks to everyone who has contributed to our 0.12.0 release: 2483 2484* Anatol Belski <anbelski@linux.microsoft.com> 2485* Julio Montes <julio.montes@intel.com> 2486* Michael Zhao <michael.zhao@arm.com> 2487* Muminul Islam <muislam@microsoft.com> 2488* Rob Bradford <robert.bradford@intel.com> 2489* Samuel Ortiz <sameo@linux.intel.com> 2490* Wei Liu <liuwe@microsoft.com> 2491 2492# v0.11.0 2493 2494Highlights for `cloud-hypervisor` version 0.11.0 include: 2495 2496### `io_uring` support by default for `virtio-block` 2497 2498Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will 2499be used for a significantly higher performance block device. 2500 2501### Windows Guest Support 2502 2503This is the first release where we officially support Windows running as a 2504guest. Full details of how to setup the image and run Cloud Hypervisor with a 2505Windows guest can be found in the dedicated [Windows 2506documentation](docs/windows.md). 2507 2508### `vhost-user` "Self Spawning" Deprecation 2509 2510Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now 2511deprecated. Users of this functionality will receive a warning and should make 2512adjustments. The functionality will be removed in the next release. 2513 2514### `virtio-mmio` Removal 2515 2516Support for using the `virtio-mmio` transport, rather than using PCI, has been 2517removed. This has been to simplify the code and significantly 2518reduce the testing burden of the project. 2519 2520### Snapshot/Restore support for ARM64 2521 2522When running on the ARM64 architecture snapshot and restore has now been 2523implemented. 2524 2525### Improved Linux Boot Time 2526 2527The time to boot the Linux kernel has been significantly improved by the 2528identifying some areas of delays around PCI bus probing, IOAPIC programming and 2529MPTABLE issues. Full details can be seen in #1728. 2530 2531### `SIGTERM/SIGINT` Interrupt Signal Handling 2532 2533When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will 2534trigger the VMM process to cleanly deallocate resources before exiting. The 2535guest VM will not be cleanly shutdown but the VMM process will clean up its 2536resources. 2537 2538### Default Log Level Changed 2539 2540The default logging level was changed to include warnings which should make it 2541easier to see potential issues. New [logging 2542documentation](docs/logging.md) was also added. 2543 2544### New `--balloon` Parameter Added 2545 2546Control of the setup of `virtio-balloon` has been moved from `--memory` to its 2547own dedicated parameter. This makes it easier to add more balloon specific 2548controls without overloading `--memory`. 2549 2550### Experimental `virtio-watchdog` Support 2551 2552Support for using a new `virtio-watchdog` has been added which can be used to 2553have the VMM reboot the guest if the guest userspace fails to ping the 2554watchdog. This is enabled with `--watchdog` and requires kernel support. 2555 2556### Notable Bug Fixes 2557 2558* MTRR bit was missing from CPUID advertised to guest 2559* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170) 2560* CPU identification string is now exposed to the guest 2561* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as 2562 read only so avoids excessive VM exits (#1795) 2563* PCI device hotplug after an unplug was fixed (#1802) 2564* When using the ACPI method to resize the guest memory the full reserved size 2565 can be used (#1803) 2566* Snapshot and restore followed by a second snapshot and restore now works 2567 correctly 2568* Snapshot and restore of VMs with more than 2GiB in one region now work 2569 correctly 2570 2571### Contributors 2572 2573Many thanks to everyone who has contributed to our 0.11.0 release including some new faces. 2574 2575* Anatol Belski <anbelski@linux.microsoft.com> 2576* Bo Chen <chen.bo@intel.com> 2577* Daniel Verkamp <dverkamp@chromium.org> 2578* Henry Wang <Henry.Wang@arm.com> 2579* Hui Zhu <teawater@antfin.com> 2580* Jiangbo Wu <jiangbo.wu@intel.com> 2581* Josh Soref <jsoref@users.noreply.github.com> 2582* Julio Montes <julio.montes@intel.com> 2583* Michael Zhao <michael.zhao@arm.com> 2584* Muminul Islam <muislam@microsoft.com> 2585* pierwill <19642016+pierwill@users.noreply.github.com> 2586* Praveen Paladugu <prapal@microsoft.com> 2587* Rob Bradford <robert.bradford@intel.com> 2588* Sebastien Boeuf <sebastien.boeuf@intel.com> 2589* Wei Liu <liuwe@microsoft.com> 2590 2591 2592# v0.10.0 2593 2594Highlights for `cloud-hypervisor` version 0.10.0 include: 2595 2596### `virtio-block` Support for Multiple Descriptors 2597 2598Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers. 2599 2600### Memory Zones 2601 2602Support has been added for fine-grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md). 2603 2604### `Seccomp` Sandbox Improvements 2605 2606All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`. 2607 2608### Preliminary KVM HyperV Emulation Control 2609 2610A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices. 2611 2612### Notable Bug Fixes 2613 2614- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596) 2615- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564) 2616- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593) 2617- `--version` shows the version for released binaries (#1669) 2618- Errors generated by worker threads for `virtio` devices are now printed out (#1551) 2619 2620### Contributors 2621 2622Many thanks to everyone who has contributed to our 0.10.0 release including some new faces. 2623 2624* Alyssa Ross <hi@alyssa.is> 2625* Amey Narkhede <ameynarkhede02@gmail.com> 2626* Anatol Belski <ab@php.net> 2627* Bo Chen <chen.bo@intel.com> 2628* Hui Zhu <teawater@antfin.com> 2629* Michael Zhao <michael.zhao@arm.com> 2630* Muminul Islam <muislam@microsoft.com> 2631* Rob Bradford <robert.bradford@intel.com> 2632* Samuel Ortiz <sameo@linux.intel.com> 2633* Sebastien Boeuf <sebastien.boeuf@intel.com> 2634* Wei Liu <liuwe@microsoft.com> 2635 2636# v0.9.0 2637 2638Highlights for `cloud-hypervisor` version 0.9.0 include: 2639 2640### `io_uring` Based Block Device Support 2641 2642If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement. 2643 2644### Block and Network Device Statistics 2645 2646Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM. 2647 2648### HTTP API Responses 2649 2650The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF. 2651 2652### CPU Topology 2653 2654A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core. 2655 2656### Release Build Optimization 2657 2658Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size. 2659 2660### Hypervisor Abstraction 2661 2662A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`. 2663 2664### Snapshot/Restore Improvements 2665 2666Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs. 2667 2668### Virtio Memory Ballooning Support 2669 2670A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest. 2671 2672### Enhancements to ARM64 Support 2673 2674The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting. 2675 2676### Intel SGX Support 2677 2678The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md). 2679 2680### `Seccomp` Sandbox Improvements 2681 2682The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development. 2683 2684### Notable Bug Fixes 2685 2686* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes. 2687* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338) 2688* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543). 2689* `virtio-mmio` based devices are now more widely tested (#275). 2690* Multiple issues have been fixed with virtio device configuration (#1217) 2691* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521) 2692 2693### Contributors 2694 2695Many thanks to everyone who has contributed to our 0.9.0 release including some new faces. 2696 2697* Anatol Belski <ab@php.net> 2698* Bo Chen <chen.bo@intel.com> 2699* Dr. David Alan Gilbert <dgilbert@redhat.com> 2700* Henry Wang <Henry.Wang@arm.com> 2701* Howard Zhang <howard.zhang@arm.com> 2702* Hui Zhu <teawater@antfin.com> 2703* Jianyong Wu <jianyong.wu@arm.com> 2704* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2705* LiYa'nan <oliverliyn@gmail.com> 2706* Michael Zhao <michael.zhao@arm.com> 2707* Muminul Islam <muislam@microsoft.com> 2708* Praveen Paladugu <prapal@microsoft.com> 2709* Ricardo Koller <ricarkol@gmail.com> 2710* Rob Bradford <robert.bradford@intel.com> 2711* Samuel Ortiz <sameo@linux.intel.com> 2712* Sebastien Boeuf <sebastien.boeuf@intel.com> 2713* Stefano Garzarella <sgarzare@redhat.com> 2714* Wei Liu <liuwe@microsoft.com> 2715 2716 2717# v0.8.0 2718 2719Highlights for `cloud-hypervisor` version 0.8.0 include: 2720 2721### Experimental Snapshot and Restore Support 2722 2723This release includes the first version of the snapshot and restore feature. 2724This allows a VM to be paused and then subsequently snapshotted. At a later 2725point that snapshot may be restored into a new running VM identical to the 2726original VM at the point it was paused. 2727 2728This feature can be used for offline migration from one VM host to another, to 2729allow the upgrading or rebooting of the host machine transparently to the guest 2730or for templating the VM. This is an experimental feature and cannot be used on 2731a VM using passthrough (VFIO) devices. Issues with SMP have also been observed 2732(#1176). 2733 2734### Experimental ARM64 Support 2735 2736Included in this release is experimental support for running on ARM64. 2737Currently only `virtio-mmio` devices and a serial port are supported. Full 2738details can be found in the [ARM64 documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/8ab15b9a984a448006f97b1211498c1bb583de3b/docs/arm64.md). 2739 2740### Support for Using 5-level Paging in Guests 2741 2742If the host supports it the guest is now enabled for 5-level paging (aka LA57). 2743This works when booting the Linux kernel with a vmlinux, bzImage or firmware 2744based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not 2745work due to current limitations in the PVH boot process. 2746 2747### Virtio Device Interrupt Suppression for Network Devices 2748 2749With `virtio-net` and `vhost-user-net` devices the guest can suppress 2750interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This 2751can lead to an improvement in performance by reducing the number of interrupts 2752the guest must service. 2753 2754### `vhost_user_fs` Improvements 2755 2756The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`. 2757 2758 2759### Notable Bug Fixes 2760 2761* VMs that have not yet been booted can now be deleted (#1110). 2762* By creating the `tap` device ahead of creating the VM it is not required to 2763 run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273). 2764* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to 2765 the specification and synchronizes to the underlying filesystem as required 2766 based on guest feature negotiation. This avoids potential data loss (#399, 2767 #1216). 2768* When booting with a large number of vCPUs then the ACPI table would be 2769 overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the 2770 `MPTABLE` will no longer be generated (#1132). 2771* Shutting down VMs that have been paused is now supported (#816). 2772* Created socket files are deleted on shutdown (#1083). 2773* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds 2774 (#751). 2775 2776### Command Line and API Changes 2777 2778This is non exhaustive list of HTTP API and command line changes: 2779 2780* All user visible socket parameters are now consistently called `socket` 2781 rather than `sock` in some cases. 2782* The `ch-remote` tool now shows any error message generated by the VMM 2783* The `wce` parameter has been removed from `--disk` as the feature is always 2784 offered for negotiation. 2785* `--net` has gained a `host_mac` option that allows the setting of the MAC 2786 address for the `tap` device on the host. 2787 2788### Contributors 2789 2790Many thanks to everyone who has contributed to our 0.8.0 release including some new faces. 2791 2792* Anatol Belski <ab@php.net> 2793* Arron Wang <arron.wang@intel.com> 2794* Bo Chen <chen.bo@intel.com> 2795* Dr. David Alan Gilbert <dgilbert@redhat.com> 2796* Henry Wang <Henry.Wang@arm.com> 2797* Hui Zhu <teawater@antfin.com> 2798* LiYa'nan <oliverliyn@gmail.com> 2799* Michael Zhao <michael.zhao@arm.com> 2800* Rob Bradford <robert.bradford@intel.com> 2801* Samuel Ortiz <sameo@linux.intel.com> 2802* Sebastien Boeuf <sebastien.boeuf@intel.com> 2803* Sergio Lopez <slp@redhat.com> 2804 2805# v0.7.0 2806 2807Highlights for `cloud-hypervisor` version 0.7.0 include: 2808 2809### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug 2810 2811Further to our effort to support modifying a running guest we now support 2812hotplug and unplug of the following virtio backed devices: block, network, 2813pmem, virtio-fs and vsock. This functionality is available on the (default) PCI 2814based transport and is exposed through the HTTP API. The `ch-remote` utility 2815provides a CLI for adding or removing these device types after the VM has 2816booted. User can use the `id` parameter on the devices to choose names for 2817devices to ease their removal. 2818 2819### Alternative `libc` Support 2820 2821Cloud Hypervisor can now be compiled with the `musl` C library and this release 2822contains a static binary compiled using that toolchain. 2823 2824### Multithreaded Multi Queued `vhost-user` Backends 2825 2826The `vhost-user` backends for network and block support that are shipped by 2827Cloud Hypervisor have been enhanced to support multiple threads and queues to 2828improve throughput. These backends are used automatically if `vhost_user=true` 2829is passed when the devices are created. 2830 2831### Initial RamFS Support 2832 2833By passing the `--initramfs` command line option the user can specify a file to 2834be loaded into the guest memory to be used as the kernel initial filesystem. 2835This is usually used to allow the loading of drivers needed to be able to 2836access the real root filesystem but it can also be used standalone for a very 2837minimal image. 2838 2839### Alternative Memory Hotplug: `virtio-mem` 2840 2841As well as supporting ACPI based hotplug Cloud Hypervisor now supports using 2842the `virtio-mem` hotplug alternative. This can be controlled by the 2843`hotplug_method` parameter on the `--memory` command line option. It currently 2844requires kernel patches to be able to support it. 2845 2846### `Seccomp` Sandboxing 2847 2848Cloud Hypervisor now has support for restricting the system calls that the 2849process can use via the `seccomp` security API. This on by default and is 2850controlled by the `--seccomp` command line option. 2851 2852### Updated Distribution Support 2853 2854With the release of Ubuntu 20.04 we have added that to the list of supported 2855distributions and is part of our regular testing programme. 2856 2857### Command Line and API Changes 2858 2859This is non exhaustive list of HTTP API and command line changes 2860 2861* New `id` fields added for devices to allow them to be named to ease removal. 2862 If no name is specified the VMM chooses one. 2863* Use `--memory`'s `shared` and `hugepages` controls for determining backing 2864 memory instead of providing a path. 2865* The `--vsock` parameter only takes one device as the Linux kernel only 2866 supports a single Vsock device. The REST API has removed the vector for this 2867 option and replaced it with a single optional field. 2868* There is enhanced validation of the command line and API provided 2869 configurations to ensure that the provided options are compatible e.g. that 2870 shared memory is in use if any attempt is made to used a `vhost-user` backed 2871 device. 2872* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and 2873 `add-vsock` subcommands. For removal `remove-device` is used. The REST API 2874 has appropriate new HTTP endpoints too. 2875* Specifying a `size` with `--pmem` is no longer required and instead the size 2876 will be obtained from the file. A `discard_writes` option has also been added 2877 to provide the equivalent of a read-only file. 2878* The parameters to `--block-backend` have been changed to more closely align 2879 with those used by `--disk`. 2880 2881### Contributors 2882 2883Many thanks to everyone who has contributed to our 0.7.0 release including some new faces. 2884 2885* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 2886* Bo Chen <chen.bo@intel.com> 2887* Cathy Zhang <cathy.zhang@intel.com> 2888* Damjan Georgievski <gdamjan@gmail.com> 2889* Dean Sheather <dean@coder.com> 2890* Eryu Guan <eguan@linux.alibaba.com> 2891* Hui Zhu <teawater@antfin.com> 2892* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2893* Martin Xu <martin.xu@intel.com> 2894* Muminul Islam <muislam@microsoft.com> 2895* Rob Bradford <robert.bradford@intel.com> 2896* Samuel Ortiz <sameo@linux.intel.com> 2897* Sebastien Boeuf <sebastien.boeuf@intel.com> 2898* Sergio Lopez <slp@redhat.com> 2899* Yang Zhong <yang.zhong@intel.com> 2900* Yi Sun <yi.y.sun@linux.intel.com> 2901 2902# v0.6.0 2903 2904Highlights for `cloud-hypervisor` version 0.6.0 include: 2905 2906### Directly Assigned Devices Hotplug 2907 2908We continued our efforts around supporting dynamically changing the guest 2909resources. After adding support for CPU and memory hotplug, Cloud Hypervisor 2910now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`) 2911devices into an already running guest. This closes the features gap for 2912providing a complete Kata Containers workloads support with Cloud Hypervisor. 2913 2914### Shared Filesystem Improvements 2915 2916We enhanced our shared filesystem support through many `virtio-fs` improvements. 2917By adding support for DAX, parallel processing of multiple requests, `FS_IO`, 2918`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we 2919improved our filesystem sharing performance, but also made it more stable and 2920compatible with other `virtio-fs` implementations. 2921 2922### Block and Networking IO Self Offloading 2923 2924When choosing to offload the paravirtualized block and networking I/O to an 2925external process (through the `vhost-user` protocol), Cloud Hypervisor now 2926automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends 2927into their own, separate processes. 2928This provides a seamless paravirtualized I/O user experience for those who want 2929to run their guest I/O into separate executions contexts. 2930 2931### Command Line Interface 2932 2933More and more Cloud Hypervisor services are exposed through the 2934[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only 2935accessible via relatively cumbersome HTTP calls. In order to abstract 2936those calls into a more user friendly tool, we created a Cloud Hypervisor 2937Command Line Interface (CLI) called `ch-remote`. The `ch-remote` binary 2938is created with each build and available e.g. at 2939`cloud-hypervisor/target/debug/ch-remote` when doing a debug build. 2940 2941Please check `ch-remote --help` for a complete description of all available 2942commands. 2943 2944### PVH Boot 2945 2946In addition to the traditional Linux boot protocol, Cloud Hypervisor now 2947supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html). 2948 2949### Contributors 2950 2951With the 0.6.0 release, we are welcoming a few new contributors. Many thanks 2952to them and to everyone that contributed to this release: 2953 2954* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 2955* Arron Wang <arron.wang@intel.com> 2956* Bin Liu <liubin0329@gmail.com> 2957* Bo Chen <chen.bo@intel.com> 2958* Cathy Zhang <cathy.zhang@intel.com> 2959* Eryu Guan <eguan@linux.alibaba.com> 2960* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 2961* Liu Bo <bo.liu@linux.alibaba.com> 2962* Qiu Wenbo <qiuwenbo@phytium.com.cn> 2963* Rob Bradford <robert.bradford@intel.com> 2964* Samuel Ortiz <sameo@linux.intel.com> 2965* Sebastien Boeuf <sebastien.boeuf@intel.com> 2966* Sergio Lopez <slp@redhat.com> 2967 2968# v0.5.1 2969 2970This is a bugfix release branched off v0.5.0. It contains the following fixes: 2971 2972* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez 2973* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf 2974* Fixed error reporting for resize command - Samuel Ortiz 2975* Double reboot workaround (#783) - Rob Bradford 2976* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford 2977 2978# v0.5.0 2979 2980Highlights for `cloud-hypervisor` version 0.5.0 include: 2981 2982### Virtual Machine Dynamic Resizing 2983 2984With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and 2985memory hot plug as well. This allows to dynamically resize Cloud Hypervisor 2986guests which is needed for e.g. Kubernetes related use cases. 2987The memory hot plug implementation is based on the same framework as the CPU hot 2988plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest. 2989 2990Next on our VM resizing roadmap is the PCI devices hotplug feature. 2991 2992### Multi-Queue, Multi-Threaded Paravirtualization 2993 2994We enhanced our virtio networking and block support by having both devices use 2995multiple I/O queues handled by multiple threads. This improves our default 2996paravirtualized networking and block devices throughput. 2997 2998### New Interrupt Management Framework 2999 3000We improved our interrupt management implementation by introducing an Interrupt 3001Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device) 3002crates discussions. This move made the code significantly cleaner, and allowed 3003us to remove several KVM related dependencies from crates like the PCI and 3004virtio ones. 3005 3006### Development Tools 3007 3008In order to provide a better developer experience, we worked on improving our 3009build, development and testing tools. 3010Somehow similar to the excellent 3011[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool), 3012we now provide a [dev_cli script](scripts/dev_cli.sh). 3013 3014With this new tool, our users and contributors will be able to build and test 3015Cloud Hypervisor through a containerized environment. 3016 3017### Kata Containers Integration 3018 3019We spent some significant time and efforts debugging and fixing our integration 3020with the [Kata Containers](https://github.com/kata-containers) project. Cloud 3021Hypervisor is now a fully supported Kata Containers hypervisor, and is 3022integrated into the project's CI. 3023 3024### Contributors 3025 3026Many thanks to everyone that contributed to the 0.5.0 release: 3027 3028* Bo Chen <chen.bo@intel.com> 3029* Cathy Zhang <cathy.zhang@intel.com> 3030* Qiu Wenbo <qiuwenbo@phytium.com.cn> 3031* Rob Bradford <robert.bradford@intel.com> 3032* Samuel Ortiz <sameo@linux.intel.com> 3033* Sebastien Boeuf <sebastien.boeuf@intel.com> 3034* Sergio Lopez <slp@redhat.com> 3035* Yang Zhong <yang.zhong@intel.com> 3036 3037# v0.4.0 3038 3039Highlights for `cloud-hypervisor` version 0.4.0 include: 3040 3041### Dynamic virtual CPUs addition 3042 3043As a way to vertically scale Cloud Hypervisor guests, we now support dynamically 3044adding virtual CPUs to the guests, a mechanism also known as CPU hot plug. 3045Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs 3046to an already running guest and the high level operations for that process are 3047documented [here](docs/hotplug.md) 3048 3049During the next release cycles we are planning to extend Cloud Hypervisor 3050hot plug framework to other resources, namely PCI devices and memory. 3051 3052### Programmatic firmware tables generation 3053 3054As part of the CPU hot plug feature enablement, and as a requirement for hot 3055plugging other resources like devices or RAM, we added support for 3056programmatically generating the needed ACPI tables. Through a dedicated 3057`acpi-tables` crate, we now have a flexible and clean way of generating those 3058tables based on the VMM device model and topology. 3059 3060### Filesystem and block devices vhost-user backends 3061 3062Our objective of running all Cloud Hypervisor paravirtualized I/O to a 3063vhost-user based framework is getting closer as we've added Rust based 3064implementations for vhost-user-blk and virtiofs backends. Together with the 3065vhost-user-net backend that came with the 0.3.0 release, this will form the 3066default Cloud Hypervisor I/O architecture. 3067 3068### Guest pause and resume 3069 3070As an initial requirement for enabling live migration, we added support for 3071pausing and resuming any VMM components. As an intermediate step towards live 3072migration, the upcoming guest snapshotting feature will be based on the pause 3073and resume capabilities. 3074 3075### Userspace IOAPIC by default 3076 3077As a way to simplify our device manager implementation, but also in order to 3078stay away from privileged rings as often as possible, any device that relies on 3079pin based interrupts will be using the userspace IOAPIC implementation by 3080default. 3081 3082### PCI BAR reprogramming 3083 3084In order to allow for a more flexible device model, and also support guests 3085that would want to move PCI devices, we added support for PCI devices BAR 3086reprogramming. 3087 3088### New `cloud-hypervisor` organization 3089 3090As we wanted to be more flexible on how we manage the Cloud Hypervisor project, 3091we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor). 3092Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) 3093project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux) 3094and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 3095repositories. We may also use it to host any rust-vmm that we'd need to 3096temporarily fork. 3097Thanks to GitHub's seamless repository redirections, the move is completely 3098transparent to all Cloud Hypervisor contributors, users and followers. 3099 3100### Contributors 3101 3102Many thanks to everyone that contributed to the 0.4.0 release: 3103 3104* Cathy Zhang <cathy.zhang@intel.com> 3105* Emin Ghuliev <drmint80@gmail.com> 3106* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 3107* Qiu Wenbo <qiuwenbo@phytium.com.cn> 3108* Rob Bradford <robert.bradford@intel.com> 3109* Samuel Ortiz <sameo@linux.intel.com> 3110* Sebastien Boeuf <sebastien.boeuf@intel.com> 3111* Sergio Lopez <slp@redhat.com> 3112* Wu Zongyong <wuzongyong@linux.alibaba.com> 3113 3114# v0.3.0 3115 3116Highlights for `cloud-hypervisor` version 0.3.0 include: 3117 3118### Block device offloading 3119 3120We continue to work on offloading paravirtualized I/O to external processes, 3121and we added support for 3122[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends. 3123This enables `cloud-hypervisor` users to plug a `vhost-user` based block device 3124like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage 3125backend. 3126 3127### Network device backend 3128 3129The previous release provided support for 3130[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we 3131also provide a TAP based vhost-user-net backend, implemented in Rust. Together 3132with the vhost-user-net device implementation, this will eventually become the 3133Cloud Hypervisor default paravirtualized networking architecture. 3134 3135### Virtual sockets 3136 3137In order to more efficiently and securely communicate between host and guest, 3138we added an hybrid implementation of the 3139[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address 3140family over virtio. Credits go to the 3141[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md) 3142project as our implementation is a copy of theirs. 3143 3144### HTTP based API 3145 3146In anticipation of the need to support asynchronous operations to Cloud 3147Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP 3148based API to the VMM. The API will be more extensively documented during the 3149next release cycle. 3150 3151### Memory mapped virtio transport 3152 3153In order to support potential PCI-free use cases, we added support for the 3154[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002) 3155transport layer. This will allow us to support simple, minimal guest 3156configurations that do not require a PCI bus emulation. 3157 3158### Paravirtualized IOMMU 3159 3160As we want to improve our nested guests support, we added support for exposing 3161a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows 3162for a safer nested virtio and directly assigned devices support. 3163 3164To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor 3165users to be able to specify if devices had to be handled through this virtual 3166IOMMU or not. In particular, the `--disk` option now expects disk paths to be 3167prefixed with a `path=` string, and supports an optional `iommu=[on|off]` 3168setting. 3169 3170### Ubuntu 19.10 3171 3172With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware), 3173we can now support the latest 3174[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images. 3175 3176### Large memory guests 3177 3178After simplifying and changing our guest address space handling, we can now 3179support guests with large amount of memory (more than 64GB). 3180 3181# v0.2.0 3182 3183Highlights for `cloud-hypervisor` version 0.2.0 include: 3184 3185### Network device offloading 3186 3187As part of our general effort to offload paravirtualized I/O to external 3188processes, we added support for 3189[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This 3190enables `cloud-hypervisor` users to plug a `vhost-user` based networking device 3191(e.g. [DPDK](https://www.dpdk.org)) into the VMM as their virtio network backend. 3192 3193### Minimal hardware-reduced ACPI 3194 3195In order to properly implement and guest reset and shutdown, we implemented 3196a minimal version of the hardware-reduced ACPI specification. Together with 3197a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to 3198cleanly reboot and shutdown. 3199 3200The ACPI implementation is a `cloud-hypervisor` build time option that is 3201enabled by default. 3202 3203### Debug I/O port 3204 3205Based on the Firecracker idea of using a dedicated I/O port to measure guest 3206boot times, we added support for logging guest events through the 3207[0x80](https://web.archive.org/web/20211028033025/https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html) 3208PC debug port. This allows, among other things, for granular guest boot time 3209measurements. See our [debug port documentation](docs/debug-port.md) for more 3210details. 3211 3212### Improved direct device assignment 3213 3214We fixed a major performance issue with our initial VFIO implementation: When 3215enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads 3216were (in many cases) not cached. After correctly tagging the guest memory from 3217`cloud-hypervisor` we're now able to reach the expected performance from 3218directly assigned devices. 3219 3220### Improved shared filesystem 3221 3222We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt) 3223support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system. 3224This provides better shared filesystem IO performance with a smaller guest 3225memory footprint. 3226 3227### Ubuntu bionic based CI 3228 3229Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 3230improvements, we are now able to boot Ubuntu bionic images. We added those to 3231our CI pipeline. 3232 3233# v0.1.0 3234 3235Highlights for `cloud-hypervisor` version 0.1.0 include: 3236 3237### Shared filesystem 3238 3239We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file 3240system, allowing for an efficient and reliable way of sharing a filesystem 3241between the host and the `cloud-hypervisor` guest. 3242 3243See our [filesystem sharing](docs/fs.md) documentation for more details on how 3244to use virtio-fs with `cloud-hypervisor`. 3245 3246### Initial direct device assignment support 3247 3248VFIO (Virtual Function I/O) is a kernel framework that exposes direct device 3249access to userspace. `cloud-hypervisor` uses VFIO to directly assign host 3250physical devices into its guest. 3251 3252See our [VFIO](docs/vfio.md) documentation for more detail on how to directly 3253assign host devices to `cloud-hypervisor` guests. 3254 3255### Userspace IOAPIC 3256 3257`cloud-hypervisor` supports a so-called split IRQ chip implementation by 3258implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC). 3259By moving part of the IRQ chip implementation from kernel space to user space, 3260the IRQ chip emulation does not always run in a fully privileged mode. 3261 3262### Virtual persistent memory 3263 3264The `virtio-pmem` implementation emulates a virtual persistent memory device 3265that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device 3266allows to bypass the guest page cache and improve the guest memory footprint. 3267 3268### Linux kernel bzImage 3269 3270The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from 3271`bzImage` kernel images, which is usually the format that Linux distributions 3272use to ship their kernels. For example, this allows for booting from the host 3273distribution kernel image. 3274 3275### Console over virtio 3276 3277`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although 3278using this device as a guest console can potentially cut some early boot 3279messages, it can reduce the guest boot time and provides a complete console 3280implementation. 3281 3282The `virtio-console` device is enabled by default for the guest console. 3283Switching back to the legacy serial port is done by selecting 3284`--serial tty --console off` from the command line. 3285 3286### Unit testing 3287 3288We now run all unit tests from all our crates directly from our CI. 3289 3290### Integration tests parallelization 3291 3292The CI cycle run time has been significantly reduced by refactoring our 3293integration tests; allowing them to all be run in parallel. 3294