1- [v17.0](#v170) 2 - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi) 3 - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend) 4 - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices) 5 - [Improved SGX support](#improved-sgx-support) 6 - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices) 7 - [Notable bug fixes](#notable-bug-fixes) 8 - [Contributors](#contributors) 9- [v16.0](#v160) 10 - [Improved live migration support](#improved-live-migration-support) 11 - [Improved `vhost-user` support](#improved-vhost-user-support) 12 - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support) 13 - [Notable bug fixes](#notable-bug-fixes-1) 14 - [Removed functionality](#removed-functionality) 15 - [Contributors](#contributors-1) 16- [v15.0](#v150) 17 - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees) 18 - [Network device rate limiting](#network-device-rate-limiting) 19 - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload) 20 - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter) 21 - [Bug fixes](#bug-fixes) 22 - [Deprecations](#deprecations) 23 - [Contributors](#contributors-2) 24- [v0.14.1](#v0141) 25- [v0.14.0](#v0140) 26 - [Structured event monitoring](#structured-event-monitoring) 27 - [MSHV improvements](#mshv-improvements) 28 - [Improved aarch64 platform](#improved-aarch64-platform) 29 - [Updated hotplug documentation](#updated-hotplug-documentation) 30 - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console) 31 - [Block device rate limiting](#block-device-rate-limiting) 32 - [Deprecations](#deprecations-1) 33 - [Contributors](#contributors-3) 34- [v0.13.0](#v0130) 35 - [Wider VFIO device support](#wider-vfio-device-support) 36 - [Improved huge page support](#improved-huge-page-support) 37 - [MACvTAP support](#macvtap-support) 38 - [VHD disk image support](#vhd-disk-image-support) 39 - [Improved Virtio device threading](#improved-virtio-device-threading) 40 - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button) 41 - [Contributors](#contributors-4) 42- [v0.12.0](#v0120) 43 - [ARM64 enhancements](#arm64-enhancements) 44 - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning) 45 - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend) 46 - [Enhanced "info" API](#enhanced-info-api) 47 - [Contributors](#contributors-5) 48- [v0.11.0](#v0110) 49 - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block) 50 - [Windows Guest Support](#windows-guest-support) 51 - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation) 52 - [`virtio-mmmio` Removal](#virtio-mmmio-removal) 53 - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64) 54 - [Improved Linux Boot Time](#improved-linux-boot-time) 55 - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling) 56 - [Default Log Level Changed](#default-log-level-changed) 57 - [New `--balloon` Parameter Added](#new---balloon-parameter-added) 58 - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support) 59 - [Notable Bug Fixes](#notable-bug-fixes-2) 60 - [Contributors](#contributors-6) 61- [v0.10.0](#v0100) 62 - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors) 63 - [Memory Zones](#memory-zones) 64 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements) 65 - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control) 66 - [Notable Bug Fixes](#notable-bug-fixes-3) 67 - [Contributors](#contributors-7) 68- [v0.9.0](#v090) 69 - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support) 70 - [Block and Network Device Statistics](#block-and-network-device-statistics) 71 - [HTTP API Responses](#http-api-responses) 72 - [CPU Topology](#cpu-topology) 73 - [Release Build Optimization](#release-build-optimization) 74 - [Hypervisor Abstraction](#hypervisor-abstraction) 75 - [Snapshot/Restore Improvements](#snapshotrestore-improvements) 76 - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support) 77 - [Enhancements to ARM64 Support](#enhancements-to-arm64-support) 78 - [Intel SGX Support](#intel-sgx-support) 79 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1) 80 - [Notable Bug Fixes](#notable-bug-fixes-4) 81 - [Contributors](#contributors-8) 82- [v0.8.0](#v080) 83 - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support) 84 - [Experimental ARM64 Support](#experimental-arm64-support) 85 - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests) 86 - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices) 87 - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements) 88 - [Notable Bug Fixes](#notable-bug-fixes-5) 89 - [Command Line and API Changes](#command-line-and-api-changes) 90 - [Contributors](#contributors-9) 91- [v0.7.0](#v070) 92 - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug) 93 - [Alternative `libc` Support](#alternative-libc-support) 94 - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends) 95 - [Initial RamFS Support](#initial-ramfs-support) 96 - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem) 97 - [`Seccomp` Sandboxing](#seccomp-sandboxing) 98 - [Updated Distribution Support](#updated-distribution-support) 99 - [Command Line and API Changes](#command-line-and-api-changes-1) 100 - [Contributors](#contributors-10) 101- [v0.6.0](#v060) 102 - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug) 103 - [Shared Filesystem Improvements](#shared-filesystem-improvements) 104 - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading) 105 - [Command Line Interface](#command-line-interface) 106 - [PVH Boot](#pvh-boot) 107 - [Contributors](#contributors-11) 108- [v0.5.1](#v051) 109- [v0.5.0](#v050) 110 - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing) 111 - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization) 112 - [New Interrupt Management Framework](#new-interrupt-management-framework) 113 - [Development Tools](#development-tools) 114 - [Kata Containers Integration](#kata-containers-integration) 115 - [Contributors](#contributors-12) 116- [v0.4.0](#v040) 117 - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition) 118 - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation) 119 - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends) 120 - [Guest pause and resume](#guest-pause-and-resume) 121 - [Userspace IOAPIC by default](#userspace-ioapic-by-default) 122 - [PCI BAR reprogramming](#pci-bar-reprogramming) 123 - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization) 124 - [Contributors](#contributors-13) 125- [v0.3.0](#v030) 126 - [Block device offloading](#block-device-offloading) 127 - [Network device backend](#network-device-backend) 128 - [Virtual sockets](#virtual-sockets) 129 - [HTTP based API](#http-based-api) 130 - [Memory mapped virtio transport](#memory-mapped-virtio-transport) 131 - [Paravirtualized IOMMU](#paravirtualized-iommu) 132 - [Ubuntu 19.10](#ubuntu-1910) 133 - [Large memory guests](#large-memory-guests) 134- [v0.2.0](#v020) 135 - [Network device offloading](#network-device-offloading) 136 - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi) 137 - [Debug I/O port](#debug-io-port) 138 - [Improved direct device assignment](#improved-direct-device-assignment) 139 - [Improved shared filesystem](#improved-shared-filesystem) 140 - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci) 141- [v0.1.0](#v010) 142 - [Shared filesystem](#shared-filesystem) 143 - [Initial direct device assignment support](#initial-direct-device-assignment-support) 144 - [Userspace IOAPIC](#userspace-ioapic) 145 - [Virtual persistent memory](#virtual-persistent-memory) 146 - [Linux kernel bzImage](#linux-kernel-bzimage) 147 - [Console over virtio](#console-over-virtio) 148 - [Unit testing](#unit-testing) 149 - [Integration tests parallelization](#integration-tests-parallelization) 150 151 152# v17.0 153 154This release has been tracked through the [v17.0 155project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/20). 156 157### ARM64 NUMA support using ACPI 158 159The support for ACPI on ARM64 has been enhanced to include support for 160specifying a NUMA configuration using the existing control options. 161 162### `Seccomp` support for MSHV backend 163 164The `seccomp` rules have now been extended to support running against the MSHV 165hypervisor backend. 166 167### Hotplug of `macvtap` devices 168 169Hotplug of `macvtap` devices is now supported with the file descriptor for the 170network device if opened by the user and passed to the VMM. The `ch-remote` 171tool supports this functionality when adding a network device. 172 173### Improved SGX support 174 175The SGX support has been updated to match the latest Linux kernel support and 176now supports SGX provisioning and associating EPC sections to NUMA nodes. 177 178### Inflight tracking for `vhost-user` devices 179 180Support for handling inflight tracking of I/O requests has been added to the 181`vhost-user` devices allowing recovery after device reconnection. 182 183### Notable bug fixes 184 185* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821). 186* The VMM side of `vhost-user` devices no longer advertise the 187 `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM 188(#2833). 189* On ARM64 VMs can be created with more than 16 vCPUs (#2763). 190 191### Contributors 192 193Many thanks to everyone who has contributed to our release: 194 195* Anatol Belski <anbelski@linux.microsoft.com> 196* Arafatms <arafatms@outlook.com> 197* Bo Chen <chen.bo@intel.com> 198* Fei Li <lifei.shirley@bytedance.com> 199* Henry Wang <Henry.Wang@arm.com> 200* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 201* Jianyong Wu <jianyong.wu@arm.com> 202* Li Hangjing <lihangjing@bytedance.com> 203* Michael Zhao <michael.zhao@arm.com> 204* Muminul Islam <muislam@microsoft.com> 205* Rob Bradford <robert.bradford@intel.com> 206* Sebastien Boeuf <sebastien.boeuf@intel.com> 207* Wei Liu <liuwe@microsoft.com> 208* Yukiteru <wfly1998@sina.com> 209 210# v16.0 211 212This release has been tracked through the [v16.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/19). 213 214### Improved live migration support 215 216The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details. 217 218### Improved `vhost-user` support 219 220When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch. 221 222### ARM64 ACPI and UEFI support 223 224Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot. 225 226### Notable bug fixes 227 228* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578). 229* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563). 230* The KVM clock is now correctly handled during live migration / snapshot & restore. 231 232### Removed functionality 233 234The following formerly deprecated features have been removed: 235 236* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 237 binaries has been deprecated. When using direct boot users should configure 238 their kernel with `CONFIG_PVH=y`. 239 240### Contributors 241 242Many thanks to everyone who has contributed to our release including some new faces. 243 244* Anatol Belski <anbelski@linux.microsoft.com> 245* Bo Chen <chen.bo@intel.com> 246* Dayu Liu <liu.dayu@zte.com.cn> 247* Henry Wang <Henry.Wang@arm.com> 248* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 249* Jianyong Wu <jianyong.wu@arm.com> 250* Michael Zhao <michael.zhao@arm.com> 251* Mikko Ylinen <mikko.ylinen@intel.com> 252* Muminul Islam <muislam@microsoft.com> 253* Ren Lei <ren.lei4@zte.com.cn> 254* Rob Bradford <robert.bradford@intel.com> 255* Sebastien Boeuf <sebastien.boeuf@intel.com> 256* Wei Liu <liuwe@microsoft.com> 257* Yi Wang <wang.yi59@zte.com.cn> 258 259# v15.0 260 261This release has been tracked through the [v15.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/18). 262 263Highlights for `cloud-hypervisor` version v15.0 include: 264 265### Version numbering and stability guarantees 266 267This release is the first in a new version numbering scheme to represent that 268we believe Cloud Hypervisor is maturing and entering a period of stability. 269With this new release we are beginning our new stability guarantees: 270 271* The API (including command line options) will not be removed or changed in a 272 breaking way without a minimum of 2 releases notice. Where possible warnings 273 will be given about the use of deprecated functionality and the deprecations 274 will be documented in the release notes. 275* Point releases will be made between individual releases where there are 276 substantial bug fixes or security issues that need to be fixed. 277 278Currently the following items are **not** guaranteed across updates: 279 280* Snapshot/restore is not supported across different versions 281* Live migration is not supported across different versions 282* The following features are considered experimental and may change 283 substantially between releases: TDX, SGX. 284 285### Network device rate limiting 286 287Building on our existing support for rate limiting block activity the network 288device also now supports rate limiting. Full details of the controls are in the 289[IO throttling documentation.](docs/io_throttling.md) 290 291### Support for runtime control of `virtio-net` guest offload 292 293The guest is now able to change the offload settings for the `virtio-net` 294device. As well as providing a useful control this mitigates an issue in the 295Linux kernel where the guest will attempt to reprogram the offload settings 296even if they are not advertised as configurable (#2528). 297 298### `--api-socket` supports file descriptor parameter 299 300The `--api-socket` can now take an `fd=` parameter to specify an existing file 301descriptor to use. This is particularly beneficial for frameworks that need to 302programmatically control Cloud Hypervisor. 303 304### Bug fixes 305 306* A workaround has been put in place to mitigate a Linux kernel issues that 307 results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277). 308* PCI BARs are now correctly aligned removing the need for the guest to 309 reprogram them (#1797,#1798) 310* Handle TAP interface not being writable within virtio-net (due to the buffer 311 exhaustion on the host) (#2517) 312* The recommended Linux kernel is now v5.12.0 as it contains a fix that 313 prevents snapshot & restore working (#2535) 314 315### Deprecations 316 317Deprecated features will be removed in a subsequent release and users should plan to use alternatives 318 319* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 320 binaries has been deprecated. When using direct boot users should configure 321 their kernel with `CONFIG_PVH=y`. Will be removed in v16.0. 322 323### Contributors 324 325Many thanks to everyone who has contributed to our release including some new faces. 326 327* Alyssa Ross <hi@alyssa.is> 328* Anatol Belski <anbelski@linux.microsoft.com> 329* Bo Chen <chen.bo@intel.com> 330* Gaelan Steele <gbs@canishe.com> 331* Jianyong Wu <jianyong.wu@arm.com> 332* Michael Zhao <michael.zhao@arm.com> 333* Muminul Islam <muislam@microsoft.com> 334* Rob Bradford <robert.bradford@intel.com> 335* Sebastien Boeuf <sebastien.boeuf@intel.com> 336* Wei Liu <liuwe@microsoft.com> 337* William Douglas <william.douglas@intel.com> 338 339# v0.14.1 340 341Bug fix release branched off the v0.14.0 release. The following bugs were fixed 342in this release: 343 344* CPU hotplug on Windows failed due to misreported CPU state information and 345 the lack of HyperV CPUID bit enabled (#2437, #2449, #2436) 346* A seccomp rule was missing that was triggered on CPU unplug (#2455) 347* A bounds check in VIRTIO queue validation was erroneously generating 348 DescriptorChainTooShort errors in certain circumstances (#2450, #2424) 349 350# v0.14.0 351 352This release has been tracked through the [0.14.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/17). 353 354Highlights for `cloud-hypervisor` version 0.14.0 include: 355 356### Structured event monitoring 357 358A new option was added to the VMM `--event-monitor` which reports structured 359events (JSON) over a file or file descriptor at key events in the lifecycle of 360the VM. The list of events is limited at the moment but will be further 361extended over subsequent releases. The events exposed form part of the Cloud 362Hypervisor API surface. 363 364### MSHV improvements 365 366Basic support has been added for running Windows guests atop the MSHV 367hypervisor as an alternative to KVM and further improvements have been made to 368the MSHV support. 369 370### Improved aarch64 platform 371 372The aarch64 platform has been enhanced with more devices exposed to the running 373VM including an enhanced serial UART. 374 375### Updated hotplug documentation 376 377The documentation for the hotplug support has been updated to reflect the use 378of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as 379well as documenting hotplug of paravirtualised and VFIO devices. 380 381### PTY control for serial and `virtio-console` 382 383The `--serial` and `--console` parameters can now direct the console to a PTY 384allowing programmatic control of the console from another process through the 385PTY subsystem. 386 387### Block device rate limiting 388 389The block device performance can now be constrained as part of the VM 390configuration allowing rate limiting. Full details of the controls are in the 391[IO throttling documentation.](docs/io_throttling.md) 392 393 394### Deprecations 395 396Deprecated features will be removed in a subsequent release and users should plan to use alternatives 397 398* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 399 binaries has been deprecated. When using direct boot users should configure 400 their kernel with `CONFIG_PVH=y`. 401 402 403### Contributors 404 405Many thanks to everyone who has contributed to our 0.14.0 release including 406some new faces. 407 408Bo Chen <chen.bo@intel.com> 409Henry Wang <Henry.Wang@arm.com> 410Iggy Jackson <iggy@theiggy.com> 411Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 412Michael Zhao <michael.zhao@arm.com> 413Muminul Islam <muislam@microsoft.com> 414Penny Zheng <Penny.Zheng@arm.com> 415Rob Bradford <robert.bradford@intel.com> 416Sebastien Boeuf <sebastien.boeuf@intel.com> 417Vineeth Pillai <viremana@linux.microsoft.com> 418Wei Liu <liuwe@microsoft.com> 419William Douglas <william.r.douglas@gmail.com> 420Zide Chen <zide.chen@intel.com> 421 422# v0.13.0 423 424This release has been tracked through the [0.13.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/16). 425 426Highlights for `cloud-hypervisor` version 0.13.0 include: 427 428### Wider VFIO device support 429 430It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI 431devices that do not support MSI or MSI-X and instead rely on INTx interrupts. 432Most notably this widens the support to most NVIDIA cards with the proprietary 433drivers. 434 435### Improved huge page support 436 437Through the addition of `hugepage_size` on `--memory` it is now possible to 438specify the desired size of the huge pages used when allocating the guest 439memory. The user is required to ensure they have sufficient pages of the 440desired size in their pool. 441 442### MACvTAP support 443 444It is now possible to provide file descriptors using the `fd` parameter to 445`--net` which point at TAP devices that have already been opened by the user. 446This aids integration with `libvirt` but also permits the use of MACvTAP 447support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md). 448 449### VHD disk image support 450 451It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw 452disk image with Cloud Hypervisor. 453 454### Improved Virtio device threading 455 456Device threads are now derived from the main VMM thread which allows more 457restrictive seccomp filters to be applied to them. The threads also have a 458predictable name derived from the device id. 459 460### Clean shutdown support via synthetic power button 461 462It is now possible to request that the guest VM shut itself down by triggering 463a synthetic ACPI power button press from the VMM. If the guest is listening for 464such an event (e.g. using systemd) then it will process the event and cleanly 465shut down. This functionality is exposed through the HTTP API and can be 466triggered via `ch-remote --api-socket=<API socket> power-button`. 467 468### Contributors 469 470Many thanks to everyone who has contributed to our 0.13.0 release including 471some new faces. 472 473* Bo Chen <chen.bo@intel.com> 474* Mikko Ylinen <mikko.ylinen@intel.com> 475* Muminul Islam <muislam@microsoft.com> 476* Rob Bradford <robert.bradford@intel.com> 477* Samuel Ortiz <sameo@linux.intel.com> 478* Sebastien Boeuf <sebastien.boeuf@intel.com> 479* Vineeth Pillai <viremana@linux.microsoft.com> 480* Wei Liu <liuwe@microsoft.com> 481* William Douglas <william.r.douglas@gmail.com> 482* Xie Yongji <xieyongji@bytedance.com> 483 484# v0.12.0 485 486This release has been tracked through the [0.12.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/15). 487 488Highlights for `cloud-hypervisor` version 0.12.0 include: 489 490### ARM64 enhancements 491 492The use of `--watchdog` is now fully supported as is the ability to reboot the 493VM from within the guest when running Cloud Hypervisor on an ARM64 system. 494 495### Removal of `vhost-user-net` and `vhost-user-block` self spawning 496 497In order to use `vhost-user-net` or `vhost-user-block` backends the user is now 498responsible for starting the backend and providing the socket for the VMM to 499use. This functionality was deprecated in the last release and how now been 500removed. 501 502### Migration of `vhost-user-fs` backend 503 504The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is 505instead hosted in [it's own 506repository](https://gitlab.com/virtio-fs/virtiofsd-rs) 507 508### Enhanced "info" API 509 510The `vm.info` HTTP API endpoint has been extended to include the details of the 511devices used by the VM including any VFIO devices used. 512 513### Contributors 514 515Many thanks to everyone who has contributed to our 0.12.0 release: 516 517* Anatol Belski <anbelski@linux.microsoft.com> 518* Julio Montes <julio.montes@intel.com> 519* Michael Zhao <michael.zhao@arm.com> 520* Muminul Islam <muislam@microsoft.com> 521* Rob Bradford <robert.bradford@intel.com> 522* Samuel Ortiz <sameo@linux.intel.com> 523* Wei Liu <liuwe@microsoft.com> 524 525# v0.11.0 526 527This release has been tracked through the [0.11.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/14). 528 529Highlights for `cloud-hypervisor` version 0.11.0 include: 530 531### `io_uring` support by default for `virtio-block` 532 533Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will 534be used for a significantly higher performance block device. 535 536### Windows Guest Support 537 538This is the first release where we officially support Windows running as a 539guest. Full details of how to setup the image and run Cloud Hypervisor with a 540Windows guest can be found in the dedicated [Windows 541documentation](docs/windows.md). 542 543### `vhost-user` "Self Spawning" Deprecation 544 545Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now 546deprecated. Users of this functionality will receive a warning and should make 547adjustments. The functionality will be removed in the next release. 548 549### `virtio-mmmio` Removal 550 551Support for using the `virtio-mmio` transport, rather than using PCI, has been 552removed. This has been to simplify the code and significantly 553reduce the testing burden of the project. 554 555### Snapshot/Restore support for ARM64 556 557When running on the ARM64 architecture snapshot and restore has now been 558implemented. 559 560### Improved Linux Boot Time 561 562The time to boot the Linux kernel has been significantly improved by the 563identifying some areas of delays around PCI bus probing, IOAPIC programming and 564MPTABLE issues. Full details can be seen in #1728. 565 566### `SIGTERM/SIGINT` Interrupt Signal Handling 567 568When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will 569trigger the VMM process to cleanly deallocate resources before exiting. The 570guest VM will not be cleanly shutdown but the VMM process will clean up its 571resources. 572 573### Default Log Level Changed 574 575The default logging level was changed to include warnings which should make it 576easier to see potential issues. New [logging 577documentation](docs/logging) was also added. 578 579### New `--balloon` Parameter Added 580 581Control of the setup of `virtio-balloon` has been moved from `--memory` to its 582own dedicated parameter. This makes it easier to add more balloon specific 583controls without overloading `--memory`. 584 585### Experimental `virtio-watchdog` Support 586 587Support for using a new `virtio-watchdog` has been added which can be used to 588have the VMM reboot the guest if the guest userspace fails to ping the 589watchdog. This is enabled with `--watchdog` and requires kernel support. 590 591### Notable Bug Fixes 592 593* MTRR bit was missing from CPUID advertised to guest 594* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170) 595* CPU identification string is now exposed to the guest 596* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as 597 read only so avoids excessive VM exits (#1795) 598* PCI device hotplug after an unplug was fixed (#1802) 599* When using the ACPI method to resize the guest memory the full reserved size 600 can be used (#1803) 601* Snapshot and restore followed by a second snapshot and restore now works 602 correctly 603* Snapshot and restore of VMs with more than 2GiB in one region now work 604 correctly 605 606### Contributors 607 608Many thanks to everyone who has contributed to our 0.11.0 release including some new faces. 609 610* Anatol Belski <anbelski@linux.microsoft.com> 611* Bo Chen <chen.bo@intel.com> 612* Daniel Verkamp <dverkamp@chromium.org> 613* Henry Wang <Henry.Wang@arm.com> 614* Hui Zhu <teawater@antfin.com> 615* Jiangbo Wu <jiangbo.wu@intel.com> 616* Josh Soref <jsoref@users.noreply.github.com> 617* Julio Montes <julio.montes@intel.com> 618* Michael Zhao <michael.zhao@arm.com> 619* Muminul Islam <muislam@microsoft.com> 620* pierwill <19642016+pierwill@users.noreply.github.com> 621* Praveen Paladugu <prapal@microsoft.com> 622* Rob Bradford <robert.bradford@intel.com> 623* Sebastien Boeuf <sebastien.boeuf@intel.com> 624* Wei Liu <liuwe@microsoft.com> 625 626 627# v0.10.0 628 629This release has been tracked through the [0.10.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/13). 630 631Highlights for `cloud-hypervisor` version 0.10.0 include: 632 633### `virtio-block` Support for Multiple Descriptors 634 635Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers. 636 637### Memory Zones 638 639Support has been added for fine grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md). 640 641### `Seccomp` Sandbox Improvements 642 643All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`. 644 645### Preliminary KVM HyperV Emulation Control 646 647A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices. 648 649### Notable Bug Fixes 650 651- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596) 652- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564) 653- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593) 654- `--version` shows the version for released binaries (#1669) 655- Errors generated by worker threads for `virtio` devices are now printed out (#1551) 656 657### Contributors 658 659Many thanks to everyone who has contributed to our 0.10.0 release including some new faces. 660 661* Alyssa Ross <hi@alyssa.is> 662* Amey Narkhede <ameynarkhede02@gmail.com> 663* Anatol Belski <ab@php.net> 664* Bo Chen <chen.bo@intel.com> 665* Hui Zhu <teawater@antfin.com> 666* Michael Zhao <michael.zhao@arm.com> 667* Muminul Islam <muislam@microsoft.com> 668* Rob Bradford <robert.bradford@intel.com> 669* Samuel Ortiz <sameo@linux.intel.com> 670* Sebastien Boeuf <sebastien.boeuf@intel.com> 671* Wei Liu <liuwe@microsoft.com> 672 673# v0.9.0 674 675This release has been tracked through the [0.9.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/12). 676 677Highlights for `cloud-hypervisor` version 0.9.0 include: 678 679### `io_uring` Based Block Device Support 680 681If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement. 682 683### Block and Network Device Statistics 684 685Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM. 686 687### HTTP API Responses 688 689The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF. 690 691### CPU Topology 692 693A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core. 694 695### Release Build Optimization 696 697Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size. 698 699### Hypervisor Abstraction 700 701A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`. 702 703### Snapshot/Restore Improvements 704 705Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs. 706 707### Virtio Memory Ballooning Support 708 709A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest. 710 711### Enhancements to ARM64 Support 712 713The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting. 714 715### Intel SGX Support 716 717The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md). 718 719### `Seccomp` Sandbox Improvements 720 721The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development. 722 723### Notable Bug Fixes 724 725* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes. 726* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338) 727* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543). 728* `virtio-mmio` based devices are now more widely tested (#275). 729* Multiple issues have been fixed with virtio device configuration (#1217) 730* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521) 731 732### Contributors 733 734Many thanks to everyone who has contributed to our 0.9.0 release including some new faces. 735 736* Anatol Belski <ab@php.net> 737* Bo Chen <chen.bo@intel.com> 738* Dr. David Alan Gilbert <dgilbert@redhat.com> 739* Henry Wang <Henry.Wang@arm.com> 740* Howard Zhang <howard.zhang@arm.com> 741* Hui Zhu <teawater@antfin.com> 742* Jianyong Wu <jianyong.wu@arm.com> 743* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 744* LiYa'nan <oliverliyn@gmail.com> 745* Michael Zhao <michael.zhao@arm.com> 746* Muminul Islam <muislam@microsoft.com> 747* Praveen Paladugu <prapal@microsoft.com> 748* Ricardo Koller <ricarkol@gmail.com> 749* Rob Bradford <robert.bradford@intel.com> 750* Samuel Ortiz <sameo@linux.intel.com> 751* Sebastien Boeuf <sebastien.boeuf@intel.com> 752* Stefano Garzarella <sgarzare@redhat.com> 753* Wei Liu <liuwe@microsoft.com> 754 755 756# v0.8.0 757 758This release has been tracked through the [0.8.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/10). 759 760Highlights for `cloud-hypervisor` version 0.8.0 include: 761 762### Experimental Snapshot and Restore Support 763 764This release includes the first version of the snapshot and restore feature. 765This allows a VM to be paused and then subsequently snapshotted. At a later 766point that snapshot may be restored into a new running VM identical to the 767original VM at the point it was paused. 768 769This feature can be used for offline migration from one VM host to another, to 770allow the upgrading or rebooting of the host machine transparently to the guest 771or for templating the VM. This is an experimental feature and cannot be used on 772a VM using passthrough (VFIO) devices. Issues with SMP have also been observed 773(#1176). 774 775### Experimental ARM64 Support 776 777Included in this release is experimental support for running on ARM64. 778Currently only `virtio-mmio` devices and a serial port are supported. Full 779details can be found in the [ARM64 documentation](docs/arm64.md). 780 781### Support for Using 5-level Paging in Guests 782 783If the host supports it the guest is now enabled for 5-level paging (aka LA57). 784This works when booting the Linux kernel with a vmlinux, bzImage or firmware 785based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not 786work due to current limitations in the PVH boot process. 787 788### Virtio Device Interrupt Suppression for Network Devices 789 790With `virtio-net` and `vhost-user-net` devices the guest can suppress 791interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This 792can lead to an improvement in performance by reducing the number of interrupts 793the guest must service. 794 795### `vhost_user_fs` Improvements 796 797The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`. 798 799 800### Notable Bug Fixes 801 802* VMs that have not yet been booted can now be deleted (#1110). 803* By creating the `tap` device ahead of creating the VM it is not required to 804 run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273). 805* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to 806 the specification and synchronizes to the underlying filesystem as required 807 based on guest feature negotiation. This avoids potential data loss (#399, 808 #1216). 809* When booting with a large number of vCPUs then the ACPI table would be 810 overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the 811 `MPTABLE` will no longer be generated (#1132). 812* Shutting down VMs that have been paused is now supported (#816). 813* Created socket files are deleted on shutdown (#1083). 814* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds 815 (#751). 816 817### Command Line and API Changes 818 819This is non exhaustive list of HTTP API and command line changes: 820 821* All user visible socket parameters are now consistently called `socket` 822 rather than `sock` in some cases. 823* The `ch-remote` tool now shows any error message generated by the VMM 824* The `wce` parameter has been removed from `--disk` as the feature is always 825 offered for negotiation. 826* `--net` has gained a `host_mac` option that allows the setting of the MAC 827 address for the `tap` device on the host. 828 829### Contributors 830 831Many thanks to everyone who has contributed to our 0.8.0 release including some new faces. 832 833* Anatol Belski <ab@php.net> 834* Arron Wang <arron.wang@intel.com> 835* Bo Chen <chen.bo@intel.com> 836* Dr. David Alan Gilbert <dgilbert@redhat.com> 837* Henry Wang <Henry.Wang@arm.com> 838* Hui Zhu <teawater@antfin.com> 839* LiYa'nan <oliverliyn@gmail.com> 840* Michael Zhao <michael.zhao@arm.com> 841* Rob Bradford <robert.bradford@intel.com> 842* Samuel Ortiz <sameo@linux.intel.com> 843* Sebastien Boeuf <sebastien.boeuf@intel.com> 844* Sergio Lopez <slp@redhat.com> 845 846# v0.7.0 847 848This release has been tracked through the [0.7.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 849 850Highlights for `cloud-hypervisor` version 0.7.0 include: 851 852### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug 853 854Further to our effort to support modifying a running guest we now support 855hotplug and unplug of the following virtio backed devices: block, network, 856pmem, virtio-fs and vsock. This functionality is available on the (default) PCI 857based transport and is exposed through the HTTP API. The `ch-remote` utility 858provides a CLI for adding or removing these device types after the VM has 859booted. User can use the `id` parameter on the devices to choose names for 860devices to ease their removal. 861 862### Alternative `libc` Support 863 864Cloud Hypervisor can now be compiled with the `musl` C library and this release 865contains a static binary compiled using that toolchain. 866 867### Multithreaded Multi Queued `vhost-user` Backends 868 869The `vhost-user` backends for network and block support that are shipped by 870Cloud Hypervisor have been enhanced to support multiple threads and queues to 871improve throughput. These backends are used automatically if `vhost_user=true` 872is passed when the devices are created. 873 874### Initial RamFS Support 875 876By passing the `--initramfs` command line option the user can specify a file to 877be loaded into the guest memory to be used as the kernel initial filesystem. 878This is usually used to allow the loading of drivers needed to be able to 879access the real root filesystem but it can also be used standalone for a very 880minimal image. 881 882### Alternative Memory Hotplug: `virtio-mem` 883 884As well as supporting ACPI based hotplug Cloud Hypervisor now supports using 885the `virtio-mem` hotplug alternative. This can be controlled by the 886`hotplug_method` parameter on the `--memory` command line option. It currently 887requires kernel patches to be able to support it. 888 889### `Seccomp` Sandboxing 890 891Cloud Hypervisor now has support for restricting the system calls that the 892process can use via the `seccomp` security API. This on by default and is 893controlled by the `--seccomp` command line option. 894 895### Updated Distribution Support 896 897With the release of Ubuntu 20.04 we have added that to the list of supported 898distributions and is part of our regular testing programme. 899 900### Command Line and API Changes 901 902This is non exhaustive list of HTTP API and command line changes 903 904* New `id` fields added for devices to allow them to be named to ease removal. 905 If no name is specified the VMM chooses one. 906* Use `--memory`'s `shared` and `hugepages` controls for determining backing 907 memory instead of providing a path. 908* The `--vsock` parameter only takes one device as the Linux kernel only 909 supports a single Vsock device. The REST API has removed the vector for this 910 option and replaced it with a single optional field. 911* There is enhanced validation of the command line and API provided 912 configurations to ensure that the provided options are compatible e.g. that 913 shared memory is in use if any attempt is made to used a `vhost-user` backed 914 device. 915* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and 916 `add-vsock` subcommands. For removal `remove-device` is used. The REST API 917 has appropriate new HTTP endpoints too. 918* Specifying a `size` with `--pmem` is no longer required and instead the size 919 will be obtained from the file. A `discard_writes` option has also been added 920 to provide the equivalent of a read-only file. 921* The parameters to `--block-backend` have been changed to more closely align 922 with those used by `--disk`. 923 924### Contributors 925 926Many thanks to everyone who has contributed to our 0.7.0 release including some new faces. 927 928* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 929* Bo Chen <chen.bo@intel.com> 930* Cathy Zhang <cathy.zhang@intel.com> 931* Damjan Georgievski <gdamjan@gmail.com> 932* Dean Sheather <dean@coder.com> 933* Eryu Guan <eguan@linux.alibaba.com> 934* Hui Zhu <teawater@antfin.com> 935* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 936* Martin Xu <martin.xu@intel.com> 937* Muminul Islam <muislam@microsoft.com> 938* Rob Bradford <robert.bradford@intel.com> 939* Samuel Ortiz <sameo@linux.intel.com> 940* Sebastien Boeuf <sebastien.boeuf@intel.com> 941* Sergio Lopez <slp@redhat.com> 942* Yang Zhong <yang.zhong@intel.com> 943* Yi Sun <yi.y.sun@linux.intel.com> 944 945# v0.6.0 946 947This release has been tracked through the [0.6.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 948 949Highlights for `cloud-hypervisor` version 0.6.0 include: 950 951### Directly Assigned Devices Hotplug 952 953We continued our efforts around supporting dynamically changing the guest 954resources. After adding support for CPU and memory hotplug, Cloud Hypervisor 955now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`) 956devices into an already running guest. This closes the features gap for 957providing a complete Kata Containers workloads support with Cloud Hypervisor. 958 959### Shared Filesystem Improvements 960 961We enhanced our shared filesystem support through many `virtio-fs` improvements. 962By adding support for DAX, parallel processing of multiple requests, `FS_IO`, 963`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we 964improved our filesystem sharing performance, but also made it more stable and 965compatible with other `virtio-fs` implementations. 966 967### Block and Networking IO Self Offloading 968 969When choosing to offload the paravirtualized block and networking I/O to an 970external process (through the `vhost-user` protocol), Cloud Hypervisor now 971automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends 972into their own, separate processes. 973This provides a seamless paravirtualized I/O user experience for those who want 974to run their guest I/O into separate executions contexts. 975 976### Command Line Interface 977 978More and more Cloud Hypervisor services are exposed through the 979[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only 980accessible via relatively cumbersome HTTP calls. In order to abstract 981those calls into a more user friendly tool, we created a Cloud Hypervisor 982Command Line Interface (CLI) called `ch-remote`. The `ch-remote` binary 983is created with each build and available e.g. at 984`cloud-hypervisor/target/debug/ch-remote` when doing a debug build. 985 986Please check `ch-remote --help` for a complete description of all available 987commands. 988 989### PVH Boot 990 991In addition to the traditional Linux boot protocol, Cloud Hypervisor now 992supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html). 993 994### Contributors 995 996With the 0.6.0 release, we are welcoming a few new contributors. Many thanks 997to them and to everyone that contributed to this release: 998 999* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 1000* Arron Wang <arron.wang@intel.com> 1001* Bin Liu <liubin0329@gmail.com> 1002* Bo Chen <chen.bo@intel.com> 1003* Cathy Zhang <cathy.zhang@intel.com> 1004* Eryu Guan <eguan@linux.alibaba.com> 1005* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1006* Liu Bo <bo.liu@linux.alibaba.com> 1007* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1008* Rob Bradford <robert.bradford@intel.com> 1009* Samuel Ortiz <sameo@linux.intel.com> 1010* Sebastien Boeuf <sebastien.boeuf@intel.com> 1011* Sergio Lopez <slp@redhat.com> 1012 1013# v0.5.1 1014 1015This is a bugfix release branched off v0.5.0. It contains the following fixes: 1016 1017* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez 1018* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf 1019* Fixed error reporting for resize command - Samuel Ortiz 1020* Double reboot workaround (#783) - Rob Bradford 1021* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford 1022 1023# v0.5.0 1024 1025This release has been tracked through the [0.5.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/6). 1026 1027Highlights for `cloud-hypervisor` version 0.5.0 include: 1028 1029### Virtual Machine Dynamic Resizing 1030 1031With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and 1032memory hot plug as well. This allows to dynamically resize Cloud Hypervisor 1033guests which is needed for e.g. Kubernetes related use cases. 1034The memory hot plug implementation is based on the same framework as the CPU hot 1035plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest. 1036 1037Next on our VM resizing roadmap is the PCI devices hotplug feature. 1038 1039### Multi-Queue, Multi-Threaded Paravirtualization 1040 1041We enhanced our virtio networking and block support by having both devices use 1042multiple I/O queues handled by multiple threads. This improves our default 1043paravirtualized networking and block devices throughput. 1044 1045### New Interrupt Management Framework 1046 1047We improved our interrupt management implementation by introducing an Interrupt 1048Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device) 1049crates discussions. This move made the code significantly cleaner, and allowed 1050us to remove several KVM related dependencies from crates like the PCI and 1051virtio ones. 1052 1053### Development Tools 1054 1055In order to provide a better developer experience, we worked on improving our 1056build, development and testing tools. 1057Somehow similar to the excellent 1058[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool), 1059we now provide a [dev_cli script](scripts/dev_cli.sh). 1060 1061With this new tool, our users and contributors will be able to build and test 1062Cloud Hypervisor through a containerized environment. 1063 1064### Kata Containers Integration 1065 1066We spent some significant time and efforts debugging and fixing our integration 1067with the [Kata Containers](https://github.com/kata-containers) project. Cloud 1068Hypervisor is now a fully supported Kata Containers hypervisor, and is 1069integrated into the project's CI. 1070 1071### Contributors 1072 1073Many thanks to everyone that contributed to the 0.5.0 release: 1074 1075* Bo Chen <chen.bo@intel.com> 1076* Cathy Zhang <cathy.zhang@intel.com> 1077* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1078* Rob Bradford <robert.bradford@intel.com> 1079* Samuel Ortiz <sameo@linux.intel.com> 1080* Sebastien Boeuf <sebastien.boeuf@intel.com> 1081* Sergio Lopez <slp@redhat.com> 1082* Yang Zhong <yang.zhong@intel.com> 1083 1084# v0.4.0 1085 1086This release has been tracked through the [0.4.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/4). 1087 1088Highlights for `cloud-hypervisor` version 0.4.0 include: 1089 1090### Dynamic virtual CPUs addition 1091 1092As a way to vertically scale Cloud-Hypervisor guests, we now support dynamically 1093adding virtual CPUs to the guests, a mechanism also known as CPU hot plug. 1094Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs 1095to an already running guest and the high level operations for that process are 1096documented [here](docs/hotplug.md) 1097 1098During the next release cycles we are planning to extend Cloud Hypervisor 1099hot plug framework to other resources, namely PCI devices and memory. 1100 1101### Programmatic firmware tables generation 1102 1103As part of the CPU hot plug feature enablement, and as a requirement for hot 1104plugging other resources like devices or RAM, we added support for 1105programmatically generating the needed ACPI tables. Through a dedicated 1106`acpi-tables` crate, we now have a flexible and clean way of generating those 1107tables based on the VMM device model and topology. 1108 1109### Filesystem and block devices vhost-user backends 1110 1111Our objective of running all Cloud Hypervisor paravirtualized I/O to a 1112vhost-user based framework is getting closer as we've added Rust based 1113implementations for vhost-user-blk and virtiofs backends. Together with the 1114vhost-user-net backend that came with the 0.3.0 release, this will form the 1115default Cloud Hypervisor I/O architecture. 1116 1117### Guest pause and resume 1118 1119As an initial requirement for enabling live migration, we added support for 1120pausing and resuming any VMM components. As an intermediate step towards live 1121migration, the upcoming guest snapshotting feature will be based on the pause 1122and resume capabilities. 1123 1124### Userspace IOAPIC by default 1125 1126As a way to simplify our device manager implementation, but also in order to 1127stay away from privileged rings as often as possible, any device that relies on 1128pin based interrupts will be using the userspace IOAPIC implementation by 1129default. 1130 1131### PCI BAR reprogramming 1132 1133In order to allow for a more flexible device model, and also support guests 1134that would want to move PCI devices, we added support for PCI devices BAR 1135reprogramming. 1136 1137### New `cloud-hypervisor` organization 1138 1139As we wanted to be more flexible on how we manage the Cloud Hypervisor project, 1140we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor). 1141Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) 1142project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux) 1143and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1144repositories. We may also use it to host any rust-vmm that we'd need to 1145temporarily fork. 1146Thanks to GitHub's seamless repository redirections, the move is completely 1147transparent to all Cloud Hypervisor contributors, users and followers. 1148 1149### Contributors 1150 1151Many thanks to everyone that contributed to the 0.4.0 release: 1152 1153* Cathy Zhang <cathy.zhang@intel.com> 1154* Emin Ghuliev <drmint80@gmail.com> 1155* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1156* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1157* Rob Bradford <robert.bradford@intel.com> 1158* Samuel Ortiz <sameo@linux.intel.com> 1159* Sebastien Boeuf <sebastien.boeuf@intel.com> 1160* Sergio Lopez <slp@redhat.com> 1161* Wu Zongyong <wuzongyong@linux.alibaba.com> 1162 1163# v0.3.0 1164 1165This release has been tracked through the [0.3.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/3). 1166 1167Highlights for `cloud-hypervisor` version 0.3.0 include: 1168 1169### Block device offloading 1170 1171We continue to work on offloading paravirtualized I/O to external processes, 1172and we added support for 1173[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends. 1174This enables `cloud-hypervisor` users to plug a `vhost-user` based block device 1175like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage 1176backend. 1177 1178### Network device backend 1179 1180The previous release provided support for 1181[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we 1182also provide a TAP based vhost-user-net backend, implemented in Rust. Together 1183with the vhost-user-net device implementation, this will eventually become the 1184Cloud Hypervisor default paravirtualized networking architecture. 1185 1186### Virtual sockets 1187 1188In order to more efficiently and securely communicate between host and guest, 1189we added an hybrid implementation of the 1190[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address 1191family over virtio. Credits go to the 1192[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md) 1193project as our implementation is a copy of theirs. 1194 1195### HTTP based API 1196 1197In anticipation of the need to support asynchronous operations to Cloud 1198Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP 1199based API to the VMM. The API will be more extensively documented during the 1200next release cycle. 1201 1202### Memory mapped virtio transport 1203 1204In order to support potential PCI-free use cases, we added support for the 1205[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002) 1206transport layer. This will allow us to support simple, minimal guest 1207configurations that do not require a PCI bus emulation. 1208 1209### Paravirtualized IOMMU 1210 1211As we want to improve our nested guests support, we added support for exposing 1212a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows 1213for a safer nested virtio and directly assigned devices support. 1214 1215To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor 1216users to be able to specify if devices had to be handled through this virtual 1217IOMMU or not. In particular, the `--disk` option now expects disk paths to be 1218prefixed with a `path=` string, and supports an optional `iommu=[on|off]` 1219setting. 1220 1221### Ubuntu 19.10 1222 1223With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware), 1224we can now support the latest 1225[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images. 1226 1227### Large memory guests 1228 1229After simplifying and changing our guest address space handling, we can now 1230support guests with large amount of memory (more than 64GB). 1231 1232# v0.2.0 1233 1234This release has been tracked through the [0.2.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/2). 1235 1236Highlights for `cloud-hypervisor` version 0.2.0 include: 1237 1238### Network device offloading 1239 1240As part of our general effort to offload paravirtualized I/O to external 1241processes, we added support for 1242[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This 1243enables `cloud-hypervisor` users to plug a `vhost-user` based networking device 1244(e.g. [DPDK](https://dpdk.org)) into the VMM as their virtio network backend. 1245 1246### Minimal hardware-reduced ACPI 1247 1248In order to properly implement and guest reset and shutdown, we implemented 1249a minimal version of the hardware-reduced ACPI specification. Together with 1250a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to 1251cleanly reboot and shutdown. 1252 1253The ACPI implementation is a `cloud-hypervisor` build time option that is 1254enabled by default. 1255 1256### Debug I/O port 1257 1258Based on the Firecracker idea of using a dedicated I/O port to measure guest 1259boot times, we added support for logging guest events through the 1260[0x80](https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html) 1261PC debug port. This allows, among other things, for granular guest boot time 1262measurements. See our [debug port documentation](docs/debug-port.md) for more 1263details. 1264 1265### Improved direct device assignment 1266 1267We fixed a major performance issue with our initial VFIO implementation: When 1268enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads 1269were (in many cases) not cached. After correctly tagging the guest memory from 1270`cloud-hypervisor` we're now able to reach the expected performance from 1271directly assigned devices. 1272 1273### Improved shared filesystem 1274 1275We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt) 1276support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system. 1277This provides better shared filesystem IO performance with a smaller guest 1278memory footprint. 1279 1280### Ubuntu bionic based CI 1281 1282Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1283improvements, we are now able to boot Ubuntu bionic images. We added those to 1284our CI pipeline. 1285 1286# v0.1.0 1287 1288This release has been tracked through the [0.1.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/1). 1289 1290Highlights for `cloud-hypervisor` version 0.1.0 include: 1291 1292### Shared filesystem 1293 1294We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file 1295system, allowing for an efficient and reliable way of sharing a filesystem 1296between the host and the `cloud-hypervisor` guest. 1297 1298See our [filesystem sharing](docs/fs.md) documentation for more details on how 1299to use virtio-fs with `cloud-hypervisor`. 1300 1301### Initial direct device assignment support 1302 1303VFIO (Virtual Function I/O) is a kernel framework that exposes direct device 1304access to userspace. `cloud-hypervisor` uses VFIO to directly assign host 1305physical devices into its guest. 1306 1307See our [VFIO](docs/vfio.md) documentation for more detail on how to directly 1308assign host devices to `cloud-hypervisor` guests. 1309 1310### Userspace IOAPIC 1311 1312`cloud-hypervisor` supports a so-called split IRQ chip implementation by 1313implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC). 1314By moving part of the IRQ chip implementation from kernel space to user space, 1315the IRQ chip emulation does not always run in a fully privileged mode. 1316 1317### Virtual persistent memory 1318 1319The `virtio-pmem` implementation emulates a virtual persistent memory device 1320that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device 1321allows to bypass the guest page cache and improve the guest memory footprint. 1322 1323### Linux kernel bzImage 1324 1325The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from 1326`bzImage` kernel images, which is usually the format that Linux distributions 1327use to ship their kernels. For example, this allows for booting from the host 1328distribution kernel image. 1329 1330### Console over virtio 1331 1332`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although 1333using this device as a guest console can potentially cut some early boot 1334messages, it can reduce the guest boot time and provides a complete console 1335implementation. 1336 1337The `virtio-console` device is enabled by default for the guest console. 1338Switching back to the legacy serial port is done by selecting 1339`--serial tty --console off` from the command line. 1340 1341### Unit testing 1342 1343We now run all unit tests from all our crates directly from our CI. 1344 1345### Integration tests parallelization 1346 1347The CI cycle run time has been significantly reduced by refactoring our 1348integration tests; allowing them to all be run in parallel. 1349