1- [v16.0](#v160) 2 - [Improved live migration support](#improved-live-migration-support) 3 - [Improved `vhost-user` support](#improved-vhost-user-support) 4 - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support) 5 - [Notable bug fixes](#notable-bug-fixes) 6 - [Removed functionality](#removed-functionality) 7 - [Contributors](#contributors) 8- [v15.0](#v150) 9 - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees) 10 - [Network device rate limiting](#network-device-rate-limiting) 11 - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload) 12 - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter) 13 - [Bug fixes](#bug-fixes) 14 - [Deprecations](#deprecations) 15 - [Contributors](#contributors-1) 16- [v0.14.1](#v0141) 17- [v0.14.0](#v0140) 18 - [Structured event monitoring](#structured-event-monitoring) 19 - [MSHV improvements](#mshv-improvements) 20 - [Improved aarch64 platform](#improved-aarch64-platform) 21 - [Updated hotplug documentation](#updated-hotplug-documentation) 22 - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console) 23 - [Block device rate limiting](#block-device-rate-limiting) 24 - [Deprecations](#deprecations-1) 25 - [Contributors](#contributors-2) 26- [v0.13.0](#v0130) 27 - [Wider VFIO device support](#wider-vfio-device-support) 28 - [Improved huge page support](#improved-huge-page-support) 29 - [MACvTAP support](#macvtap-support) 30 - [VHD disk image support](#vhd-disk-image-support) 31 - [Improved Virtio device threading](#improved-virtio-device-threading) 32 - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button) 33 - [Contributors](#contributors-3) 34- [v0.12.0](#v0120) 35 - [ARM64 enhancements](#arm64-enhancements) 36 - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning) 37 - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend) 38 - [Enhanced "info" API](#enhanced-info-api) 39 - [Contributors](#contributors-4) 40- [v0.11.0](#v0110) 41 - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block) 42 - [Windows Guest Support](#windows-guest-support) 43 - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation) 44 - [`virtio-mmmio` Removal](#virtio-mmmio-removal) 45 - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64) 46 - [Improved Linux Boot Time](#improved-linux-boot-time) 47 - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling) 48 - [Default Log Level Changed](#default-log-level-changed) 49 - [New `--balloon` Parameter Added](#new---balloon-parameter-added) 50 - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support) 51 - [Notable Bug Fixes](#notable-bug-fixes-1) 52 - [Contributors](#contributors-5) 53- [v0.10.0](#v0100) 54 - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors) 55 - [Memory Zones](#memory-zones) 56 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements) 57 - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control) 58 - [Notable Bug Fixes](#notable-bug-fixes-2) 59 - [Contributors](#contributors-6) 60- [v0.9.0](#v090) 61 - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support) 62 - [Block and Network Device Statistics](#block-and-network-device-statistics) 63 - [HTTP API Responses](#http-api-responses) 64 - [CPU Topology](#cpu-topology) 65 - [Release Build Optimization](#release-build-optimization) 66 - [Hypervisor Abstraction](#hypervisor-abstraction) 67 - [Snapshot/Restore Improvements](#snapshotrestore-improvements) 68 - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support) 69 - [Enhancements to ARM64 Support](#enhancements-to-arm64-support) 70 - [Intel SGX Support](#intel-sgx-support) 71 - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1) 72 - [Notable Bug Fixes](#notable-bug-fixes-3) 73 - [Contributors](#contributors-7) 74- [v0.8.0](#v080) 75 - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support) 76 - [Experimental ARM64 Support](#experimental-arm64-support) 77 - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests) 78 - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices) 79 - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements) 80 - [Notable Bug Fixes](#notable-bug-fixes-4) 81 - [Command Line and API Changes](#command-line-and-api-changes) 82 - [Contributors](#contributors-8) 83- [v0.7.0](#v070) 84 - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug) 85 - [Alternative `libc` Support](#alternative-libc-support) 86 - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends) 87 - [Initial RamFS Support](#initial-ramfs-support) 88 - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem) 89 - [`Seccomp` Sandboxing](#seccomp-sandboxing) 90 - [Updated Distribution Support](#updated-distribution-support) 91 - [Command Line and API Changes](#command-line-and-api-changes-1) 92 - [Contributors](#contributors-9) 93- [v0.6.0](#v060) 94 - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug) 95 - [Shared Filesystem Improvements](#shared-filesystem-improvements) 96 - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading) 97 - [Command Line Interface](#command-line-interface) 98 - [PVH Boot](#pvh-boot) 99 - [Contributors](#contributors-10) 100- [v0.5.1](#v051) 101- [v0.5.0](#v050) 102 - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing) 103 - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization) 104 - [New Interrupt Management Framework](#new-interrupt-management-framework) 105 - [Development Tools](#development-tools) 106 - [Kata Containers Integration](#kata-containers-integration) 107 - [Contributors](#contributors-11) 108- [v0.4.0](#v040) 109 - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition) 110 - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation) 111 - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends) 112 - [Guest pause and resume](#guest-pause-and-resume) 113 - [Userspace IOAPIC by default](#userspace-ioapic-by-default) 114 - [PCI BAR reprogramming](#pci-bar-reprogramming) 115 - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization) 116 - [Contributors](#contributors-12) 117- [v0.3.0](#v030) 118 - [Block device offloading](#block-device-offloading) 119 - [Network device backend](#network-device-backend) 120 - [Virtual sockets](#virtual-sockets) 121 - [HTTP based API](#http-based-api) 122 - [Memory mapped virtio transport](#memory-mapped-virtio-transport) 123 - [Paravirtualized IOMMU](#paravirtualized-iommu) 124 - [Ubuntu 19.10](#ubuntu-1910) 125 - [Large memory guests](#large-memory-guests) 126- [v0.2.0](#v020) 127 - [Network device offloading](#network-device-offloading) 128 - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi) 129 - [Debug I/O port](#debug-io-port) 130 - [Improved direct device assignment](#improved-direct-device-assignment) 131 - [Improved shared filesystem](#improved-shared-filesystem) 132 - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci) 133- [v0.1.0](#v010) 134 - [Shared filesystem](#shared-filesystem) 135 - [Initial direct device assignment support](#initial-direct-device-assignment-support) 136 - [Userspace IOAPIC](#userspace-ioapic) 137 - [Virtual persistent memory](#virtual-persistent-memory) 138 - [Linux kernel bzImage](#linux-kernel-bzimage) 139 - [Console over virtio](#console-over-virtio) 140 - [Unit testing](#unit-testing) 141 - [Integration tests parallelization](#integration-tests-parallelization) 142 143# v16.0 144 145This release has been tracked through the [v16.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/19). 146 147### Improved live migration support 148 149The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details. 150 151### Improved `vhost-user` support 152 153When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch. 154 155### ARM64 ACPI and UEFI support 156 157Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot. 158 159### Notable bug fixes 160 161* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578). 162* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563). 163* The KVM clock is now correctly handled during live migration / snapshot & restore. 164 165### Removed functionality 166 167The following formerly deprecated features have been removed: 168 169* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 170 binaries has been deprecated. When using direct boot users should configure 171 their kernel with `CONFIG_PVH=y`. 172 173### Contributors 174 175Many thanks to everyone who has contributed to our release including some new faces. 176 177* Anatol Belski <anbelski@linux.microsoft.com> 178* Bo Chen <chen.bo@intel.com> 179* Dayu Liu <liu.dayu@zte.com.cn> 180* Henry Wang <Henry.Wang@arm.com> 181* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 182* Jianyong Wu <jianyong.wu@arm.com> 183* Michael Zhao <michael.zhao@arm.com> 184* Mikko Ylinen <mikko.ylinen@intel.com> 185* Muminul Islam <muislam@microsoft.com> 186* Ren Lei <ren.lei4@zte.com.cn> 187* Rob Bradford <robert.bradford@intel.com> 188* Sebastien Boeuf <sebastien.boeuf@intel.com> 189* Wei Liu <liuwe@microsoft.com> 190* Yi Wang <wang.yi59@zte.com.cn> 191 192# v15.0 193 194This release has been tracked through the [v15.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/18). 195 196Highlights for `cloud-hypervisor` version v15.0 include: 197 198### Version numbering and stability guarantees 199 200This release is the first in a new version numbering scheme to represent that 201we believe Cloud Hypervisor is maturing and entering a period of stability. 202With this new release we are beginning our new stability guarantees: 203 204* The API (including command line options) will not be removed or changed in a 205 breaking way without a minimum of 2 releases notice. Where possible warnings 206 will be given about the use of deprecated functionality and the deprecations 207 will be documented in the release notes. 208* Point releases will be made between individual releases where there are 209 substantial bug fixes or security issues that need to be fixed. 210 211Currently the following items are **not** guaranteed across updates: 212 213* Snapshot/restore is not supported across different versions 214* Live migration is not supported across different versions 215* The following features are considered experimental and may change 216 substantially between releases: TDX, SGX. 217 218### Network device rate limiting 219 220Building on our existing support for rate limiting block activity the network 221device also now supports rate limiting. Full details of the controls are in the 222[IO throttling documentation.](docs/io_throttling.md) 223 224### Support for runtime control of `virtio-net` guest offload 225 226The guest is now able to change the offload settings for the `virtio-net` 227device. As well as providing a useful control this mitigates an issue in the 228Linux kernel where the guest will attempt to reprogram the offload settings 229even if they are not advertised as configurable (#2528). 230 231### `--api-socket` supports file descriptor parameter 232 233The `--api-socket` can now take an `fd=` parameter to specify an existing file 234descriptor to use. This is particularly beneficial for frameworks that need to 235programmatically control Cloud Hypervisor. 236 237### Bug fixes 238 239* A workaround has been put in place to mitigate a Linux kernel issues that 240 results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277). 241* PCI BARs are now correctly aligned removing the need for the guest to 242 reprogram them (#1797,#1798) 243* Handle TAP interface not being writable within virtio-net (due to the buffer 244 exhaustion on the host) (#2517) 245* The recommended Linux kernel is now v5.12.0 as it contains a fix that 246 prevents snapshot & restore working (#2535) 247 248### Deprecations 249 250Deprecated features will be removed in a subsequent release and users should plan to use alternatives 251 252* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 253 binaries has been deprecated. When using direct boot users should configure 254 their kernel with `CONFIG_PVH=y`. Will be removed in v16.0. 255 256### Contributors 257 258Many thanks to everyone who has contributed to our release including some new faces. 259 260* Alyssa Ross <hi@alyssa.is> 261* Anatol Belski <anbelski@linux.microsoft.com> 262* Bo Chen <chen.bo@intel.com> 263* Gaelan Steele <gbs@canishe.com> 264* Jianyong Wu <jianyong.wu@arm.com> 265* Michael Zhao <michael.zhao@arm.com> 266* Muminul Islam <muislam@microsoft.com> 267* Rob Bradford <robert.bradford@intel.com> 268* Sebastien Boeuf <sebastien.boeuf@intel.com> 269* Wei Liu <liuwe@microsoft.com> 270* William Douglas <william.douglas@intel.com> 271 272# v0.14.1 273 274Bug fix release branched off the v0.14.0 release. The following bugs were fixed 275in this release: 276 277* CPU hotplug on Windows failed due to misreported CPU state information and 278 the lack of HyperV CPUID bit enabled (#2437, #2449, #2436) 279* A seccomp rule was missing that was triggered on CPU unplug (#2455) 280* A bounds check in VIRTIO queue validation was erroneously generating 281 DescriptorChainTooShort errors in certain circumstances (#2450, #2424) 282 283# v0.14.0 284 285This release has been tracked through the [0.14.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/17). 286 287Highlights for `cloud-hypervisor` version 0.14.0 include: 288 289### Structured event monitoring 290 291A new option was added to the VMM `--event-monitor` which reports structured 292events (JSON) over a file or file descriptor at key events in the lifecycle of 293the VM. The list of events is limited at the moment but will be further 294extended over subsequent releases. The events exposed form part of the Cloud 295Hypervisor API surface. 296 297### MSHV improvements 298 299Basic support has been added for running Windows guests atop the MSHV 300hypervisor as an alternative to KVM and further improvements have been made to 301the MSHV support. 302 303### Improved aarch64 platform 304 305The aarch64 platform has been enhanced with more devices exposed to the running 306VM including an enhanced serial UART. 307 308### Updated hotplug documentation 309 310The documentation for the hotplug support has been updated to reflect the use 311of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as 312well as documenting hotplug of paravirtualised and VFIO devices. 313 314### PTY control for serial and `virtio-console` 315 316The `--serial` and `--console` parameters can now direct the console to a PTY 317allowing programmatic control of the console from another process through the 318PTY subsystem. 319 320### Block device rate limiting 321 322The block device performance can now be constrained as part of the VM 323configuration allowing rate limiting. Full details of the controls are in the 324[IO throttling documentation.](docs/io_throttling.md) 325 326 327### Deprecations 328 329Deprecated features will be removed in a subsequent release and users should plan to use alternatives 330 331* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage` 332 binaries has been deprecated. When using direct boot users should configure 333 their kernel with `CONFIG_PVH=y`. 334 335 336### Contributors 337 338Many thanks to everyone who has contributed to our 0.14.0 release including 339some new faces. 340 341Bo Chen <chen.bo@intel.com> 342Henry Wang <Henry.Wang@arm.com> 343Iggy Jackson <iggy@theiggy.com> 344Jiachen Zhang <zhangjiachen.jaycee@bytedance.com> 345Michael Zhao <michael.zhao@arm.com> 346Muminul Islam <muislam@microsoft.com> 347Penny Zheng <Penny.Zheng@arm.com> 348Rob Bradford <robert.bradford@intel.com> 349Sebastien Boeuf <sebastien.boeuf@intel.com> 350Vineeth Pillai <viremana@linux.microsoft.com> 351Wei Liu <liuwe@microsoft.com> 352William Douglas <william.r.douglas@gmail.com> 353Zide Chen <zide.chen@intel.com> 354 355# v0.13.0 356 357This release has been tracked through the [0.13.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/16). 358 359Highlights for `cloud-hypervisor` version 0.13.0 include: 360 361### Wider VFIO device support 362 363It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI 364devices that do not support MSI or MSI-X and instead rely on INTx interrupts. 365Most notably this widens the support to most NVIDIA cards with the proprietary 366drivers. 367 368### Improved huge page support 369 370Through the addition of `hugepage_size` on `--memory` it is now possible to 371specify the desired size of the huge pages used when allocating the guest 372memory. The user is required to ensure they have sufficient pages of the 373desired size in their pool. 374 375### MACvTAP support 376 377It is now possible to provide file descriptors using the `fd` parameter to 378`--net` which point at TAP devices that have already been opened by the user. 379This aids integration with `libvirt` but also permits the use of MACvTAP 380support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md). 381 382### VHD disk image support 383 384It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw 385disk image with Cloud Hypervisor. 386 387### Improved Virtio device threading 388 389Device threads are now derived from the main VMM thread which allows more 390restrictive seccomp filters to be applied to them. The threads also have a 391predictable name derived from the device id. 392 393### Clean shutdown support via synthetic power button 394 395It is now possible to request that the guest VM shut itself down by triggering 396a synthetic ACPI power button press from the VMM. If the guest is listening for 397such an event (e.g. using systemd) then it will process the event and cleanly 398shut down. This functionality is exposed through the HTTP API and can be 399triggered via `ch-remote --api-socket=<API socket> power-button`. 400 401### Contributors 402 403Many thanks to everyone who has contributed to our 0.13.0 release including 404some new faces. 405 406* Bo Chen <chen.bo@intel.com> 407* Mikko Ylinen <mikko.ylinen@intel.com> 408* Muminul Islam <muislam@microsoft.com> 409* Rob Bradford <robert.bradford@intel.com> 410* Samuel Ortiz <sameo@linux.intel.com> 411* Sebastien Boeuf <sebastien.boeuf@intel.com> 412* Vineeth Pillai <viremana@linux.microsoft.com> 413* Wei Liu <liuwe@microsoft.com> 414* William Douglas <william.r.douglas@gmail.com> 415* Xie Yongji <xieyongji@bytedance.com> 416 417# v0.12.0 418 419This release has been tracked through the [0.12.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/15). 420 421Highlights for `cloud-hypervisor` version 0.12.0 include: 422 423### ARM64 enhancements 424 425The use of `--watchdog` is now fully supported as is the ability to reboot the 426VM from within the guest when running Cloud Hypervisor on an ARM64 system. 427 428### Removal of `vhost-user-net` and `vhost-user-block` self spawning 429 430In order to use `vhost-user-net` or `vhost-user-block` backends the user is now 431responsible for starting the backend and providing the socket for the VMM to 432use. This functionality was deprecated in the last release and how now been 433removed. 434 435### Migration of `vhost-user-fs` backend 436 437The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is 438instead hosted in [it's own 439repository](https://gitlab.com/virtio-fs/virtiofsd-rs) 440 441### Enhanced "info" API 442 443The `vm.info` HTTP API endpoint has been extended to include the details of the 444devices used by the VM including any VFIO devices used. 445 446### Contributors 447 448Many thanks to everyone who has contributed to our 0.12.0 release: 449 450* Anatol Belski <anbelski@linux.microsoft.com> 451* Julio Montes <julio.montes@intel.com> 452* Michael Zhao <michael.zhao@arm.com> 453* Muminul Islam <muislam@microsoft.com> 454* Rob Bradford <robert.bradford@intel.com> 455* Samuel Ortiz <sameo@linux.intel.com> 456* Wei Liu <liuwe@microsoft.com> 457 458# v0.11.0 459 460This release has been tracked through the [0.11.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/14). 461 462Highlights for `cloud-hypervisor` version 0.11.0 include: 463 464### `io_uring` support by default for `virtio-block` 465 466Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will 467be used for a significantly higher performance block device. 468 469### Windows Guest Support 470 471This is the first release where we officially support Windows running as a 472guest. Full details of how to setup the image and run Cloud Hypervisor with a 473Windows guest can be found in the dedicated [Windows 474documentation](docs/windows.md). 475 476### `vhost-user` "Self Spawning" Deprecation 477 478Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now 479deprecated. Users of this functionality will receive a warning and should make 480adjustments. The functionality will be removed in the next release. 481 482### `virtio-mmmio` Removal 483 484Support for using the `virtio-mmio` transport, rather than using PCI, has been 485removed. This has been to simplify the code and significantly 486reduce the testing burden of the project. 487 488### Snapshot/Restore support for ARM64 489 490When running on the ARM64 architecture snapshot and restore has now been 491implemented. 492 493### Improved Linux Boot Time 494 495The time to boot the Linux kernel has been significantly improved by the 496identifying some areas of delays around PCI bus probing, IOAPIC programming and 497MPTABLE issues. Full details can be seen in #1728. 498 499### `SIGTERM/SIGINT` Interrupt Signal Handling 500 501When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will 502trigger the VMM process to cleanly deallocate resources before exiting. The 503guest VM will not be cleanly shutdown but the VMM process will clean up its 504resources. 505 506### Default Log Level Changed 507 508The default logging level was changed to include warnings which should make it 509easier to see potential issues. New [logging 510documentation](docs/logging) was also added. 511 512### New `--balloon` Parameter Added 513 514Control of the setup of `virtio-balloon` has been moved from `--memory` to its 515own dedicated parameter. This makes it easier to add more balloon specific 516controls without overloading `--memory`. 517 518### Experimental `virtio-watchdog` Support 519 520Support for using a new `virtio-watchdog` has been added which can be used to 521have the VMM reboot the guest if the guest userspace fails to ping the 522watchdog. This is enabled with `--watchdog` and requires kernel support. 523 524### Notable Bug Fixes 525 526* MTRR bit was missing from CPUID advertised to guest 527* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170) 528* CPU identification string is now exposed to the guest 529* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as 530 read only so avoids excessive VM exits (#1795) 531* PCI device hotplug after an unplug was fixed (#1802) 532* When using the ACPI method to resize the guest memory the full reserved size 533 can be used (#1803) 534* Snapshot and restore followed by a second snapshot and restore now works 535 correctly 536* Snapshot and restore of VMs with more than 2GiB in one region now work 537 correctly 538 539### Contributors 540 541Many thanks to everyone who has contributed to our 0.11.0 release including some new faces. 542 543* Anatol Belski <anbelski@linux.microsoft.com> 544* Bo Chen <chen.bo@intel.com> 545* Daniel Verkamp <dverkamp@chromium.org> 546* Henry Wang <Henry.Wang@arm.com> 547* Hui Zhu <teawater@antfin.com> 548* Jiangbo Wu <jiangbo.wu@intel.com> 549* Josh Soref <jsoref@users.noreply.github.com> 550* Julio Montes <julio.montes@intel.com> 551* Michael Zhao <michael.zhao@arm.com> 552* Muminul Islam <muislam@microsoft.com> 553* pierwill <19642016+pierwill@users.noreply.github.com> 554* Praveen Paladugu <prapal@microsoft.com> 555* Rob Bradford <robert.bradford@intel.com> 556* Sebastien Boeuf <sebastien.boeuf@intel.com> 557* Wei Liu <liuwe@microsoft.com> 558 559 560# v0.10.0 561 562This release has been tracked through the [0.10.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/13). 563 564Highlights for `cloud-hypervisor` version 0.10.0 include: 565 566### `virtio-block` Support for Multiple Descriptors 567 568Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers. 569 570### Memory Zones 571 572Support has been added for fine grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md). 573 574### `Seccomp` Sandbox Improvements 575 576All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`. 577 578### Preliminary KVM HyperV Emulation Control 579 580A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices. 581 582### Notable Bug Fixes 583 584- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596) 585- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564) 586- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593) 587- `--version` shows the version for released binaries (#1669) 588- Errors generated by worker threads for `virtio` devices are now printed out (#1551) 589 590### Contributors 591 592Many thanks to everyone who has contributed to our 0.10.0 release including some new faces. 593 594* Alyssa Ross <hi@alyssa.is> 595* Amey Narkhede <ameynarkhede02@gmail.com> 596* Anatol Belski <ab@php.net> 597* Bo Chen <chen.bo@intel.com> 598* Hui Zhu <teawater@antfin.com> 599* Michael Zhao <michael.zhao@arm.com> 600* Muminul Islam <muislam@microsoft.com> 601* Rob Bradford <robert.bradford@intel.com> 602* Samuel Ortiz <sameo@linux.intel.com> 603* Sebastien Boeuf <sebastien.boeuf@intel.com> 604* Wei Liu <liuwe@microsoft.com> 605 606# v0.9.0 607 608This release has been tracked through the [0.9.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/12). 609 610Highlights for `cloud-hypervisor` version 0.9.0 include: 611 612### `io_uring` Based Block Device Support 613 614If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement. 615 616### Block and Network Device Statistics 617 618Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM. 619 620### HTTP API Responses 621 622The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF. 623 624### CPU Topology 625 626A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core. 627 628### Release Build Optimization 629 630Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size. 631 632### Hypervisor Abstraction 633 634A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`. 635 636### Snapshot/Restore Improvements 637 638Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs. 639 640### Virtio Memory Ballooning Support 641 642A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest. 643 644### Enhancements to ARM64 Support 645 646The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting. 647 648### Intel SGX Support 649 650The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md). 651 652### `Seccomp` Sandbox Improvements 653 654The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development. 655 656### Notable Bug Fixes 657 658* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes. 659* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338) 660* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543). 661* `virtio-mmio` based devices are now more widely tested (#275). 662* Multiple issues have been fixed with virtio device configuration (#1217) 663* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521) 664 665### Contributors 666 667Many thanks to everyone who has contributed to our 0.9.0 release including some new faces. 668 669* Anatol Belski <ab@php.net> 670* Bo Chen <chen.bo@intel.com> 671* Dr. David Alan Gilbert <dgilbert@redhat.com> 672* Henry Wang <Henry.Wang@arm.com> 673* Howard Zhang <howard.zhang@arm.com> 674* Hui Zhu <teawater@antfin.com> 675* Jianyong Wu <jianyong.wu@arm.com> 676* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 677* LiYa'nan <oliverliyn@gmail.com> 678* Michael Zhao <michael.zhao@arm.com> 679* Muminul Islam <muislam@microsoft.com> 680* Praveen Paladugu <prapal@microsoft.com> 681* Ricardo Koller <ricarkol@gmail.com> 682* Rob Bradford <robert.bradford@intel.com> 683* Samuel Ortiz <sameo@linux.intel.com> 684* Sebastien Boeuf <sebastien.boeuf@intel.com> 685* Stefano Garzarella <sgarzare@redhat.com> 686* Wei Liu <liuwe@microsoft.com> 687 688 689# v0.8.0 690 691This release has been tracked through the [0.8.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/10). 692 693Highlights for `cloud-hypervisor` version 0.8.0 include: 694 695### Experimental Snapshot and Restore Support 696 697This release includes the first version of the snapshot and restore feature. 698This allows a VM to be paused and then subsequently snapshotted. At a later 699point that snapshot may be restored into a new running VM identical to the 700original VM at the point it was paused. 701 702This feature can be used for offline migration from one VM host to another, to 703allow the upgrading or rebooting of the host machine transparently to the guest 704or for templating the VM. This is an experimental feature and cannot be used on 705a VM using passthrough (VFIO) devices. Issues with SMP have also been observed 706(#1176). 707 708### Experimental ARM64 Support 709 710Included in this release is experimental support for running on ARM64. 711Currently only `virtio-mmio` devices and a serial port are supported. Full 712details can be found in the [ARM64 documentation](docs/arm64.md). 713 714### Support for Using 5-level Paging in Guests 715 716If the host supports it the guest is now enabled for 5-level paging (aka LA57). 717This works when booting the Linux kernel with a vmlinux, bzImage or firmware 718based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not 719work due to current limitations in the PVH boot process. 720 721### Virtio Device Interrupt Suppression for Network Devices 722 723With `virtio-net` and `vhost-user-net` devices the guest can suppress 724interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This 725can lead to an improvement in performance by reducing the number of interrupts 726the guest must service. 727 728### `vhost_user_fs` Improvements 729 730The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`. 731 732 733### Notable Bug Fixes 734 735* VMs that have not yet been booted can now be deleted (#1110). 736* By creating the `tap` device ahead of creating the VM it is not required to 737 run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273). 738* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to 739 the specification and synchronizes to the underlying filesystem as required 740 based on guest feature negotiation. This avoids potential data loss (#399, 741 #1216). 742* When booting with a large number of vCPUs then the ACPI table would be 743 overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the 744 `MPTABLE` will no longer be generated (#1132). 745* Shutting down VMs that have been paused is now supported (#816). 746* Created socket files are deleted on shutdown (#1083). 747* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds 748 (#751). 749 750### Command Line and API Changes 751 752This is non exhaustive list of HTTP API and command line changes: 753 754* All user visible socket parameters are now consistently called `socket` 755 rather than `sock` in some cases. 756* The `ch-remote` tool now shows any error message generated by the VMM 757* The `wce` parameter has been removed from `--disk` as the feature is always 758 offered for negotiation. 759* `--net` has gained a `host_mac` option that allows the setting of the MAC 760 address for the `tap` device on the host. 761 762### Contributors 763 764Many thanks to everyone who has contributed to our 0.8.0 release including some new faces. 765 766* Anatol Belski <ab@php.net> 767* Arron Wang <arron.wang@intel.com> 768* Bo Chen <chen.bo@intel.com> 769* Dr. David Alan Gilbert <dgilbert@redhat.com> 770* Henry Wang <Henry.Wang@arm.com> 771* Hui Zhu <teawater@antfin.com> 772* LiYa'nan <oliverliyn@gmail.com> 773* Michael Zhao <michael.zhao@arm.com> 774* Rob Bradford <robert.bradford@intel.com> 775* Samuel Ortiz <sameo@linux.intel.com> 776* Sebastien Boeuf <sebastien.boeuf@intel.com> 777* Sergio Lopez <slp@redhat.com> 778 779# v0.7.0 780 781This release has been tracked through the [0.7.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 782 783Highlights for `cloud-hypervisor` version 0.7.0 include: 784 785### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug 786 787Further to our effort to support modifying a running guest we now support 788hotplug and unplug of the following virtio backed devices: block, network, 789pmem, virtio-fs and vsock. This functionality is available on the (default) PCI 790based transport and is exposed through the HTTP API. The `ch-remote` utility 791provides a CLI for adding or removing these device types after the VM has 792booted. User can use the `id` parameter on the devices to choose names for 793devices to ease their removal. 794 795### Alternative `libc` Support 796 797Cloud Hypervisor can now be compiled with the `musl` C library and this release 798contains a static binary compiled using that toolchain. 799 800### Multithreaded Multi Queued `vhost-user` Backends 801 802The `vhost-user` backends for network and block support that are shipped by 803Cloud Hypervisor have been enhanced to support multiple threads and queues to 804improve throughput. These backends are used automatically if `vhost_user=true` 805is passed when the devices are created. 806 807### Initial RamFS Support 808 809By passing the `--initramfs` command line option the user can specify a file to 810be loaded into the guest memory to be used as the kernel initial filesystem. 811This is usually used to allow the loading of drivers needed to be able to 812access the real root filesystem but it can also be used standalone for a very 813minimal image. 814 815### Alternative Memory Hotplug: `virtio-mem` 816 817As well as supporting ACPI based hotplug Cloud Hypervisor now supports using 818the `virtio-mem` hotplug alternative. This can be controlled by the 819`hotplug_method` parameter on the `--memory` command line option. It currently 820requires kernel patches to be able to support it. 821 822### `Seccomp` Sandboxing 823 824Cloud Hypervisor now has support for restricting the system calls that the 825process can use via the `seccomp` security API. This on by default and is 826controlled by the `--seccomp` command line option. 827 828### Updated Distribution Support 829 830With the release of Ubuntu 20.04 we have added that to the list of supported 831distributions and is part of our regular testing programme. 832 833### Command Line and API Changes 834 835This is non exhaustive list of HTTP API and command line changes 836 837* New `id` fields added for devices to allow them to be named to ease removal. 838 If no name is specified the VMM chooses one. 839* Use `--memory`'s `shared` and `hugepages` controls for determining backing 840 memory instead of providing a path. 841* The `--vsock` parameter only takes one device as the Linux kernel only 842 supports a single Vsock device. The REST API has removed the vector for this 843 option and replaced it with a single optional field. 844* There is enhanced validation of the command line and API provided 845 configurations to ensure that the provided options are compatible e.g. that 846 shared memory is in use if any attempt is made to used a `vhost-user` backed 847 device. 848* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and 849 `add-vsock` subcommands. For removal `remove-device` is used. The REST API 850 has appropriate new HTTP endpoints too. 851* Specifying a `size` with `--pmem` is no longer required and instead the size 852 will be obtained from the file. A `discard_writes` option has also been added 853 to provide the equivalent of a read-only file. 854* The parameters to `--block-backend` have been changed to more closely align 855 with those used by `--disk`. 856 857### Contributors 858 859Many thanks to everyone who has contributed to our 0.7.0 release including some new faces. 860 861* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 862* Bo Chen <chen.bo@intel.com> 863* Cathy Zhang <cathy.zhang@intel.com> 864* Damjan Georgievski <gdamjan@gmail.com> 865* Dean Sheather <dean@coder.com> 866* Eryu Guan <eguan@linux.alibaba.com> 867* Hui Zhu <teawater@antfin.com> 868* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 869* Martin Xu <martin.xu@intel.com> 870* Muminul Islam <muislam@microsoft.com> 871* Rob Bradford <robert.bradford@intel.com> 872* Samuel Ortiz <sameo@linux.intel.com> 873* Sebastien Boeuf <sebastien.boeuf@intel.com> 874* Sergio Lopez <slp@redhat.com> 875* Yang Zhong <yang.zhong@intel.com> 876* Yi Sun <yi.y.sun@linux.intel.com> 877 878# v0.6.0 879 880This release has been tracked through the [0.6.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/7). 881 882Highlights for `cloud-hypervisor` version 0.6.0 include: 883 884### Directly Assigned Devices Hotplug 885 886We continued our efforts around supporting dynamically changing the guest 887resources. After adding support for CPU and memory hotplug, Cloud Hypervisor 888now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`) 889devices into an already running guest. This closes the features gap for 890providing a complete Kata Containers workloads support with Cloud Hypervisor. 891 892### Shared Filesystem Improvements 893 894We enhanced our shared filesystem support through many `virtio-fs` improvements. 895By adding support for DAX, parallel processing of multiple requests, `FS_IO`, 896`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we 897improved our filesystem sharing performance, but also made it more stable and 898compatible with other `virtio-fs` implementations. 899 900### Block and Networking IO Self Offloading 901 902When choosing to offload the paravirtualized block and networking I/O to an 903external process (through the `vhost-user` protocol), Cloud Hypervisor now 904automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends 905into their own, separate processes. 906This provides a seamless paravirtualized I/O user experience for those who want 907to run their guest I/O into separate executions contexts. 908 909### Command Line Interface 910 911More and more Cloud Hypervisor services are exposed through the 912[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only 913accessible via relatively cumbersome HTTP calls. In order to abstract 914those calls into a more user friendly tool, we created a Cloud Hypervisor 915Command Line Interface (CLI) called `ch-remote`. The `ch-remote` binary 916is created with each build and available e.g. at 917`cloud-hypervisor/target/debug/ch-remote` when doing a debug build. 918 919Please check `ch-remote --help` for a complete description of all available 920commands. 921 922### PVH Boot 923 924In addition to the traditional Linux boot protocol, Cloud Hypervisor now 925supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html). 926 927### Contributors 928 929With the 0.6.0 release, we are welcoming a few new contributors. Many thanks 930to them and to everyone that contributed to this release: 931 932* Alejandro Jimenez <alejandro.j.jimenez@oracle.com> 933* Arron Wang <arron.wang@intel.com> 934* Bin Liu <liubin0329@gmail.com> 935* Bo Chen <chen.bo@intel.com> 936* Cathy Zhang <cathy.zhang@intel.com> 937* Eryu Guan <eguan@linux.alibaba.com> 938* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 939* Liu Bo <bo.liu@linux.alibaba.com> 940* Qiu Wenbo <qiuwenbo@phytium.com.cn> 941* Rob Bradford <robert.bradford@intel.com> 942* Samuel Ortiz <sameo@linux.intel.com> 943* Sebastien Boeuf <sebastien.boeuf@intel.com> 944* Sergio Lopez <slp@redhat.com> 945 946# v0.5.1 947 948This is a bugfix release branched off v0.5.0. It contains the following fixes: 949 950* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez 951* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf 952* Fixed error reporting for resize command - Samuel Ortiz 953* Double reboot workaround (#783) - Rob Bradford 954* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford 955 956# v0.5.0 957 958This release has been tracked through the [0.5.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/6). 959 960Highlights for `cloud-hypervisor` version 0.5.0 include: 961 962### Virtual Machine Dynamic Resizing 963 964With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and 965memory hot plug as well. This allows to dynamically resize Cloud Hypervisor 966guests which is needed for e.g. Kubernetes related use cases. 967The memory hot plug implementation is based on the same framework as the CPU hot 968plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest. 969 970Next on our VM resizing roadmap is the PCI devices hotplug feature. 971 972### Multi-Queue, Multi-Threaded Paravirtualization 973 974We enhanced our virtio networking and block support by having both devices use 975multiple I/O queues handled by multiple threads. This improves our default 976paravirtualized networking and block devices throughput. 977 978### New Interrupt Management Framework 979 980We improved our interrupt management implementation by introducing an Interrupt 981Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device) 982crates discussions. This move made the code significantly cleaner, and allowed 983us to remove several KVM related dependencies from crates like the PCI and 984virtio ones. 985 986### Development Tools 987 988In order to provide a better developer experience, we worked on improving our 989build, development and testing tools. 990Somehow similar to the excellent 991[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool), 992we now provide a [dev_cli script](scripts/dev_cli.sh). 993 994With this new tool, our users and contributors will be able to build and test 995Cloud Hypervisor through a containerized environment. 996 997### Kata Containers Integration 998 999We spent some significant time and efforts debugging and fixing our integration 1000with the [Kata Containers](https://github.com/kata-containers) project. Cloud 1001Hypervisor is now a fully supported Kata Containers hypervisor, and is 1002integrated into the project's CI. 1003 1004### Contributors 1005 1006Many thanks to everyone that contributed to the 0.5.0 release: 1007 1008* Bo Chen <chen.bo@intel.com> 1009* Cathy Zhang <cathy.zhang@intel.com> 1010* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1011* Rob Bradford <robert.bradford@intel.com> 1012* Samuel Ortiz <sameo@linux.intel.com> 1013* Sebastien Boeuf <sebastien.boeuf@intel.com> 1014* Sergio Lopez <slp@redhat.com> 1015* Yang Zhong <yang.zhong@intel.com> 1016 1017# v0.4.0 1018 1019This release has been tracked through the [0.4.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/4). 1020 1021Highlights for `cloud-hypervisor` version 0.4.0 include: 1022 1023### Dynamic virtual CPUs addition 1024 1025As a way to vertically scale Cloud-Hypervisor guests, we now support dynamically 1026adding virtual CPUs to the guests, a mechanism also known as CPU hot plug. 1027Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs 1028to an already running guest and the high level operations for that process are 1029documented [here](docs/hotplug.md) 1030 1031During the next release cycles we are planning to extend Cloud Hypervisor 1032hot plug framework to other resources, namely PCI devices and memory. 1033 1034### Programmatic firmware tables generation 1035 1036As part of the CPU hot plug feature enablement, and as a requirement for hot 1037plugging other resources like devices or RAM, we added support for 1038programmatically generating the needed ACPI tables. Through a dedicated 1039`acpi-tables` crate, we now have a flexible and clean way of generating those 1040tables based on the VMM device model and topology. 1041 1042### Filesystem and block devices vhost-user backends 1043 1044Our objective of running all Cloud Hypervisor paravirtualized I/O to a 1045vhost-user based framework is getting closer as we've added Rust based 1046implementations for vhost-user-blk and virtiofs backends. Together with the 1047vhost-user-net backend that came with the 0.3.0 release, this will form the 1048default Cloud Hypervisor I/O architecture. 1049 1050### Guest pause and resume 1051 1052As an initial requirement for enabling live migration, we added support for 1053pausing and resuming any VMM components. As an intermediate step towards live 1054migration, the upcoming guest snapshotting feature will be based on the pause 1055and resume capabilities. 1056 1057### Userspace IOAPIC by default 1058 1059As a way to simplify our device manager implementation, but also in order to 1060stay away from privileged rings as often as possible, any device that relies on 1061pin based interrupts will be using the userspace IOAPIC implementation by 1062default. 1063 1064### PCI BAR reprogramming 1065 1066In order to allow for a more flexible device model, and also support guests 1067that would want to move PCI devices, we added support for PCI devices BAR 1068reprogramming. 1069 1070### New `cloud-hypervisor` organization 1071 1072As we wanted to be more flexible on how we manage the Cloud Hypervisor project, 1073we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor). 1074Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) 1075project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux) 1076and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1077repositories. We may also use it to host any rust-vmm that we'd need to 1078temporarily fork. 1079Thanks to GitHub's seamless repository redirections, the move is completely 1080transparent to all Cloud Hypervisor contributors, users and followers. 1081 1082### Contributors 1083 1084Many thanks to everyone that contributed to the 0.4.0 release: 1085 1086* Cathy Zhang <cathy.zhang@intel.com> 1087* Emin Ghuliev <drmint80@gmail.com> 1088* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com> 1089* Qiu Wenbo <qiuwenbo@phytium.com.cn> 1090* Rob Bradford <robert.bradford@intel.com> 1091* Samuel Ortiz <sameo@linux.intel.com> 1092* Sebastien Boeuf <sebastien.boeuf@intel.com> 1093* Sergio Lopez <slp@redhat.com> 1094* Wu Zongyong <wuzongyong@linux.alibaba.com> 1095 1096# v0.3.0 1097 1098This release has been tracked through the [0.3.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/3). 1099 1100Highlights for `cloud-hypervisor` version 0.3.0 include: 1101 1102### Block device offloading 1103 1104We continue to work on offloading paravirtualized I/O to external processes, 1105and we added support for 1106[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends. 1107This enables `cloud-hypervisor` users to plug a `vhost-user` based block device 1108like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage 1109backend. 1110 1111### Network device backend 1112 1113The previous release provided support for 1114[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we 1115also provide a TAP based vhost-user-net backend, implemented in Rust. Together 1116with the vhost-user-net device implementation, this will eventually become the 1117Cloud Hypervisor default paravirtualized networking architecture. 1118 1119### Virtual sockets 1120 1121In order to more efficiently and securely communicate between host and guest, 1122we added an hybrid implementation of the 1123[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address 1124family over virtio. Credits go to the 1125[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md) 1126project as our implementation is a copy of theirs. 1127 1128### HTTP based API 1129 1130In anticipation of the need to support asynchronous operations to Cloud 1131Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP 1132based API to the VMM. The API will be more extensively documented during the 1133next release cycle. 1134 1135### Memory mapped virtio transport 1136 1137In order to support potential PCI-free use cases, we added support for the 1138[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002) 1139transport layer. This will allow us to support simple, minimal guest 1140configurations that do not require a PCI bus emulation. 1141 1142### Paravirtualized IOMMU 1143 1144As we want to improve our nested guests support, we added support for exposing 1145a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows 1146for a safer nested virtio and directly assigned devices support. 1147 1148To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor 1149users to be able to specify if devices had to be handled through this virtual 1150IOMMU or not. In particular, the `--disk` option now expects disk paths to be 1151prefixed with a `path=` string, and supports an optional `iommu=[on|off]` 1152setting. 1153 1154### Ubuntu 19.10 1155 1156With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware), 1157we can now support the latest 1158[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images. 1159 1160### Large memory guests 1161 1162After simplifying and changing our guest address space handling, we can now 1163support guests with large amount of memory (more than 64GB). 1164 1165# v0.2.0 1166 1167This release has been tracked through the [0.2.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/2). 1168 1169Highlights for `cloud-hypervisor` version 0.2.0 include: 1170 1171### Network device offloading 1172 1173As part of our general effort to offload paravirtualized I/O to external 1174processes, we added support for 1175[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This 1176enables `cloud-hypervisor` users to plug a `vhost-user` based networking device 1177(e.g. [DPDK](https://dpdk.org)) into the VMM as their virtio network backend. 1178 1179### Minimal hardware-reduced ACPI 1180 1181In order to properly implement and guest reset and shutdown, we implemented 1182a minimal version of the hardware-reduced ACPI specification. Together with 1183a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to 1184cleanly reboot and shutdown. 1185 1186The ACPI implementation is a `cloud-hypervisor` build time option that is 1187enabled by default. 1188 1189### Debug I/O port 1190 1191Based on the Firecracker idea of using a dedicated I/O port to measure guest 1192boot times, we added support for logging guest events through the 1193[0x80](https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html) 1194PC debug port. This allows, among other things, for granular guest boot time 1195measurements. See our [debug port documentation](docs/debug-port.md) for more 1196details. 1197 1198### Improved direct device assignment 1199 1200We fixed a major performance issue with our initial VFIO implementation: When 1201enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads 1202were (in many cases) not cached. After correctly tagging the guest memory from 1203`cloud-hypervisor` we're now able to reach the expected performance from 1204directly assigned devices. 1205 1206### Improved shared filesystem 1207 1208We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt) 1209support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system. 1210This provides better shared filesystem IO performance with a smaller guest 1211memory footprint. 1212 1213### Ubuntu bionic based CI 1214 1215Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) 1216improvements, we are now able to boot Ubuntu bionic images. We added those to 1217our CI pipeline. 1218 1219# v0.1.0 1220 1221This release has been tracked through the [0.1.0 project](https://github.com/cloud-hypervisor/cloud-hypervisor/projects/1). 1222 1223Highlights for `cloud-hypervisor` version 0.1.0 include: 1224 1225### Shared filesystem 1226 1227We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file 1228system, allowing for an efficient and reliable way of sharing a filesystem 1229between the host and the `cloud-hypervisor` guest. 1230 1231See our [filesystem sharing](docs/fs.md) documentation for more details on how 1232to use virtio-fs with `cloud-hypervisor`. 1233 1234### Initial direct device assignment support 1235 1236VFIO (Virtual Function I/O) is a kernel framework that exposes direct device 1237access to userspace. `cloud-hypervisor` uses VFIO to directly assign host 1238physical devices into its guest. 1239 1240See our [VFIO](docs/vfio.md) documentation for more detail on how to directly 1241assign host devices to `cloud-hypervisor` guests. 1242 1243### Userspace IOAPIC 1244 1245`cloud-hypervisor` supports a so-called split IRQ chip implementation by 1246implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC). 1247By moving part of the IRQ chip implementation from kernel space to user space, 1248the IRQ chip emulation does not always run in a fully privileged mode. 1249 1250### Virtual persistent memory 1251 1252The `virtio-pmem` implementation emulates a virtual persistent memory device 1253that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device 1254allows to bypass the guest page cache and improve the guest memory footprint. 1255 1256### Linux kernel bzImage 1257 1258The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from 1259`bzImage` kernel images, which is usually the format that Linux distributions 1260use to ship their kernels. For example, this allows for booting from the host 1261distribution kernel image. 1262 1263### Console over virtio 1264 1265`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although 1266using this device as a guest console can potentially cut some early boot 1267messages, it can reduce the guest boot time and provides a complete console 1268implementation. 1269 1270The `virtio-console` device is enabled by default for the guest console. 1271Switching back to the legacy serial port is done by selecting 1272`--serial tty --console off` from the command line. 1273 1274### Unit testing 1275 1276We now run all unit tests from all our crates directly from our CI. 1277 1278### Integration tests parallelization 1279 1280The CI cycle run time has been significantly reduced by refactoring our 1281integration tests; allowing them to all be run in parallel. 1282