docs/devel/multi-process.rst

1 Multi-process QEMU
6   This is the design document for multi-process QEMU. It does not
31 -------------
34 VM control point, where VMs can be created, migrated, re-configured, and
40 A multi-process QEMU
43 A multi-process QEMU involves separating QEMU services into separate
51 A QEMU control process would remain, but in multi-process mode, will
53 provide the user interface to hot-plug devices or live migrate the VM.
55 A first step in creating a multi-process QEMU is to separate IO services
62 ----------------------
82 VM. (e.g., via the QEMU command line as *-device foo*)
85 parent object (such as "pci-device" for a PCI device) and QEMU will
104 mission-mode IO is performed by the application. The vhost user
121 VMs are often run using HW virtualization features via the KVM kernel
125 execution returns to the KVM driver so it can inform QEMU to emulate the
134 application can directly receive MMIO store notifications from the KVM
141 directly inject interrupts into the VM via the KVM driver, again,
143 The QEMU virtio setup code configures the KVM driver with an eventfd
168 Much of the vhost model can be re-used by separated device emulation. In
171 the VM via KVM, and allowing the application to ``mmap()`` the guest
179 break vhost store acceleration since they are synchronous - guest
188 #### qemu-io model
190 ``qemu-io`` is a test harness used to test changes to the QEMU block backend
192 emulation). ``qemu-io`` is not a device emulation application per se, but it
198 -------------------------------------------
212 a tractable to re-implement both the object model and the many device
223     disk-proc -blockdev driver=file,node-name=file0,filename=disk-file0  \
224     -blockdev driver=qcow2,node-name=drive0,file=file0
226 would indicate process *disk-proc* uses a qcow2 emulated disk named
242     disk-proc <socket number> <backend list>
253     disk-proc -qmp unix:/tmp/disk-mon,server
255 can be monitored over the UNIX socket path */tmp/disk-mon*.
261 represented as a *-device* of type *pci-proxy-dev*. A socket
262 sub-option to this option specifies the Unix socket that connects
263 to the remote process. An *id* sub-option is required, and it should
268     qemu-system-x86_64 ... -device pci-proxy-dev,id=lsi0,socket=3
286 the remote process. It is also used to pass on device-agnostic commands
289 per-device channels
299 QEMU has an object model based on sub-classes inherited from the
300 "object" super-class. The sub-classes that are of interest here are the
301 "device" and "bus" sub-classes whose child sub-classes make up the
308 sub-class of the "pci-device" class, and will have the same PCI bus
322 - Parses the "socket" sub option and connects to the remote process
324 - Uses the "id" sub-option to connect to the emulated device on the
342 Other tasks will be device-specific. For example, PCI device objects
368 read-only, but certain registers (especially BAR and MSI-related ones)
375 "pci-device-proxy" class that can serve as the parent of a PCI device
376 proxy object. This class's parent would be "pci-device" and it would
412 must be backed by shared file-backed memory, for example, using
413 *-object memory-backend-file,share=on* and setting that memory backend
429 device hot-plug via QMP
434 *-device* command line option does. The remote process may either be one
435 started at QEMU startup, or be one added by the "add-process" QMP
460 handle requests from the QEMU process, and route machine-level requests
471 -  address spaces
478 -  RAM
487 -  PCI
517 -  PCI pin interrupts
521 it. The IOAPIC object, in turn, calls the KVM driver to inject the
528 -  PCI MSI/X interrupts
531 CPU-specific PCI address. In QEMU on x86, a KVM APIC object receives
532 these DMA writes, then calls into the KVM driver to inject the interrupt
555 -  IOTLB cache
564 -  IOTLB purge
578 restore - the channel will be passed to ``qemu_loadvm_state()`` to
587 emulation process to communicate directly with the kernel KVM driver.
588 The KVM file descriptors created would be passed to the emulation process
593 from KVM. The issue with the eventfd mechanism used by vhost user is
598 The expanded idea would require a new type of KVM device:
599 *KVM\_DEV\_TYPE\_USER*. This device has two file descriptors: a master
602 would create both descriptors using the KVM driver, and pass the slave
608 -  guest physical range
620 +--------+----------------------------+
624 +--------+----------------------------+
626 +--------+----------------------------+
628 +--------+----------------------------+
630 +--------+----------------------------+
632 -  MMIO request structure
640 +----------+------------------------+
644 +----------+------------------------+
646 +----------+------------------------+
648 +----------+------------------------+
650 +----------+------------------------+
652 +----------+------------------------+
654 +----------+------------------------+
656 -  MMIO request queues
664 -  scoreboard
669 wait queue and sequence number for the per-CPU threads, allowing them to
676 -  device shadow memory
678 Some MMIO loads do not have device side-effects. These MMIOs can be
681 with the KVM driver.
683 The emulation program will ask the KVM driver to allocate memory for the
685 emulation program can control KVM access to the shadow image by sending
686 KVM an access map telling it which areas of the image have no
687 side-effects (and can be completed immediately), and which require a
689 the KVM drive which size accesses are allowed to the image.
694 The master descriptor is used by QEMU to configure the new KVM device.
695 The descriptor would be returned by the KVM driver when QEMU issues a
696 *KVM\_CREATE\_DEVICE* ``ioctl()`` with a *KVM\_DEV\_TYPE\_USER* type.
698 KVM\_DEV\_TYPE\_USER device ops
701 The *KVM\_DEV\_TYPE\_USER* operations vector will be registered by a
702 ``kvm_register_device_ops()`` call when the KVM system in initialized by
703 ``kvm_init()``. These device ops are called by the KVM driver when QEMU
704 executes certain ``ioctl()`` operations on its KVM file descriptor. They
707 -  create
709 This routine is called when QEMU issues a *KVM\_CREATE\_DEVICE*
710 ``ioctl()`` on its per-VM file descriptor. It will allocate and
711 initialize a KVM user device specific data structure, and assign the
712 *kvm\_device* private field to it.
714 -  ioctl
717 descriptor. The ``ioctl()`` commands supported are defined by the KVM
718 device type. *KVM\_DEV\_TYPE\_USER* ones will need several commands:
720 *KVM\_DEV\_USER\_SLAVE\_FD* creates the slave file descriptor that will
725 The *KVM\_DEV\_USER\_PA\_RANGE* command configures a guest physical
732 *KVM\_DEV\_USER\_PA\_RANGE* will use ``kvm_io_bus_register_dev()`` to
733 register *kvm\_io\_device\_ops* callbacks to be invoked when the guest
738 *KVM\_DEV\_USER\_TIMEOUT* will configure a timeout value that specifies
739 how long KVM will wait for the emulation process to respond to a MMIO
742 -  destroy
746 driver, as well as the *kvm\_device* structure itself.
755 -  read
757 A read returns any pending MMIO requests from the KVM driver as MMIO
764 -  write
769 removed, then the number of posted stores in the per-CPU scoreboard is
770 decremented. When the number is zero, and a non side-effect load was
773 -  ioctl
778 A *KVM\_DEV\_USER\_SHADOW\_SIZE* ``ioctl()`` causes the KVM driver to
781 device memory with the KVM driver.
783 A *KVM\_DEV\_USER\_SHADOW\_CTRL* ``ioctl()`` controls access to the
784 shadow image. It will send the KVM driver a shadow control map, which
789 -  poll
795 -  mmap
798 image allocated by the KVM driver. As device emulation updates device
799 memory, changes with no side-effects will be reflected in the shadow,
800 and the KVM driver can satisfy guest loads from the shadow image without
803 kvm\_io\_device ops
806 Each KVM per-CPU thread can handle MMIO operation on behalf of the guest
807 VM. KVM will use the MMIO's guest physical address to search for a
808 matching *kvm\_io\_device* to see if the MMIO can be handled by the KVM
812 -  read
815 Loads with side-effects must be handled synchronously, with the KVM
817 process reply before re-starting the guest. Loads that do not have
818 side-effects may be optimized by satisfying them from the shadow image,
823 -  write
828 the per-CPU scoreboard, in order to implement the PCI ordering
836 the device's corresponding interrupt to be triggered by the KVM driver.
845 irq file descriptor, a re-sampling file descriptor needs to be sent to
848 acknowledged by the guest, so they can re-trigger the interrupt if their
849 device has not de-asserted its interrupt.
855 ``using event_notifier_init()`` to create the irq and re-sampling
867 interrupt logic to change the route: de-assigning the existing irq
890 The guest may dynamically update several MSI-related tables in the
891 device's PCI config space. These include per-MSI interrupt enables and
897 --------------
900 ---------------------------
908 --------------------
946 types separate from the main QEMU process and non-disk emulation
950 and non-network emulation process, and only that type can access the