devel/migration/vfio.rst

7 destination host. This document details how saving and restoring of VFIO
10 Migration of VFIO devices consists of two phases: the optional pre-copy phase,
11 and the stop-and-copy phase. The pre-copy phase is iterative and allows to
13 transferred. The iterative pre-copy phase of migration allows for the guest to
14 continue whilst the VFIO device state is transferred to the destination, this
15 helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy
19 When pre-copy is supported, it's possible to further reduce downtime by
20 enabling "switchover-ack" migration capability.
21 VFIO migration uAPI defines "initial bytes" as part of its pre-copy data stream
22 and recommends that the initial bytes are sent and loaded in the destination
35 safe P2P-wise, since starting and stopping the devices is not done atomically
44 linux-headers/linux/vfio.h.
50 * A ``load_setup`` function that sets the VFIO device on the destination in
54   remaining pre-copy data that the vendor driver has yet to save for the VFIO
62   active only when the VFIO device is in pre-copy states.
65   vendor driver during iterative pre-copy phase.
68   "switchover-ack" migration capability when this capability is enabled.
71   reassembles the multifd received data and loads it in-order into the device.
72   In the non-multifd mode this function is a NOP.
75   in the non-multifd mode.
90   In the non-multifd mode this thread is a NOP.
104 device state when the VM state changes from running to not-running, and
113 ----------------------------------
118 dirty tracking module and marks system memory pages which were DMA-ed by the
130 linux-headers/linux/vfio.h.
141 By default, dirty pages are tracked during pre-copy as well as stop-and-copy
142 phase. So, a page marked as dirty will be copied to the destination in both
143 phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can
144 achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding
145 dirty pages continuously, then it understands that even in stop-and-copy phase,
148 QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
149 which disables querying the dirty bitmap during pre-copy phase. If it is set to
150 off, all dirty pages will be copied to the destination in stop-and-copy phase
154 ---------------------------------------------------------
156 With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
159 stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
175 ------------------------
189                 Data of VFIO device for pre-copy phase is copied
199               For the VFIO device: in the non-multifd mode iterate in
212 --------------------------
240 _STOP_COPY state via multifd channels. This helps reduce downtime - especially
247 "x-migration-multifd-transfer" VFIO device property. This property defaults to