devel/migration/vfio.rst

10 Migration of VFIO devices consists of two phases: the optional pre-copy phase,
11 and the stop-and-copy phase. The pre-copy phase is iterative and allows to
13 transferred. The iterative pre-copy phase of migration allows for the guest to
15 helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy
19 When pre-copy is supported, it's possible to further reduce downtime by
20 enabling "switchover-ack" migration capability.
21 VFIO migration uAPI defines "initial bytes" as part of its pre-copy data stream
27 themselves, VFIO migration uAPI defines an intermediate P2P quiescent state.
35 safe P2P-wise, since starting and stopping the devices is not done atomically
42 A detailed description of the UAPI for VFIO device migration can be found in
43 the comment for the ``vfio_device_mig_state`` structure in the header file
44 linux-headers/linux/vfio.h.
54   remaining pre-copy data that the vendor driver has yet to save for the VFIO
62   active only when the VFIO device is in pre-copy states.
65   vendor driver during iterative pre-copy phase.
68   "switchover-ack" migration capability when this capability is enabled.
71   reassembles the multifd received data and loads it in-order into the device.
72   In the non-multifd mode this function is a NOP.
75   in the non-multifd mode.
90   In the non-multifd mode this thread is a NOP.
104 device state when the VM state changes from running to not-running, and
113 ----------------------------------
118 dirty tracking module and marks system memory pages which were DMA-ed by the
126 syncing the dirty bitmaps from the device are done using the DMA logging uAPI.
127 More info about the uAPI can be found in the comments of the
129 ``vfio_device_feature_dma_logging_report`` structures in the header file
130 linux-headers/linux/vfio.h.
141 By default, dirty pages are tracked during pre-copy as well as stop-and-copy
143 phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can
144 achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding
145 dirty pages continuously, then it understands that even in stop-and-copy phase,
148 QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
149 which disables querying the dirty bitmap during pre-copy phase. If it is set to
150 off, all dirty pages will be copied to the destination in stop-and-copy phase
154 ---------------------------------------------------------
156 With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
159 stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
175 ------------------------
189                 Data of VFIO device for pre-copy phase is copied
199               For the VFIO device: in the non-multifd mode iterate in
212 --------------------------
240 _STOP_COPY state via multifd channels. This helps reduce downtime - especially
247 "x-migration-multifd-transfer" VFIO device property. This property defaults to