devel/migration/vfio.rst

2 VFIO device migration
5 Migration of virtual machine involves saving the state for each device that
14 continue whilst the VFIO device state is transferred to the destination, this
29 the device, but the device can respond to incoming ones. Additionally, all
31 the device enters this state.
39 support P2P migration. Single VFIO device migration is allowed regardless of
42 A detailed description of the UAPI for VFIO device migration can be found in
46 VFIO implements the device hooks for the iterative approach as follows:
50 * A ``load_setup`` function that sets the VFIO device on the destination in
55   device.
59   save for the VFIO device.
62   active only when the VFIO device is in pre-copy states.
64 * A ``save_live_iterate`` function that reads the VFIO device's data from the
67 * A ``switchover_ack_needed`` function that checks if the VFIO device uses
71   reassembles the multifd received data and loads it in-order into the device.
74 * A ``save_state`` function to save the device config space if it is present
78 * A ``save_live_complete_precopy`` function that sets the VFIO device in
79   _STOP_COPY state and iteratively copies the data for the VFIO device until
84   provides thread handler performing multifd device state transfer.
85   It sets the VFIO device to _STOP_COPY state, iteratively reads the data
86   from the VFIO device and queues it for multifd transmission until the vendor
88   After that, it saves the device config space and queues it for multifd
95 * A ``load_state_buffer`` function that loads the device state and the device
104 device state when the VM state changes from running to not-running, and
108 the VFIO device state when certain changes of the migration state occur. For
109 example, the VFIO device state is transitioned back to _RUNNING in case a
119 VFIO device as dirty. The dirty page bitmap is queried per container.
122 (1) Device dirty tracking:
123 In this method the device is responsible to log and report its DMAs. This
124 method can be used only if the device is capable of tracking its DMAs.
125 Discovering device capability, starting and stopping dirty tracking, and
126 syncing the dirty bitmaps from the device are done using the DMA logging uAPI.
135 perpetually marked dirty, unless the device driver pins pages through external
148 QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
161 mapped ranges. If device dirty tracking is enabled with vIOMMU, live migration
167 Below is the state change flow during live migration for a VFIO device that
172 the VFIO device state, respectively.
183             Migration thread then calls each device's .save_setup()
187   If device is active, get pending_bytes by .state_pending_{estimate,exact}()
189                 Data of VFIO device for pre-copy phase is copied
192        On migration completion, the vCPUs and the VFIO device are stopped
193               The VFIO device is first put in P2P quiescent state
196                 Then the VFIO device is put in _STOP_COPY state
198          .save_live_complete_precopy() is called for each active device
199               For the VFIO device: in the non-multifd mode iterate in
216              Incoming migration calls .load_setup() for each device
219      For each device, .load_state() is called for that device section data
225   At the end, .load_cleanup() is called for each device and vCPUs are started
226               The VFIO device is first put in P2P quiescent state
239 Starting from QEMU version 10.0 there's a possibility to transfer VFIO device
242 As an additional benefit, setting the VFIO device to _STOP_COPY state and
246 The multifd VFIO device state transfer is controlled by
247 "x-migration-multifd-transfer" VFIO device property. This property defaults to
248 AUTO, which means that VFIO device state transfer via multifd channels is