1===================== 2VFIO device migration 3===================== 4 5Migration of virtual machine involves saving the state for each device that 6the guest is running on source host and restoring this saved state on the 7destination host. This document details how saving and restoring of VFIO 8devices is done in QEMU. 9 10Migration of VFIO devices consists of two phases: the optional pre-copy phase, 11and the stop-and-copy phase. The pre-copy phase is iterative and allows to 12accommodate VFIO devices that have a large amount of data that needs to be 13transferred. The iterative pre-copy phase of migration allows for the guest to 14continue whilst the VFIO device state is transferred to the destination, this 15helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy 16support by reporting the VFIO_MIGRATION_PRE_COPY flag in the 17VFIO_DEVICE_FEATURE_MIGRATION ioctl. 18 19When pre-copy is supported, it's possible to further reduce downtime by 20enabling "switchover-ack" migration capability. 21VFIO migration uAPI defines "initial bytes" as part of its pre-copy data stream 22and recommends that the initial bytes are sent and loaded in the destination 23before stopping the source VM. Enabling this migration capability will 24guarantee that and thus, can potentially reduce downtime even further. 25 26To support migration of multiple devices that might do P2P transactions between 27themselves, VFIO migration uAPI defines an intermediate P2P quiescent state. 28While in the P2P quiescent state, P2P DMA transactions cannot be initiated by 29the device, but the device can respond to incoming ones. Additionally, all 30outstanding P2P transactions are guaranteed to have been completed by the time 31the device enters this state. 32 33All the devices that support P2P migration are first transitioned to the P2P 34quiescent state and only then are they stopped or started. This makes migration 35safe P2P-wise, since starting and stopping the devices is not done atomically 36for all the devices together. 37 38Thus, multiple VFIO devices migration is allowed only if all the devices 39support P2P migration. Single VFIO device migration is allowed regardless of 40P2P migration support. 41 42A detailed description of the UAPI for VFIO device migration can be found in 43the comment for the ``vfio_device_mig_state`` structure in the header file 44linux-headers/linux/vfio.h. 45 46VFIO implements the device hooks for the iterative approach as follows: 47 48* A ``save_setup`` function that sets up migration on the source. 49 50* A ``load_setup`` function that sets the VFIO device on the destination in 51 _RESUMING state. 52 53* A ``state_pending_estimate`` function that reports an estimate of the 54 remaining pre-copy data that the vendor driver has yet to save for the VFIO 55 device. 56 57* A ``state_pending_exact`` function that reads pending_bytes from the vendor 58 driver, which indicates the amount of data that the vendor driver has yet to 59 save for the VFIO device. 60 61* An ``is_active_iterate`` function that indicates ``save_live_iterate`` is 62 active only when the VFIO device is in pre-copy states. 63 64* A ``save_live_iterate`` function that reads the VFIO device's data from the 65 vendor driver during iterative pre-copy phase. 66 67* A ``switchover_ack_needed`` function that checks if the VFIO device uses 68 "switchover-ack" migration capability when this capability is enabled. 69 70* A ``switchover_start`` function that in the multifd mode starts a thread that 71 reassembles the multifd received data and loads it in-order into the device. 72 In the non-multifd mode this function is a NOP. 73 74* A ``save_state`` function to save the device config space if it is present 75 in the non-multifd mode. 76 In the multifd mode it just emits either a dummy EOS marker. 77 78* A ``save_live_complete_precopy`` function that sets the VFIO device in 79 _STOP_COPY state and iteratively copies the data for the VFIO device until 80 the vendor driver indicates that no data remains. 81 In the multifd mode it just emits a dummy EOS marker. 82 83* A ``save_live_complete_precopy_thread`` function that in the multifd mode 84 provides thread handler performing multifd device state transfer. 85 It sets the VFIO device to _STOP_COPY state, iteratively reads the data 86 from the VFIO device and queues it for multifd transmission until the vendor 87 driver indicates that no data remains. 88 After that, it saves the device config space and queues it for multifd 89 transfer too. 90 In the non-multifd mode this thread is a NOP. 91 92* A ``load_state`` function that loads the config section and the data 93 sections that are generated by the save functions above. 94 95* A ``load_state_buffer`` function that loads the device state and the device 96 config that arrived via multifd channels. 97 It's used only in the multifd mode. 98 99* ``cleanup`` functions for both save and load that perform any migration 100 related cleanup. 101 102 103The VFIO migration code uses a VM state change handler to change the VFIO 104device state when the VM state changes from running to not-running, and 105vice versa. 106 107Similarly, a migration state change handler is used to trigger a transition of 108the VFIO device state when certain changes of the migration state occur. For 109example, the VFIO device state is transitioned back to _RUNNING in case a 110migration failed or was canceled. 111 112System memory dirty pages tracking 113---------------------------------- 114 115A ``log_global_start`` and ``log_global_stop`` memory listener callback informs 116the VFIO dirty tracking module to start and stop dirty page tracking. A 117``log_sync`` memory listener callback queries the dirty page bitmap from the 118dirty tracking module and marks system memory pages which were DMA-ed by the 119VFIO device as dirty. The dirty page bitmap is queried per container. 120 121Currently there are two ways dirty page tracking can be done: 122(1) Device dirty tracking: 123In this method the device is responsible to log and report its DMAs. This 124method can be used only if the device is capable of tracking its DMAs. 125Discovering device capability, starting and stopping dirty tracking, and 126syncing the dirty bitmaps from the device are done using the DMA logging uAPI. 127More info about the uAPI can be found in the comments of the 128``vfio_device_feature_dma_logging_control`` and 129``vfio_device_feature_dma_logging_report`` structures in the header file 130linux-headers/linux/vfio.h. 131 132(2) VFIO IOMMU module: 133In this method dirty tracking is done by IOMMU. However, there is currently no 134IOMMU support for dirty page tracking. For this reason, all pages are 135perpetually marked dirty, unless the device driver pins pages through external 136APIs in which case only those pinned pages are perpetually marked dirty. 137 138If the above two methods are not supported, all pages are perpetually marked 139dirty by QEMU. 140 141By default, dirty pages are tracked during pre-copy as well as stop-and-copy 142phase. So, a page marked as dirty will be copied to the destination in both 143phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can 144achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding 145dirty pages continuously, then it understands that even in stop-and-copy phase, 146it is likely to find dirty pages and can predict the downtime accordingly. 147 148QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking`` 149which disables querying the dirty bitmap during pre-copy phase. If it is set to 150off, all dirty pages will be copied to the destination in stop-and-copy phase 151only. 152 153System memory dirty pages tracking when vIOMMU is enabled 154--------------------------------------------------------- 155 156With vIOMMU, an IO virtual address range can get unmapped while in pre-copy 157phase of migration. In that case, the unmap ioctl returns any dirty pages in 158that range and QEMU reports corresponding guest physical pages dirty. During 159stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped 160pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those 161mapped ranges. If device dirty tracking is enabled with vIOMMU, live migration 162will be blocked. 163 164Flow of state changes during Live migration 165=========================================== 166 167Below is the state change flow during live migration for a VFIO device that 168supports both precopy and P2P migration. The flow for devices that don't 169support it is similar, except that the relevant states for precopy and P2P are 170skipped. 171The values in the parentheses represent the VM state, the migration state, and 172the VFIO device state, respectively. 173 174Live migration save path 175------------------------ 176 177:: 178 179 QEMU normal running state 180 (RUNNING, _NONE, _RUNNING) 181 | 182 migrate_init spawns migration_thread 183 Migration thread then calls each device's .save_setup() 184 (RUNNING, _SETUP, _PRE_COPY) 185 | 186 (RUNNING, _ACTIVE, _PRE_COPY) 187 If device is active, get pending_bytes by .state_pending_{estimate,exact}() 188 If total pending_bytes >= threshold_size, call .save_live_iterate() 189 Data of VFIO device for pre-copy phase is copied 190 Iterate till total pending bytes converge and are less than threshold 191 | 192 On migration completion, the vCPUs and the VFIO device are stopped 193 The VFIO device is first put in P2P quiescent state 194 (FINISH_MIGRATE, _ACTIVE, _PRE_COPY_P2P) 195 | 196 Then the VFIO device is put in _STOP_COPY state 197 (FINISH_MIGRATE, _ACTIVE, _STOP_COPY) 198 .save_live_complete_precopy() is called for each active device 199 For the VFIO device: in the non-multifd mode iterate in 200 .save_live_complete_precopy() until 201 pending data is 0 202 In the multifd mode this iteration is done in 203 .save_live_complete_precopy_thread() instead. 204 | 205 (POSTMIGRATE, _COMPLETED, _STOP_COPY) 206 Migraton thread schedules cleanup bottom half and exits 207 | 208 .save_cleanup() is called 209 (POSTMIGRATE, _COMPLETED, _STOP) 210 211Live migration resume path 212-------------------------- 213 214:: 215 216 Incoming migration calls .load_setup() for each device 217 (RESTORE_VM, _ACTIVE, _STOP) 218 | 219 For each device, .load_state() is called for that device section data 220 transmitted via the main migration channel. 221 For data transmitted via multifd channels .load_state_buffer() is called 222 instead. 223 (RESTORE_VM, _ACTIVE, _RESUMING) 224 | 225 At the end, .load_cleanup() is called for each device and vCPUs are started 226 The VFIO device is first put in P2P quiescent state 227 (RUNNING, _ACTIVE, _RUNNING_P2P) 228 | 229 (RUNNING, _NONE, _RUNNING) 230 231Postcopy 232======== 233 234Postcopy migration is currently not supported for VFIO devices. 235 236Multifd 237======= 238 239Starting from QEMU version 10.0 there's a possibility to transfer VFIO device 240_STOP_COPY state via multifd channels. This helps reduce downtime - especially 241with multiple VFIO devices or with devices having a large migration state. 242As an additional benefit, setting the VFIO device to _STOP_COPY state and 243saving its config space is also parallelized (run in a separate thread) in 244such migration mode. 245 246The multifd VFIO device state transfer is controlled by 247"x-migration-multifd-transfer" VFIO device property. This property defaults to 248AUTO, which means that VFIO device state transfer via multifd channels is 249attempted in configurations that otherwise support it. 250