1Intel Graphics Device (IGD) assignment with vfio-pci 2==================================================== 3 4Using vfio-pci, we can passthrough Intel Graphics Device (IGD) to guest, either 5serve as primary and exclusive graphics adapter, or used in combination with an 6emulated primary graphics device, depending on the config and guest driver 7support. However, IGD devices are not "clean" PCI devices, they use extra 8memory regions other than BARs. Special handling is required to make them work 9properly, including: 10 11* OpRegion for accessing Virtual BIOS Table (VBT) that contains display output 12 information. 13* Data Stolen Memory (DSM) region used as VRAM at early stage (BIOS/UEFI) 14 15Certain guest software also depends on following conditions to work: 16(*-Required by) 17 18| Condition | Linux | Windows | VBIOS | EFI GOP | 19|---------------------------------------------|-------|---------|-------|---------| 20| #1 IGD has a valid OpRegion containing VBT | * ^1 | * | * | * | 21| #2 VID/DID of LPC bridge at 00:1f.0 matches | | | * | * | 22| #3 IGD is assigned to BDF 00:02.0 | | | * | * | 23| #4 IGD has VGA controller device class | | | * | * | 24| #5 Host's VGA ranges are mapped to IGD | | | * | | 25| #6 Guest has valid VBIOS or UEFI Option ROM | | | * | * | 26 27^1 Though i915 driver is able to mock a OpRegion, it is still recommended to 28 use the VBT copied from host OpRegion to prevent incorrect configuration. 29 30For #1, the "x-igd-opregion=on" option exposes a copy of host IGD OpRegion to 31guest via fw_cfg, where guest firmware can set up guest OpRegion with it. 32 33For #2, "x-igd-lpc=on" option copies the IDs of host LPC bridge and host bridge 34to guest. Currently this is only supported on i440fx machines as there is 35already an ICH9 LPC bridge present on q35 machines, overwriting its IDs may 36lead to unexpected behavior. 37 38For #3, "addr=2.0" assigns IGD to 00:02.0. 39 40For #4, the primary display must be set to IGD in host BIOS. 41 42For #5, "x-vga=on" enables guest access to standard VGA IO/MMIO ranges. 43 44For #6, ROM either provided via the ROM BAR or romfile= option is needed, this 45Intel document [1] shows how to dump VBIOS to file. For UEFI Option ROM, see 46"Guest firmware" section. 47 48QEMU also provides a "Legacy" mode that implicitly enables full functionality 49on IGD, it is automatically enabled when 50* Machine type is i440fx 51* IGD is assigned to guest BDF 00:02.0 52* ROM BAR or romfile is present 53 54In "Legacy" mode, QEMU will automatically setup OpRegion, LPC bridge IDs and 55VGA range access, which is equivalent to: 56 x-igd-opregion=on,x-igd-lpc=on,x-vga=on 57 58By default, "Legacy" mode won't fail, it continues on error. User can set 59"x-igd-legacy-mode=on" to force enabling legacy mode, this also checks if the 60conditions above for legacy mode is met, and if any error occurs, QEMU will 61fail immediately. Users can also set "x-igd-legacy-mode=off" to disable legacy 62mode. 63 64In legacy mode, as the guest VGA ranges are assigned to IGD device, all other 65graphics devices should be removed, this can be done using "-nographic" or 66"-vga none" or "-nodefaults", along with adding the device using vfio-pci. 67 68For either mode, depending on the host kernel, the i915 driver in the host 69may generate faults and errors upon re-binding to an IGD device after it 70has been assigned to a VM. It's therefore generally recommended to prevent 71such driver binding unless the host driver is known to work well for this. 72There are numerous ways to do this, i915 can be blacklisted on the host, 73the driver_override option can be used to ensure that only vfio-pci can bind 74to the device on the host[2], virsh nodedev-detach can be used to bind the 75device to vfio drivers and then managed='no' set in the VM xml to prevent 76re-binding to i915, etc. Also note that IGD is also typically the primary 77graphics in the host and special options may be required beyond simply 78blacklisting i915 or using pci-stub/vfio-pci to take ownership of IGD as a 79PCI class device. Lower level drivers exist that may still claim the device. 80It may therefore be necessary to use kernel boot options video=vesafb:off or 81video=efifb:off (depending on host BIOS/UEFI) or these can be combined to 82a catch-all, video=vesafb:off,efifb:off. Error messages such as: 83 84 Failed to mmap 0000:00:02.0 BAR <>. Performance may be slow 85 86are a good indicator that such a problem exists. The host files /proc/iomem 87and /proc/ioports are often useful for identifying drivers consuming ranges 88of the device to cause such conflicts. 89 90Additionally, IGD device are known to generate small numbers of DMAR faults 91when initially assigned. It is believed that this is simply the IGD attempting 92to access the reserved GTT space after reset, which it no longer has access to 93when accessed from userspace. So long as the DMAR faults are small in number 94and most importantly, not ongoing, these are not an indication of an error. 95 96Additionally++, analog VGA output (as opposed to digital outputs like HDMI, 97DVI, or DisplayPort) may be unsupported in some use cases. In the author's 98experience, even DP to VGA adapters can be troublesome while adapters between 99digital formats work well. 100 101 102Options 103======= 104* x-igd-opregion=[on|*off*] 105 Copy host IGD OpRegion and expose it to guest with fw_cfg 106 107* x-igd-lpc=[on|*off*] 108 Creates a dummy LPC bridge at 00:1f:0 with host VID/DID (i440fx only) 109 110* x-igd-legacy-mode=[on|off|*auto*] 111 Enable/Disable legacy mode 112 113* x-igd-gms=[hex, default 0] 114 Overriding DSM region size in GGC register, 0 means uses host value. 115 Use this only when the DSM size cannot be changed through the 116 'DVMT Pre-Allocated' option in host BIOS. 117 118 119Examples 120======== 121* Adding IGD with automatically legacy mode support 122 -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0 123 124* Adding IGD with OpRegion and LPC ID hack, but without VGA ranges 125 (For UEFI guests) 126 -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0,x-igd-legacy-mode=off,x-igd-opregion=on,x-igd-lpc=on,romfile=efi_oprom.rom 127 128 129Guest firmware 130============== 131Guest firmware is responsible for setting up OpRegion and Base of Data Stolen 132Memory (BDSM) in guest address space. IGD passthrough support imposes two 133fw_cfg requirements on the VM firmware: 134 1351) "etc/igd-opregion" 136 137 This fw_cfg file exposes the OpRegion for the IGD device. A reserved 138 region should be created below 4GB (recommended 4KB alignment), sized 139 sufficient for the fw_cfg file size, and the content of this file copied 140 to it. The dword based address of this reserved memory region must also 141 be written to the ASLS register at offset 0xFC on the IGD device. It is 142 recommended that firmware should make use of this fw_cfg entry for any 143 PCI class VGA device with Intel vendor ID. Multiple of such devices 144 within a VM is undefined. 145 1462) "etc/igd-bdsm-size" 147 148 This fw_cfg file contains an 8-byte, little endian integer indicating 149 the size of the reserved memory region required for IGD stolen memory. 150 Firmware must allocate a reserved memory below 4GB with required 1MB 151 alignment equal to this size. Additionally the base address of this 152 reserved region must be written to the dword BDSM register in PCI config 153 space of the IGD device at offset 0x5C (or 0xC0 for Gen 11+ devices using 154 64-bit BDSM). As this support is related to running the IGD ROM, which 155 has other dependencies on the device appearing at guest address 00:02.0, 156 it's expected that this fw_cfg file is only relevant to a single PCI 157 class VGA device with Intel vendor ID, appearing at PCI bus address 00:02.0. 158 159Upstream Seabios has OpRegion and BDSM (pre-Gen11 device only) support. 160However, the support is not accepted by upstream EDK2/OVMF. A recommended 161solution is to create a virtual OpRom with following DXE drivers: 162 163* IgdAssignmentDxe: Set up OpRegion and BDSM according to fw_cfg (must) 164* IntelGopDriver: Closed-source Intel GOP driver 165* PlatformGopPolicy: Protocol required by IntelGopDriver 166 167IntelGopDriver and PlatformGopPolicy is only required when enabling GOP on IGD. 168 169The original IgdAssignmentDxe can be found at [3]. A Intel maintained version 170with PlatformGopPolicy for industrial computing is at [4]. There is also an 171unofficially maintained version with newer Gen11+ device support at [5]. 172You need to build them with EDK2. 173 174For the IntelGopDriver, Intel never released it to public. You may contact 175Intel support to get one as [4] said, if you are an Intel Premier Support 176customer, or you can try extracting it from your host firmware using 177"UEFI BIOS Updater"[6]. 178 179Once you got all the required DXE drivers, a Option ROM can be generated with 180EfiRom utility in EDK2, using 181 EfiRom -f 0x8086 -i <Device ID of your IGD> -o output.rom \ 182 -e IgdAssignmentDxe.efi PlatformGOPPolicy.efi IntelGopDriver.efi 183 184 185Known issues 186============ 187When using OVMF as guest firmware, you may encounter the following warning: 188warning: vfio_container_dma_map(0x55fab36ce610, 0x380010000000, 0x108000, 0x7fd336000000) = -22 (Invalid argument) 189 190Solution: 191Set the host physical address bits to IOMMU address width using 192 -cpu host,host-phys-bits-limit=<IOMMU address width> 193Or in libvirt XML with 194 <cpu> 195 <maxphysaddr mode='passthrough' limit='<IOMMU address width>'/> 196 </cpu> 197The IOMMU address width can be determined with 198 echo $(( ((0x$(cat /sys/devices/virtual/iommu/dmar0/intel-iommu/cap) & 0x3F0000) >> 16) + 1 )) 199Refer https://edk2.groups.io/g/devel/topic/patch_v1/102359124 for more details 200 201 202Memory View 203=========== 204IGD has it own address space. To use system RAM as VRAM, a single-level page 205table named Global Graphics Translation Table (GTT) is used for the address 206translation. Each page table entry points a 4KB page. Illustration below shows 207the translation flow on IGD with 64-bit GTT PTEs. 208 209(PTE_SIZE == 8) +-------------+---+ 210 | Address | V | V: Valid Bit 211 +-------------+---+ 212 | ... | | 213IGD:0x01ae9010 0xd740| 0x70ffc000 | 1 | Mem:0x42ba3e010^ 214-----------------------> 0xd748| 0x42ba3e000 | 1 +------------------> 215(addr >> 12) * PTE_SIZE 0xd750| 0x42ba3f000 | 1 | 216 | ... | | 217 +-------------+---+ 218^ The address may be remapped by IOMMU 219 220The memory region store GTT is called GTT Stolen Memory (GSM) it is located 221right below the Data Stolen Memory (DSM). Accessing this region directly is 222not allowed, any access will immediately freeze the whole system. The only way 223to access it is through the second half of MMIO BAR0. 224 225The Data Stolen Memory is reserved by firmware, and acts as the VRAM in pre-OS 226environments. In QEMU, guest firmware (Seabios/OVMF) is responsible for 227reserving a continuous region and program its base address to BDSM register, 228then let VBIOS/GOP driver initializing this region. Illustration below shows 229how DSM is mapped. 230 231 IGD Addr Space Host Addr Space Guest Addr Space 232 +-------------+ +-------------+ +-------------+ 233 | | | | | | 234 | | | | | | 235 | | +-------------+ +-------------+ 236 | | | Data Stolen | | Data Stolen | 237 | | | (Guest) | | (Guest) | 238 | | +------------>+-------------+<------->+-------------+<--Guest BDSM 239 | | | Passthrough | | EPT | | Emulated by QEMU 240DSMSIZE+-------------+ | with IOMMU | | Mapping | | Programmed by guest FW 241 | | | | | | | 242 | | | | | | | 243 0+-------------+--+ | | | | 244 | +-------------+ | | 245 | | Data Stolen | +-------------+ 246 | | (Host) | 247 +------------>+-------------+<--Host BDSM 248 Non- | | "real" one in HW 249 Passthrough | | Programmed by host FW 250 +-------------+ 251 252Footnotes 253========= 254[1] https://www.intel.com/content/www/us/en/docs/graphics-for-linux/developer-reference/1-0/dump-video-bios.html 255[2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override 256[3] https://web.archive.org/web/20240827012422/https://bugzilla.tianocore.org/show_bug.cgi?id=935 257 Tianocore bugzilla was down since Jan 2025 :( 258[4] https://eci.intel.com/docs/3.3/components/kvm-hypervisor.html, Patch 0001-0004 259[5] https://github.com/tomitamoeko/VfioIgdPkg 260[6] https://winraid.level1techs.com/t/tool-guide-news-uefi-bios-updater-ubu/30357 261