xref: /qemu/docs/igd-assign.txt (revision 072e057ed90d6bbc4f01ac04e627e63f275f57f0)
1Intel Graphics Device (IGD) assignment with vfio-pci
2====================================================
3
4Using vfio-pci, we can passthrough Intel Graphics Device (IGD) to guest, either
5serve as primary and exclusive graphics adapter, or used in combination with an
6emulated primary graphics device, depending on the config and guest driver
7support. However, IGD devices are not "clean" PCI devices, they use extra
8memory regions other than BARs. Special handling is required to make them work
9properly, including:
10
11* OpRegion for accessing Virtual BIOS Table (VBT) that contains display output
12  information.
13* Data Stolen Memory (DSM) region used as VRAM at early stage (BIOS/UEFI)
14
15Certain guest software also depends on following conditions to work:
16(*-Required by)
17
18| Condition                                   | Linux | Windows | VBIOS | EFI GOP |
19|---------------------------------------------|-------|---------|-------|---------|
20| #1 IGD has a valid OpRegion containing VBT  |  * ^1 |    *    |   *   |    *    |
21| #2 VID/DID of LPC bridge at 00:1f.0 matches |       |         |   *   |    *    |
22| #3 IGD is assigned to BDF 00:02.0           |       |         |   *   |    *    |
23| #4 IGD has VGA controller device class      |       |         |   *   |    *    |
24| #5 Host's VGA ranges are mapped to IGD      |       |         |   *   |         |
25| #6 Guest has valid VBIOS or UEFI Option ROM |       |         |   *   |    *    |
26
27^1 Though i915 driver is able to mock a OpRegion, it is still recommended to
28   use the VBT copied from host OpRegion to prevent incorrect configuration.
29
30For #1, the "x-igd-opregion=on" option exposes a copy of host IGD OpRegion to
31guest via fw_cfg, where guest firmware can set up guest OpRegion with it.
32
33For #2, "x-igd-lpc=on" option copies the IDs of host LPC bridge and host bridge
34to guest. Currently this is only supported on i440fx machines as there is
35already an ICH9 LPC bridge present on q35 machines, overwriting its IDs may
36lead to unexpected behavior.
37
38For #3, "addr=2.0" assigns IGD to 00:02.0.
39
40For #4, the primary display must be set to IGD in host BIOS.
41
42For #5, "x-vga=on" enables guest access to standard VGA IO/MMIO ranges.
43
44For #6, ROM either provided via the ROM BAR or romfile= option is needed, this
45Intel document [1] shows how to dump VBIOS to file. For UEFI Option ROM, see
46"Guest firmware" section.
47
48QEMU also provides a "Legacy" mode that implicitly enables full functionality
49on IGD, it is automatically enabled when
50* Machine type is i440fx
51* IGD is assigned to guest BDF 00:02.0
52* ROM BAR or romfile is present
53
54In "Legacy" mode, QEMU will automatically setup OpRegion, LPC bridge IDs and
55VGA range access, which is equivalent to:
56  x-igd-opregion=on,x-igd-lpc=on,x-vga=on
57
58By default, "Legacy" mode won't fail, it continues on error. User can set
59"x-igd-legacy-mode=on" to force enabling legacy mode, this also checks if the
60conditions above for legacy mode is met, and if any error occurs, QEMU will
61fail immediately. Users can also set "x-igd-legacy-mode=off" to disable legacy
62mode.
63
64In legacy mode, as the guest VGA ranges are assigned to IGD device, all other
65graphics devices should be removed, this can be done using "-nographic" or
66"-vga none" or "-nodefaults", along with adding the device using vfio-pci.
67
68For either mode, depending on the host kernel, the i915 driver in the host
69may generate faults and errors upon re-binding to an IGD device after it
70has been assigned to a VM.  It's therefore generally recommended to prevent
71such driver binding unless the host driver is known to work well for this.
72There are numerous ways to do this, i915 can be blacklisted on the host,
73the driver_override option can be used to ensure that only vfio-pci can bind
74to the device on the host[2], virsh nodedev-detach can be used to bind the
75device to vfio drivers and then managed='no' set in the VM xml to prevent
76re-binding to i915, etc.  Also note that IGD is also typically the primary
77graphics in the host and special options may be required beyond simply
78blacklisting i915 or using pci-stub/vfio-pci to take ownership of IGD as a
79PCI class device.  Lower level drivers exist that may still claim the device.
80It may therefore be necessary to use kernel boot options video=vesafb:off or
81video=efifb:off (depending on host BIOS/UEFI) or these can be combined to
82a catch-all, video=vesafb:off,efifb:off.  Error messages such as:
83
84    Failed to mmap 0000:00:02.0 BAR <>. Performance may be slow
85
86are a good indicator that such a problem exists.  The host files /proc/iomem
87and /proc/ioports are often useful for identifying drivers consuming ranges
88of the device to cause such conflicts.
89
90Additionally, IGD device are known to generate small numbers of DMAR faults
91when initially assigned.  It is believed that this is simply the IGD attempting
92to access the reserved GTT space after reset, which it no longer has access to
93when accessed from userspace.  So long as the DMAR faults are small in number
94and most importantly, not ongoing, these are not an indication of an error.
95
96Additionally++, analog VGA output (as opposed to digital outputs like HDMI,
97DVI, or DisplayPort) may be unsupported in some use cases.  In the author's
98experience, even DP to VGA adapters can be troublesome while adapters between
99digital formats work well.
100
101
102Options
103=======
104* x-igd-opregion=[on|*off*]
105  Copy host IGD OpRegion and expose it to guest with fw_cfg
106
107* x-igd-lpc=[on|*off*]
108  Creates a dummy LPC bridge at 00:1f:0 with host VID/DID (i440fx only)
109
110* x-igd-legacy-mode=[on|off|*auto*]
111  Enable/Disable legacy mode
112
113* x-igd-gms=[hex, default 0]
114  Overriding DSM region size in GGC register, 0 means uses host value.
115  Use this only when the DSM size cannot be changed through the
116  'DVMT Pre-Allocated' option in host BIOS.
117
118
119Examples
120========
121* Adding IGD with automatically legacy mode support
122  -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0
123
124* Adding IGD with OpRegion and LPC ID hack, but without VGA ranges
125  (For UEFI guests)
126  -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0,x-igd-legacy-mode=off,x-igd-opregion=on,x-igd-lpc=on,romfile=efi_oprom.rom
127
128
129Guest firmware
130==============
131Guest firmware is responsible for setting up OpRegion and Base of Data Stolen
132Memory (BDSM) in guest address space. IGD passthrough support imposes two
133fw_cfg requirements on the VM firmware:
134
1351) "etc/igd-opregion"
136
137   This fw_cfg file exposes the OpRegion for the IGD device.  A reserved
138   region should be created below 4GB (recommended 4KB alignment), sized
139   sufficient for the fw_cfg file size, and the content of this file copied
140   to it.  The dword based address of this reserved memory region must also
141   be written to the ASLS register at offset 0xFC on the IGD device.  It is
142   recommended that firmware should make use of this fw_cfg entry for any
143   PCI class VGA device with Intel vendor ID.  Multiple of such devices
144   within a VM is undefined.
145
1462) "etc/igd-bdsm-size"
147
148   This fw_cfg file contains an 8-byte, little endian integer indicating
149   the size of the reserved memory region required for IGD stolen memory.
150   Firmware must allocate a reserved memory below 4GB with required 1MB
151   alignment equal to this size.  Additionally the base address of this
152   reserved region must be written to the dword BDSM register in PCI config
153   space of the IGD device at offset 0x5C (or 0xC0 for Gen 11+ devices using
154   64-bit BDSM).  As this support is related to running the IGD ROM, which
155   has other dependencies on the device appearing at guest address 00:02.0,
156   it's expected that this fw_cfg file is only relevant to a single PCI
157   class VGA device with Intel vendor ID, appearing at PCI bus address 00:02.0.
158
159Upstream Seabios has OpRegion and BDSM (pre-Gen11 device only) support.
160However, the support is not accepted by upstream EDK2/OVMF. A recommended
161solution is to create a virtual OpRom with following DXE drivers:
162
163* IgdAssignmentDxe: Set up OpRegion and BDSM according to fw_cfg (must)
164* IntelGopDriver: Closed-source Intel GOP driver
165* PlatformGopPolicy: Protocol required by IntelGopDriver
166
167IntelGopDriver and PlatformGopPolicy is only required when enabling GOP on IGD.
168
169The original IgdAssignmentDxe can be found at [3]. A Intel maintained version
170with PlatformGopPolicy for industrial computing is at [4]. There is also an
171unofficially maintained version with newer Gen11+ device support at [5].
172You need to build them with EDK2.
173
174For the IntelGopDriver, Intel never released it to public. You may contact
175Intel support to get one as [4] said, if you are an Intel Premier Support
176customer, or you can try extracting it from your host firmware using
177"UEFI BIOS Updater"[6].
178
179Once you got all the required DXE drivers, a Option ROM can be generated with
180EfiRom utility in EDK2, using
181  EfiRom -f 0x8086 -i <Device ID of your IGD> -o output.rom \
182  -e IgdAssignmentDxe.efi PlatformGOPPolicy.efi IntelGopDriver.efi
183
184
185Known issues
186============
187When using OVMF as guest firmware, you may encounter the following warning:
188warning: vfio_container_dma_map(0x55fab36ce610, 0x380010000000, 0x108000, 0x7fd336000000) = -22 (Invalid argument)
189
190Solution:
191Set the host physical address bits to IOMMU address width using
192  -cpu host,host-phys-bits-limit=<IOMMU address width>
193Or in libvirt XML with
194  <cpu>
195    <maxphysaddr mode='passthrough' limit='<IOMMU address width>'/>
196  </cpu>
197The IOMMU address width can be determined with
198  echo $(( ((0x$(cat /sys/devices/virtual/iommu/dmar0/intel-iommu/cap) & 0x3F0000) >> 16) + 1 ))
199Refer https://edk2.groups.io/g/devel/topic/patch_v1/102359124 for more details
200
201
202Memory View
203===========
204IGD has it own address space. To use system RAM as VRAM, a single-level page
205table named Global Graphics Translation Table (GTT) is used for the address
206translation. Each page table entry points a 4KB page. Illustration below shows
207the translation flow on IGD with 64-bit GTT PTEs.
208
209(PTE_SIZE == 8)                +-------------+---+
210                               |   Address   | V |  V: Valid Bit
211                               +-------------+---+
212                               | ...         |   |
213IGD:0x01ae9010           0xd740| 0x70ffc000  | 1 |  Mem:0x42ba3e010^
214-----------------------> 0xd748| 0x42ba3e000 | 1 +------------------>
215(addr >> 12) * PTE_SIZE  0xd750| 0x42ba3f000 | 1 |
216                               | ...         |   |
217                               +-------------+---+
218^ The address may be remapped by IOMMU
219
220The memory region store GTT is called GTT Stolen Memory (GSM) it is located
221right below the Data Stolen Memory (DSM). Accessing this region directly is
222not allowed, any access will immediately freeze the whole system. The only way
223to access it is through the second half of MMIO BAR0.
224
225The Data Stolen Memory is reserved by firmware, and acts as the VRAM in pre-OS
226environments. In QEMU, guest firmware (Seabios/OVMF) is responsible for
227reserving a continuous region and program its base address to BDSM register,
228then let VBIOS/GOP driver initializing this region. Illustration below shows
229how DSM is mapped.
230
231       IGD Addr Space                 Host Addr Space         Guest Addr Space
232       +-------------+                +-------------+         +-------------+
233       |             |                |             |         |             |
234       |             |                |             |         |             |
235       |             |                +-------------+         +-------------+
236       |             |                | Data Stolen |         | Data Stolen |
237       |             |                |   (Guest)   |         |   (Guest)   |
238       |             |  +------------>+-------------+<------->+-------------+<--Guest BDSM
239       |             |  | Passthrough |             | EPT     |             |   Emulated by QEMU
240DSMSIZE+-------------+  | with IOMMU  |             | Mapping |             |   Programmed by guest FW
241       |             |  |             |             |         |             |
242       |             |  |             |             |         |             |
243      0+-------------+--+             |             |         |             |
244                        |             +-------------+         |             |
245                        |             | Data Stolen |         +-------------+
246                        |             |   (Host)    |
247                        +------------>+-------------+<--Host BDSM
248                          Non-        |             |   "real" one in HW
249                          Passthrough |             |   Programmed by host FW
250                                      +-------------+
251
252Footnotes
253=========
254[1] https://www.intel.com/content/www/us/en/docs/graphics-for-linux/developer-reference/1-0/dump-video-bios.html
255[2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override
256[3] https://web.archive.org/web/20240827012422/https://bugzilla.tianocore.org/show_bug.cgi?id=935
257    Tianocore bugzilla was down since Jan 2025 :(
258[4] https://eci.intel.com/docs/3.3/components/kvm-hypervisor.html, Patch 0001-0004
259[5] https://github.com/tomitamoeko/VfioIgdPkg
260[6] https://winraid.level1techs.com/t/tool-guide-news-uefi-bios-updater-ubu/30357
261