1.. SPDX-License-Identifier: GPL-2.0 2 3============================== 4Falcon (FAst Logic Controller) 5============================== 6The following sections describe the Falcon core and the ucode running on it. 7The descriptions are based on the Ampere GPU or earlier designs; however, they 8should mostly apply to future designs as well, but everything is subject to 9change. The overview provided here is mainly tailored towards understanding the 10interactions of nova-core driver with the Falcon. 11 12NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which 13handle secure firmware tasks, initialization, and power management. Modern 14NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system 15processor) and SEC2 (the security engine)) and also may integrate a RISC-V core. 16This core is capable of running both RISC-V and Falcon code. 17 18The code running on the Falcon cores is also called 'ucode', and will be 19referred to as such in the following sections. 20 21Falcons have separate instruction and data memories (IMEM/DMEM) and provide a 22small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from 23system memory. The nova-core driver must reset and configure the Falcon, load 24its firmware via DMA, and start its CPU. 25 26Falcon security levels 27====================== 28Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS) 29modes. 30 31Heavy Secured (HS) also known as Privilege Level 3 (PL3) 32-------------------------------------------------------- 33HS ucode is the most trusted code and has access to pretty much everything on 34the chip. The HS binary includes a signature in it which is verified at boot. 35This signature verification is done by the hardware itself, thus establishing a 36root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the 37GSP in HS mode. FRTS, which involves setting up and loading content into the WPR 38(Write Protect Region), has to be done by the HS ucode and cannot be done by the 39host CPU or LS ucode. 40 41Light Secured (LS or PL2) and Non Secured (NS or PL0) 42----------------------------------------------------- 43These modes are less secure than HS. Like HS, the LS or NS ucode binary also 44typically includes a signature in it. To load firmware in LS or NS mode onto a 45Falcon, another Falcon needs to be running in HS mode, which also establishes the 46root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter" 47ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the 48run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an 49example, after reset on an Ampere, FWSEC runs on the GSP which then loads the 50devinit engine onto the PMU in LS mode. 51 52Root of trust establishment 53--------------------------- 54To establish a root of trust, the code running on a Falcon must be immutable and 55hardwired into a read-only memory (ROM). This follows industry norms for 56verification of firmware. This code is called the Boot ROM (BROM). The nova-core 57driver on the CPU communicates with Falcon's Boot ROM through various Falcon 58registers prefixed with "BROM" (see regs.rs). 59 60After nova-core driver reads the necessary ucode from VBIOS, it programs the 61BROM and DMA registers to trigger the Falcon to load the HS ucode from the system 62memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified 63by the Falcon's Boot ROM. 64 65Once the verified HS code is running on a Falcon, it can verify and load other 66LS/NS ucode binaries onto other Falcons and start them. The process of signature 67verification is the same as HS; just in this case, the hardware (BROM) doesn't 68compute the signature, but the HS ucode does. 69 70The root of trust is therefore established as follows: 71 Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode. 72 73On an Ampere GPU, for example, the boot verification flow is: 74 Hardware (Boot ROM running on the SEC2) -> 75 HS ucode (Booter running on the SEC2) -> 76 LS ucode (GSP-RM running on the GSP) 77 78.. note:: 79 While the CPU can load HS ucode onto a Falcon microcontroller and have it 80 verified by the hardware and run, the CPU itself typically does not load 81 LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the 82 HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the 83 SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run 84 the "SEC2-RTOS" ucode at runtime. This presents a problem: there is no 85 component to load the SEC2-RTOS ucode onto the SEC2. The CPU cannot load 86 LS code, and GSP-RM must run in LS mode. To overcome this, the GSP is 87 temporarily made to run HS ucode (which is itself loaded by the CPU via 88 the nova-core driver using a "GSP-provided sequencer") which then loads 89 the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP then resumes 90 running its own GSP-RM LS ucode. 91 92Falcon memory subsystem and DMA engine 93====================================== 94Falcons have separate instruction and data memories (IMEM/DMEM) 95and contains a small DMA engine called FBDMA (Framebuffer DMA) which does 96DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF 97(Framebuffer Interface), to external memory. 98 99DMA transfers are possible from the Falcon's memory to both the system memory 100and the framebuffer memory (VRAM). 101 102To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory 103is accessed (also known as aperture type). In the nova-core driver, this is 104determined by the `FalconFbifTarget` enum. 105 106The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon 107controls access by the FBDMA to the external memory. 108 109Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows:: 110 111 External Memory (Framebuffer / System DRAM) 112 ^ | 113 | | 114 | v 115 +-----------------------------------------------------+ 116 | | | 117 | +---------------+ | | 118 | | FBIF |-------+ | FALCON 119 | | (FrameBuffer | Memory Interface | PROCESSOR 120 | | InterFace) | | 121 | | Apertures | | 122 | | Configures | | 123 | | mem access | | 124 | +-------^-------+ | 125 | | | 126 | | FBDMA uses configured FBIF apertures | 127 | | to access External Memory 128 | | 129 | +-------v--------+ +---------------+ 130 | | FBDMA | cfg | RISC | 131 | | (FrameBuffer |<---->| CORE |----->. Direct Core Access 132 | | DMA Engine) | | | | 133 | | - Master dev. | | (can run both | | 134 | +-------^--------+ | Falcon and | | 135 | | cfg--->| RISC-V code) | | 136 | | / | | | 137 | | | +---------------+ | +------------+ 138 | | | | | BROM | 139 | | | <--->| (Boot ROM) | 140 | | / | +------------+ 141 | | v | 142 | +---------------+ | 143 | | IO-PMP | Controls access by FBDMA | 144 | | (IO Physical | and other IO Masters | 145 | | Memory Protect) | 146 | +-------^-------+ | 147 | | | 148 | | Protected Access Path for FBDMA | 149 | v | 150 | +---------------------------------------+ | 151 | | Memory | | 152 | | +---------------+ +------------+ | | 153 | | | IMEM | | DMEM | |<-----+ 154 | | | (Instruction | | (Data | | 155 | | | Memory) | | Memory) | | 156 | | +---------------+ +------------+ | 157 | +---------------------------------------+ 158 +-----------------------------------------------------+ 159