lastdragon/linux-kernel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resguardo inicial

Browse Source
This commit is contained in:
lastdragon
2026-09-06 12:53:21 -06:00
parent 662efe3c2a
commit ad2088d178
94853 changed files with 43200404 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
linux-7.2.3/Documentation/gpu/nova/core/devinit.rst
+61
View File
@@ -0,0 +1,61 @@
.. SPDX-License-Identifier: GPL-2.0
==================================
Device Initialization (devinit)
==================================
The devinit process is complex and subject to change. This document provides a high-level
overview using the Ampere GPU family as an example. The goal is to provide a conceptual
overview of the process to aid in understanding the corresponding kernel code.
Device initialization (devinit) is a crucial sequence of register read/write operations
that occur after a GPU reset. The devinit sequence is essential for properly configuring
the GPU hardware before it can be used.
The devinit engine is an interpreter program that typically runs on the PMU (Power Management
Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization
commands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as the
FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs before the
nova-core driver is even loaded. On an Ampere GPU, the devinit ucode is separate from the
FWSEC ucode. It is launched by FWSEC, which runs on the GSP in 'heavy-secure' mode, while
devinit runs on the PMU in 'light-secure' mode.
Key Functions of devinit
------------------------
devinit performs several critical tasks:
1. Programming VRAM memory controller timings
2. Power sequencing
3. Clock and PLL (Phase-Locked Loop) configuration
4. Thermal management
Low-level Firmware Initialization Flow
--------------------------------------
Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU
firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU is
considered unusable until this initialization process completes.
These low-level GPU firmware components are typically:
1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst).
2. Executed in sequence on different microcontrollers:
- The devinit engine typically but not necessarily runs on the PMU.
- On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in
heavy-secure mode.
Before the driver can proceed with further initialization, it must wait for a signal
indicating that core initialization is complete (known as GFW_BOOT). This signal is
asserted by the FWSEC running on the GSP in heavy-secure mode.
Runtime Considerations
----------------------
It's important to note that the devinit sequence also needs to run during suspend/resume
operations at runtime, not just during initial boot, as it is critical to power management.
Security and Access Control
---------------------------
The initialization process involves careful privilege management. For example, before
accessing certain completion status registers, the driver must check privilege level
masks. Some registers are only accessible after secure firmware (FWSEC) lowers the
privilege level to allow CPU (LS/low-secure) access. This is the case, for example,
when receiving the GFW_BOOT signal.
linux-7.2.3/Documentation/gpu/nova/core/falcon.rst
+158
View File
@@ -0,0 +1,158 @@
.. SPDX-License-Identifier: GPL-2.0
==============================
Falcon (FAst Logic Controller)
==============================
The following sections describe the Falcon core and the ucode running on it.
The descriptions are based on the Ampere GPU or earlier designs; however, they
should mostly apply to future designs as well, but everything is subject to
change. The overview provided here is mainly tailored towards understanding the
interactions of nova-core driver with the Falcon.
NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which
handle secure firmware tasks, initialization, and power management. Modern
NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system
processor) and SEC2 (the security engine)) and also may integrate a RISC-V core.
This core is capable of running both RISC-V and Falcon code.
The code running on the Falcon cores is also called 'ucode', and will be
referred to as such in the following sections.
Falcons have separate instruction and data memories (IMEM/DMEM) and provide a
small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from
system memory. The nova-core driver must reset and configure the Falcon, load
its firmware via DMA, and start its CPU.
Falcon security levels
======================
Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS)
modes.
Heavy Secured (HS) also known as Privilege Level 3 (PL3)
--------------------------------------------------------
HS ucode is the most trusted code and has access to pretty much everything on
the chip. The HS binary includes a signature in it which is verified at boot.
This signature verification is done by the hardware itself, thus establishing a
root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the
GSP in HS mode. FRTS, which involves setting up and loading content into the WPR
(Write Protect Region), has to be done by the HS ucode and cannot be done by the
host CPU or LS ucode.
Light Secured (LS or PL2) and Non Secured (NS or PL0)
-----------------------------------------------------
These modes are less secure than HS. Like HS, the LS or NS ucode binary also
typically includes a signature in it. To load firmware in LS or NS mode onto a
Falcon, another Falcon needs to be running in HS mode, which also establishes the
root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter"
ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the
run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an
example, after reset on an Ampere, FWSEC runs on the GSP which then loads the
devinit engine onto the PMU in LS mode.
Root of trust establishment
---------------------------
To establish a root of trust, the code running on a Falcon must be immutable and
hardwired into a read-only memory (ROM). This follows industry norms for
verification of firmware. This code is called the Boot ROM (BROM). The nova-core
driver on the CPU communicates with Falcon's Boot ROM through various Falcon
registers prefixed with "BROM" (see regs.rs).
After nova-core driver reads the necessary ucode from VBIOS, it programs the
BROM and DMA registers to trigger the Falcon to load the HS ucode from the system
memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified
by the Falcon's Boot ROM.
Once the verified HS code is running on a Falcon, it can verify and load other
LS/NS ucode binaries onto other Falcons and start them. The process of signature
verification is the same as HS; just in this case, the hardware (BROM) doesn't
compute the signature, but the HS ucode does.
The root of trust is therefore established as follows:
Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode.
On an Ampere GPU, for example, the boot verification flow is:
Hardware (Boot ROM running on the SEC2) ->
HS ucode (Booter running on the SEC2) ->
LS ucode (GSP-RM running on the GSP)
.. note::
While the CPU can load HS ucode onto a Falcon microcontroller and have it
verified by the hardware and run, the CPU itself typically does not load
LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the
HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the
SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run
the "SEC2-RTOS" ucode at runtime. This presents a problem: there is no
component to load the SEC2-RTOS ucode onto the SEC2. The CPU cannot load
LS code, and GSP-RM must run in LS mode. To overcome this, the GSP is
temporarily made to run HS ucode (which is itself loaded by the CPU via
the nova-core driver using a "GSP-provided sequencer") which then loads
the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP then resumes
running its own GSP-RM LS ucode.
Falcon memory subsystem and DMA engine
======================================
Falcons have separate instruction and data memories (IMEM/DMEM)
and contains a small DMA engine called FBDMA (Framebuffer DMA) which does
DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF
(Framebuffer Interface), to external memory.
DMA transfers are possible from the Falcon's memory to both the system memory
and the framebuffer memory (VRAM).
To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory
is accessed (also known as aperture type). In the nova-core driver, this is
determined by the `FalconFbifTarget` enum.
The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon
controls access by the FBDMA to the external memory.
Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows::
External Memory (Framebuffer / System DRAM)
^ |
| |
| v
+-----------------------------------------------------+
| | |
| +---------------+ | |
| | FBIF |-------+ | FALCON
| | (FrameBuffer | Memory Interface | PROCESSOR
| | InterFace) | |
| | Apertures | |
| | Configures | |
| | mem access | |
| +-------^-------+ |
| | |
| | FBDMA uses configured FBIF apertures |
| | to access External Memory
| |
| +-------v--------+ +---------------+
| | FBDMA | cfg | RISC |
| | (FrameBuffer |<---->| CORE |----->. Direct Core Access
| | DMA Engine) | | | |
| | - Master dev. | | (can run both | |
| +-------^--------+ | Falcon and | |
| | cfg--->| RISC-V code) | |
| | / | | |
| | | +---------------+ | +------------+
| | | | | BROM |
| | | <--->| (Boot ROM) |
| | / | +------------+
| | v |
| +---------------+ |
| | IO-PMP | Controls access by FBDMA |
| | (IO Physical | and other IO Masters |
| | Memory Protect) |
| +-------^-------+ |
| | |
| | Protected Access Path for FBDMA |
| v |
| +---------------------------------------+ |
| | Memory | |
| | +---------------+ +------------+ | |
| | | IMEM | | DMEM | |<-----+
| | | (Instruction | | (Data | |
| | | Memory) | | Memory) | |
| | +---------------+ +------------+ |
| +---------------------------------------+
+-----------------------------------------------------+
linux-7.2.3/Documentation/gpu/nova/core/fwsec.rst
+181
View File
@@ -0,0 +1,181 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
=========================
FWSEC (Firmware Security)
=========================
This document briefly/conceptually describes the FWSEC (Firmware Security) image
and its role in the GPU boot sequence. As such, this information is subject to
change in the future and is only current as of the Ampere GPU family. However,
hopefully the concepts described will be useful for understanding the kernel code
that deals with it. All the information is derived from publicly available
sources such as public drivers and documentation.
The role of FWSEC is to provide a secure boot process. It runs in
'Heavy-secure' mode, and performs firmware verification after a GPU reset
before loading various ucode images onto other microcontrollers on the GPU,
such as the PMU and GSP.
FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of
ROM (see vbios.rst for more details). It contains different commands like FRTS
(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
reset and loading them with other non-FWSEC ucode). The kernel driver only needs
to perform FRTS, since Secure Boot (SB) has already completed by the time the driver
is loaded.
The FRTS command carves out the WPR2 region (Write protected region) which contains
data required for power management. Once setup, only HS mode ucode can access it
(see falcon.rst for privilege levels).
The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains
various ucode images (also known as applications) -- one of them being FWSEC. For how
it is extracted, see vbios.rst and the vbios.rs source code.
The Falcon data for each ucode images (including the FWSEC image) is a combination
of headers, data sections (DMEM) and instruction code sections (IMEM). All these
ucode images are stored in the same ROM partition and the PMU table is used to look
up the application to load it based on its application ID (see vbios.rs).
For the nova-core driver, the FWSEC contains an 'application interface' called
DMEMMAPPER. This interface is used to execute the 'FWSEC-FRTS' command, among others.
For Ampere, FWSEC is running on the GSP in Heavy-secure mode and runs FRTS.
FWSEC Memory Layout
-------------------
The memory layout of the FWSEC image is as follows::
+---------------------------------------------------------------+
| FWSEC ROM image (type 0xE0) |
| |
| +---------------------------------+ |
| | PMU Falcon Ucode Table | |
| | (PmuLookupTable) | |
| | +-------------------------+ | |
| | | Table Header | | |
| | | - version: 0x01 | | |
| | | - header_size: 6 | | |
| | | - entry_size: 6 | | |
| | | - entry_count: N | | |
| | | - desc_version:3(unused)| | |
| | +-------------------------+ | |
| | ... | |
| | +-------------------------+ | |
| | | Entry for FWSEC (0x85) | | |
| | | (PmuLookupTableEntry) | | |
| | | - app_id: 0x85 (FWSEC) |----|----+ |
| | | - target_id: 0x01 (PMU) | | | |
| | | - data: offset ---------|----|----|---+ look up FWSEC |
| | +-------------------------+ | | | |
| +---------------------------------+ | | |
| | | |
| | | |
| +---------------------------------+ | | |
| | FWSEC Ucode Component |<---+ | |
| | (aka Falcon data) | | |
| | +-------------------------+ | | |
| | | FalconUCodeDescV3 |<---|--------+ |
| | | - hdr | | |
| | | - stored_size | | |
| | | - pkc_data_offset | | |
| | | - interface_offset -----|----|----------------+ |
| | | - imem_phys_base | | | |
| | | - imem_load_size | | | |
| | | - imem_virt_base | | | |
| | | - dmem_phys_base | | | |
| | | - dmem_load_size | | | |
| | | - engine_id_mask | | | |
| | | - ucode_id | | | |
| | | - signature_count | | look up sig | |
| | | - signature_versions --------------+ | |
| | +-------------------------+ | | | |
| | (no gap) | | | |
| | +-------------------------+ | | | |
| | | Signatures Section |<---|-----+ | |
| | | (384 bytes per sig) | | | |
| | | - RSA-3K Signature 1 | | | |
| | | - RSA-3K Signature 2 | | | |
| | | ... | | | |
| | +-------------------------+ | | |
| | | | |
| | +-------------------------+ | | |
| | | IMEM Section (Code) | | | |
| | | | | | |
| | | Contains instruction | | | |
| | | code etc. | | | |
| | +-------------------------+ | | |
| | | | |
| | +-------------------------+ | | |
| | | DMEM Section (Data) | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | Application | |<---|----------------+ |
| | | | Interface Table | | | |
| | | | (FalconAppifHdrV1) | | | |
| | | | Header: | | | |
| | | | - version: 0x01 | | | |
| | | | - header_size: 4 | | | |
| | | | - entry_size: 8 | | | |
| | | | - entry_count: N | | | |
| | | | | | | |
| | | | Entries: | | | |
| | | | +-----------------+ | | | |
| | | | | DEVINIT (ID 1) | | | | |
| | | | | - id: 0x01 | | | | |
| | | | | - dmemOffset X -|-|-|----+ |
| | | | +-----------------+ | | | |
| | | | +-----------------+ | | | |
| | | | | DMEMMAPPER(ID 4)| | | | |
| | | | | - id: 0x04 | | | | Used only for DevInit |
| | | | | (NVFW_FALCON_ | | | | application (not FWSEC) |
| | | | | APPIF_ID_DMEMMAPPER) | |
| | | | | - dmemOffset Y -|-|-|----|-----+ |
| | | | +-----------------+ | | | | |
| | | +---------------------+ | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | DEVINIT Engine |<|----+ | Used by FWSEC |
| | | | Interface | | | | app. |
| | | +---------------------+ | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | DMEM Mapper (ID 4) |<|----+-----+ |
| | | | (FalconAppifDmemmapperV3) | |
| | | | - signature: "DMAP" | | | |
| | | | - version: 0x0003 | | | |
| | | | - Size: 64 bytes | | | |
| | | | - cmd_in_buffer_off | |----|------------+ |
| | | | - cmd_in_buffer_size| | | | |
| | | | - cmd_out_buffer_off| |----|------------|-----+ |
| | | | - cmd_out_buffer_sz | | | | | |
| | | | - init_cmd | | | | | |
| | | | - features | | | | | |
| | | | - cmd_mask0/1 | | | | | |
| | | +---------------------+ | | | | |
| | | | | | | |
| | | +---------------------+ | | | | |
| | | | Command Input Buffer|<|----|------------+ | |
| | | | - Command data | | | | |
| | | | - Arguments | | | | |
| | | +---------------------+ | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | Command Output |<|----|------------------+ |
| | | | Buffer | | | |
| | | | - Results | | | |
| | | | - Status | | | |
| | | +---------------------+ | | |
| | +-------------------------+ | |
| +---------------------------------+ |
| |
+---------------------------------------------------------------+
.. note::
This is using an GA-102 Ampere GPU as an example and could vary for future GPUs.
.. note::
The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR
(Video Protected Region) initialization as well. Before the nova-core driver is even
loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit
sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer
GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this
async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM
memory allocators need to wait for this scrubbing to complete).
linux-7.2.3/Documentation/gpu/nova/core/guidelines.rst
+24
View File
@@ -0,0 +1,24 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
==========
Guidelines
==========
This documents contains the guidelines for nova-core. Additionally, all common
guidelines of the Nova project do apply.
Driver API
==========
One main purpose of nova-core is to implement the abstraction around the
firmware interface of GSP and provide a firmware (version) independent API for
2nd level drivers, such as nova-drm or the vGPU manager VFIO driver.
Therefore, it is not permitted to leak firmware (version) specifics, through the
driver API, to 2nd level drivers.
Acceptance Criteria
===================
- To the extend possible, patches submitted to nova-core must be tested for
regressions with all 2nd level drivers.
linux-7.2.3/Documentation/gpu/nova/core/todo.rst
+294
View File
@@ -0,0 +1,294 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
=========
Task List
=========
Tasks may have the following fields:
- ``Complexity``: Describes the required familiarity with Rust and / or the
corresponding kernel APIs or subsystems. There are four different complexities,
``Beginner``, ``Intermediate``, ``Advanced`` and ``Expert``.
- ``Reference``: References to other tasks.
- ``Link``: Links to external resources.
- ``Contact``: The person that can be contacted for further information about
the task.
A task might have `[ABCD]` code after its name. This code can be used to grep
into the code for `TODO` entries related to it.
Enablement (Rust)
=================
Tasks that are not directly related to nova-core, but are preconditions in terms
of required APIs.
FromPrimitive API [FPRI]
------------------------
Sometimes the need arises to convert a number to a value of an enum or a
structure.
A good example from nova-core would be the ``Chipset`` enum type, which defines
the value ``AD102``. When probing the GPU the value ``0x192`` can be read from a
certain register indication the chipset AD102. Hence, the enum value ``AD102``
should be derived from the number ``0x192``. Currently, nova-core uses a custom
implementation (``Chipset::from_u32`` for this.
Instead, it would be desirable to have something like the ``FromPrimitive``
trait [1] from the num crate.
Having this generalization also helps with implementing a generic macro that
automatically generates the corresponding mappings between a value and a number.
FromPrimitive support has been worked on in the past, but hasn't been followed
since then [1].
There also have been considerations of ToPrimitive [2].
| Complexity: Beginner
| Link: https://docs.rs/num/latest/num/trait.FromPrimitive.html
| Link: https://lore.kernel.org/all/cover.1750689857.git.y.j3ms.n@gmail.com/ [1]
| Link: https://rust-for-linux.zulipchat.com/#narrow/channel/288089-General/topic/Implement.20.60FromPrimitive.60.20trait.20.2B.20derive.20macro.20for.20nova-core/with/541971854 [2]
Numerical operations [NUMM]
---------------------------
Nova uses integer operations that are not part of the standard library (or not
implemented in an optimized way for the kernel). These include:
- The "Find Last Set Bit" (`fls` function of the C part of the kernel)
operation.
A `num` core kernel module is being designed to provide these operations.
| Complexity: Intermediate
| Contact: Alexandre Courbot
Page abstraction for foreign pages
----------------------------------
Rust abstractions for pages not created by the Rust page abstraction without
direct ownership.
There is active onging work from Abdiel Janulgue [1] and Lina [2].
| Complexity: Advanced
| Link: https://lore.kernel.org/linux-mm/20241119112408.779243-1-abdiel.janulgue@gmail.com/ [1]
| Link: https://lore.kernel.org/rust-for-linux/20250202-rust-page-v1-0-e3170d7fe55e@asahilina.net/ [2]
PCI MISC APIs
-------------
Extend the existing PCI device / driver abstractions by SR-IOV, capability, MSI
API abstractions.
SR-IOV [1] is work in progress.
| Complexity: Beginner
| Link: https://lore.kernel.org/all/20251119-rust-pci-sriov-v1-0-883a94599a97@redhat.com/ [1]
GPU (general)
=============
Initial Devinit support
-----------------------
Implement BIOS Device Initialization, i.e. memory sizing, waiting, PLL
configuration.
| Contact: Dave Airlie
| Complexity: Beginner
MMU / PT management
-------------------
Work out the architecture for MMU / page table management.
We need to consider that nova-drm will need rather fine-grained control,
especially in terms of locking, in order to be able to implement asynchronous
Vulkan queues.
While generally sharing the corresponding code is desirable, it needs to be
evaluated how (and if at all) sharing the corresponding code is expedient.
| Complexity: Expert
VRAM memory allocator
---------------------
Investigate options for a VRAM memory allocator.
Some possible options:
- Rust abstractions for
- RB tree (interval tree) / drm_mm
- maple_tree
- native Rust collections
There is work in progress for using drm_buddy [1].
| Complexity: Advanced
| Link: https://lore.kernel.org/all/20251219203805.1246586-4-joelagnelf@nvidia.com/ [1]
Instance Memory
---------------
Implement support for instmem (bar2) used to store page tables.
| Complexity: Intermediate
| Contact: Dave Airlie
GPU System Processor (GSP)
==========================
Export GSP log buffers
----------------------
Recent patches from Timur Tabi [1] added support to expose GSP-RM log buffers
(even after failure to probe the driver) through debugfs.
This is also an interesting feature for nova-core, especially in the early days.
| Link: https://lore.kernel.org/nouveau/20241030202952.694055-2-ttabi@nvidia.com/ [1]
| Reference: Debugfs abstractions
| Complexity: Intermediate
GSP firmware abstraction
------------------------
The GSP-RM firmware API is unstable and may incompatibly change from version to
version, in terms of data structures and semantics.
This problem is one of the big motivations for using Rust for nova-core, since
it turns out that Rust's procedural macro feature provides a rather elegant way
to address this issue:
1. generate Rust structures from the C headers in a separate namespace per version
2. build abstraction structures (within a generic namespace) that implement the
firmware interfaces; annotate the differences in implementation with version
identifiers
3. use a procedural macro to generate the actual per version implementation out
of this abstraction
4. instantiate the correct version type one on runtime (can be sure that all
have the same interface because it's defined by a common trait)
There is a PoC implementation of this pattern, in the context of the nova-core
PoC driver.
This task aims at refining the feature and ideally generalize it, to be usable
by other drivers as well.
| Complexity: Expert
GSP message queue
-----------------
Implement low level GSP message queue (command, status) for communication
between the kernel driver and GSP.
| Complexity: Advanced
| Contact: Dave Airlie
Bootstrap GSP
-------------
Call the boot firmware to boot the GSP processor; execute initial control
messages.
| Complexity: Intermediate
| Contact: Dave Airlie
Client / Device APIs
--------------------
Implement the GSP message interface for client / device allocation and the
corresponding client and device allocation APIs.
| Complexity: Intermediate
| Contact: Dave Airlie
Bar PDE handling
----------------
Synchronize page table handling for BARs between the kernel driver and GSP.
| Complexity: Beginner
| Contact: Dave Airlie
FIFO engine
-----------
Implement support for the FIFO engine, i.e. the corresponding GSP message
interface and provide an API for chid allocation and channel handling.
| Complexity: Advanced
| Contact: Dave Airlie
GR engine
---------
Implement support for the graphics engine, i.e. the corresponding GSP message
interface and provide an API for (golden) context creation and promotion.
| Complexity: Advanced
| Contact: Dave Airlie
CE engine
---------
Implement support for the copy engine, i.e. the corresponding GSP message
interface.
| Complexity: Intermediate
| Contact: Dave Airlie
VFN IRQ controller
------------------
Support for the VFN interrupt controller.
| Complexity: Intermediate
| Contact: Dave Airlie
External APIs
=============
nova-core base API
------------------
Work out the common pieces of the API to connect 2nd level drivers, i.e. vGPU
manager and nova-drm.
| Complexity: Advanced
vGPU manager API
----------------
Work out the API parts required by the vGPU manager, which are not covered by
the base API.
| Complexity: Advanced
nova-core C API
---------------
Implement a C wrapper for the APIs required by the vGPU manager driver.
| Complexity: Intermediate
Testing
=======
CI pipeline
-----------
Investigate option for continuous integration testing.
This can go from as simple as running KUnit tests over running (graphics) CTS to
booting up (multiple) guest VMs to test VFIO use-cases.
It might also be worth to consider the introduction of a new test suite directly
sitting on top of the uAPI for more targeted testing and debugging. There may be
options for collaboration / shared code with the Mesa project.
| Complexity: Advanced
linux-7.2.3/Documentation/gpu/nova/core/vbios.rst
+240
View File
@@ -0,0 +1,240 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
==========
VBIOS
==========
This document describes the layout of the VBIOS image which is a series of concatenated
images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
bootstrap various microcontrollers (PMU, SEC, GSP) with critical initialization before
the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
The format of the images in the ROM follow the "BIOS Specification" part of the
PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec
are the ones that contain Falcon ucode and what we are mainly looking for.
As an example, the following are the different image types that can be found in the
VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name
likely comes from the "IBM PC/AT" architecture.
- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
driver that is used to display UEFI graphics output.
- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
contains various microcodes (also known as an applications) that do a range
of different functions. The FWSEC ucode is run in heavy-secure mode and
typically runs directly on the GSP (it could be running on a different
designated processor in future generations but as of Ampere, it is the GSP).
This firmware then loads other firmware ucodes onto the PMU and SEC2
microcontrollers for gfw initialization after GPU reset and before the driver
loads (see devinit.rst). The DEVINIT ucode is itself another ucode that is
stored in this ROM partition.
Once located, the Falcon ucodes have "Application Interfaces" in their data
memory (DMEM). For FWSEC, the application interface we use for FWSEC is the
"DMEM mapper" interface which is configured to run the "FRTS" command. This
command carves out the WPR2 (Write-Protected Region) in VRAM. It then places
important power-management data, called 'FRTS', into this region. The WPR2
region is only accessible to heavy-secure ucode.
.. note::
It is not clear why FwSec has 2 different partitions in the ROM, but they both
are of type 0xE0 and can be identified as such. This could be subject to change
in future generations.
IFR Header
----------
On Kepler and later GPUs, the ROM begins with an Init-from-ROM (IFR) header
rather than a standard PCI ROM signature (0xAA55). The driver must parse the
IFR header to find where the PCI ROM images actually start.
Init-from-ROM (IFR) is a special GPU feature used for power management
on some Nvidia GPUs. It references data in the VBIOS for its operation,
but for drivers the important piece is a header that precedes the
VBIOS PCI Expansion ROM.
Most such GPUs do not need to parse the IFR header in order to find the
VBIOS, but the Nvidia GA100 is the exception. GA100 lacks a display engine,
so the PRAMIN method (which reads the VBIOS from VRAM via display hardware)
is unavailable, forcing the driver to read the ROM directly via PROM.
On other similar GPUs, either PRAMIN succeeds before PROM is tried, or the
IFR hardware has already applied the ROM offset so that PROM reads
transparently skip the IFR header.
The driver should first check for the standard 0xAA55 signature at offset 0.
If found, there is no IFR header and the PCI ROM images start at
offset 0. If not found, check for the IFR signature and parse the header to
determine the PCI ROM image offset.
Fixed Header Format
~~~~~~~~~~~~~~~~~~~
The IFR header begins with four 32-bit words at fixed offsets::
Offset Name Fields
------ ------- ------
0x00 FIXED0 bits 31:0 - Signature (must be 0x4947564E, ASCII "NVGI")
0x04 FIXED1 bit 31 - Reserved
bits 30:16 - FIXED_DATA_SIZE Fixed data size (offset to extended section)
bits 15:8 - VERSIONSW Software version
bits 7:0 - Reserved
0x08 FIXED2 bit 31 - Reserved
bits 30:20 - Reserved (zero)
bits 19:0 - TOTAL_DATA_SIZE Total data size
Finding the PCI ROM Image Offset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The method to find this offset depends on `VERSIONSW`.
- **Version 1 and 2**: Read `FIXED_DATA_SIZE` from `FIXED1` to get the extended
section offset. The PCI ROM image is the 32-bit value at `FIXED_DATA_SIZE + 4`.
- **Version 3**: Read `TOTAL_DATA_SIZE` from `FIXED2`. The 32-bit value at that
offset is a flash status offset. Add 4096 to get the ROM directory offset,
`ROM_DIRECTORY_OFFSET`. The ROM directory must have signature 0x44524652
(ASCII "RFRD"). The PCI ROM image offset is the 32-bit value at
`ROM_DIRECTORY_OFFSET + 8`.
The PCI ROM image offset must be 4-byte aligned. All offsets are relative to the
start of ROM (BAR0 + 0x300000).
VBIOS ROM Layout
----------------
The VBIOS (PCI Expansion ROM) is a series of concatenated images laid out as
follows. On GPUs with an IFR header, this layout begins at the image offset
determined by parsing the IFR header. On older GPUs, it begins at offset 0::
+----------------------------------------------------------------------------+
| VBIOS (Starting at ROM_OFFSET: 0x300000 + IFR image offset) |
+----------------------------------------------------------------------------+
| +-----------------------------------------------+ |
| | PciAt Image (Type 0x00) | |
| +-----------------------------------------------+ |
| | +-------------------+ | |
| | | ROM Header | | |
| | | (Signature 0xAA55)| | |
| | +-------------------+ | |
| | | rom header's pci_data_struct_offset | |
| | | points to the PCIR structure | |
| | V | |
| | +-------------------+ | |
| | | PCIR Structure | | |
| | | (Signature "PCIR")| | |
| | | last_image: 0x80 | | |
| | | image_len: size | | |
| | | in 512-byte units | | |
| | +-------------------+ | |
| | | | |
| | | NPDE immediately follows PCIR | |
| | V | |
| | +-------------------+ | |
| | | NPDE Structure | | |
| | | (Signature "NPDE")| | |
| | | last_image: 0x00 | | |
| | +-------------------+ | |
| | | |
| | +-------------------+ | |
| | | BIT Header | (Signature scanning | |
| | | (Signature "BIT") | provides the location | |
| | +-------------------+ of the BIT table) | |
| | | header is | |
| | | followed by a table of tokens | |
| | V one of which is for falcon data. | |
| | +-------------------+ | |
| | | BIT Tokens | | |
| | | ______________ | | |
| | | | Falcon Data | | | |
| | | | Token (0x70)|---+------------>------------+--+ |
| | | +-------------+ | falcon_data_ptr() | | |
| | +-------------------+ | V |
| +-----------------------------------------------+ | |
| (no gap between images) | |
| +-----------------------------------------------+ | |
| | EFI Image (Type 0x03) | | |
| +-----------------------------------------------+ | |
| | Contains the UEFI GOP driver (Graphics Output)| | |
| | +-------------------+ | | |
| | | ROM Header | | | |
| | +-------------------+ | | |
| | | PCIR Structure | | | |
| | +-------------------+ | | |
| | | NPDE Structure | | | |
| | +-------------------+ | | |
| | | Image data | | | |
| | +-------------------+ | | |
| +-----------------------------------------------+ | |
| (no gap between images) | |
| +-----------------------------------------------+ | |
| | First FwSec Image (Type 0xE0) | | |
| +-----------------------------------------------+ | |
| | +-------------------+ | | |
| | | ROM Header | | | |
| | +-------------------+ | | |
| | | PCIR Structure | | | |
| | +-------------------+ | | |
| | | NPDE Structure | | | |
| | +-------------------+ | | |
| | | Image data | | | |
| | +-------------------+ | | |
| +-----------------------------------------------+ | |
| (no gap between images) | |
| +-----------------------------------------------+ | |
| | Second FwSec Image (Type 0xE0) | | |
| +-----------------------------------------------+ | |
| | +-------------------+ | | |
| | | ROM Header | | | |
| | +-------------------+ | | |
| | | PCIR Structure | | | |
| | +-------------------+ | | |
| | | NPDE Structure | | | |
| | +-------------------+ | | |
| | | | |
| | +-------------------+ | | |
| | | PMU Lookup Table | <- falcon_data_offset <----+ |
| | | +-------------+ | pmu_lookup_table | |
| | | | Entry 0x85 | | | |
| | | | FWSEC_PROD | | | |
| | | +-------------+ | | |
| | +-------------------+ | |
| | | | |
| | | points to | |
| | V | |
| | +-------------------+ | |
| | | FalconUCodeDescV3 | <- falcon_ucode_offset | |
| | | (FWSEC Firmware) | fwsec_header() | |
| | +-------------------+ | |
| | | immediately followed by... | |
| | V | |
| | +----------------------------+ | |
| | | Signatures + FWSEC Ucode | | |
| | | fwsec_sigs(), fwsec_ucode()| | |
| | +----------------------------+ | |
| +-----------------------------------------------+ |
| |
+----------------------------------------------------------------------------+
.. note::
This diagram is created based on an GA-102 Ampere GPU as an example and could
vary for future or other GPUs.
.. note::
For more explanations of acronyms, see the detailed descriptions in `vbios.rs`.
Falcon data Lookup
------------------
A key part of the VBIOS extraction code (vbios.rs) is to find the location of the
Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is
used to find the required Falcon ucode based on an application ID.
The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_)
tokens for a token with the Falcon data token id (0x70) which indicates the
offset of the same from the start of the VBIOS image. Unfortunately, the offset
does not account for the EFI image located between the PciAt and FwSec images.
The `vbios.rs` code compensates for this with appropriate arithmetic.
.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
linux-7.2.3/Documentation/gpu/nova/guidelines.rst
+69
View File
@@ -0,0 +1,69 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
==========
Guidelines
==========
This document describes the general project guidelines that apply to nova-core
and nova-drm.
Language
========
The Nova project uses the Rust programming language. In this context, all rules
of the Rust for Linux project as documented in
:doc:`../../rust/general-information` apply. Additionally, the following rules
apply.
- Unless technically necessary otherwise (e.g. uAPI), any driver code is written
in Rust.
- Unless technically necessary, unsafe Rust code must be avoided. In case of
technical necessity, unsafe code should be isolated in a separate component
providing a safe API for other driver code to use.
Style
-----
All rules of the Rust for Linux project as documented in
:doc:`../../rust/coding-guidelines` apply.
For a submit checklist, please also see the `Rust for Linux Submit checklist
addendum <https://rust-for-linux.com/contributing#submit-checklist-addendum>`_.
Documentation
=============
The availability of proper documentation is essential in terms of scalability,
accessibility for new contributors and maintainability of a project in general,
but especially for a driver running as complex hardware as Nova is targeting.
Hence, adding documentation of any kind is very much encouraged by the project.
Besides that, there are some minimum requirements.
- Every non-private structure needs at least a brief doc comment explaining the
semantical sense of the structure, as well as potential locking and lifetime
requirements. It is encouraged to have the same minimum documentation for
non-trivial private structures.
- uAPIs must be fully documented with kernel-doc comments; additionally, the
semantical behavior must be explained including potential special or corner
cases.
- The APIs connecting the 1st level driver (nova-core) with 2nd level drivers
must be fully documented. This includes doc comments, potential locking and
lifetime requirements, as well as example code if applicable.
- Abbreviations must be explained when introduced; terminology must be uniquely
defined.
- Register addresses, layouts, shift values and masks must be defined properly;
unless obvious, the semantical sense must be documented. This only applies if
the author is able to obtain the corresponding information.
Acceptance Criteria
===================
- Patches must only be applied if reviewed by at least one other person on the
mailing list; this also applies for maintainers.
linux-7.2.3/Documentation/gpu/nova/index.rst
+34
View File
@@ -0,0 +1,34 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
=======================
nova NVIDIA GPU drivers
=======================
The nova driver project consists out of two separate drivers nova-core and
nova-drm and intends to supersede the nouveau driver for NVIDIA GPUs based on
the GPU System Processor (GSP).
The following documents apply to both nova-core and nova-drm.
.. toctree::
:titlesonly:
guidelines
nova-core
=========
The nova-core driver is the core driver for NVIDIA GPUs based on GSP. nova-core,
as the 1st level driver, provides an abstraction around the GPUs hard- and
firmware interfaces providing a common base for 2nd level drivers, such as the
vGPU manager VFIO driver and the nova-drm driver.
.. toctree::
:titlesonly:
core/guidelines
core/todo
core/vbios
core/devinit
core/fwsec
core/falcon