lastdragon/linux-kernel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resguardo inicial

Browse Source
This commit is contained in:
lastdragon
2026-09-06 12:53:21 -06:00
parent 662efe3c2a
commit ad2088d178
94853 changed files with 43200404 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
linux-7.2.3/Documentation/driver-api/cxl/platform/acpi.rst
+76
View File
@@ -0,0 +1,76 @@
.. SPDX-License-Identifier: GPL-2.0
===========
ACPI Tables
===========
ACPI is the "Advanced Configuration and Power Interface", which is a standard
that defines how platforms and OS manage power and configure computer hardware.
For the purpose of this theory of operation, when referring to "ACPI" we will
usually refer to "ACPI Tables" - which are the way a platform (BIOS/EFI)
communicates static configuration information to the operation system.
The Following ACPI tables contain *static* configuration and performance data
about CXL devices.
.. toctree::
:maxdepth: 1
acpi/cedt.rst
acpi/srat.rst
acpi/hmat.rst
acpi/slit.rst
acpi/dsdt.rst
The SRAT table may also contain generic port/initiator content that is intended
to describe the generic port, but not information about the rest of the path to
the endpoint.
Linux uses these tables to configure kernel resources for statically configured
(by BIOS/EFI) CXL devices, such as:
- NUMA nodes
- Memory Tiers
- NUMA Abstract Distances
- SystemRAM Memory Regions
- Weighted Interleave Node Weights
ACPI Debugging
==============
The :code:`acpidump -b` command dumps the ACPI tables into binary format.
The :code:`iasl -d` command disassembles the files into human readable format.
Example :code:`acpidump -b && iasl -d cedt.dat` ::
[000h 0000 4] Signature : "CEDT" [CXL Early Discovery Table]
Common Issues
-------------
Most failures described here result in a failure of the driver to surface
memory as a DAX device and/or kmem.
* CEDT CFMWS targets list UIDs do not match CEDT CHBS UIDs.
* CEDT CFMWS targets list UIDs do not match DSDT CXL Host Bridge UIDs.
* CEDT CFMWS Restriction Bits are not correct.
* CEDT CFMWS Memory regions are poorly aligned.
* CEDT CFMWS Memory regions spans a platform memory hole.
* CEDT CHBS UIDs do not match DSDT CXL Host Bridge UIDs.
* CEDT CHBS Specification version is incorrect.
* SRAT is missing regions described in CEDT CFMWS.
* Result: failure to create a NUMA node for the region, or
region is placed in wrong node.
* HMAT is missing data for regions described in CEDT CFMWS.
* Result: NUMA node being placed in the wrong memory tier.
* SLIT has bad data.
* Result: Lots of performance mechanisms in the kernel will be very unhappy.
All of these issues will appear to users as if the driver is failing to
support CXL - when in reality they are all the failure of a platform to
configure the ACPI tables correctly.
linux-7.2.3/Documentation/driver-api/cxl/platform/acpi/cedt.rst
+62
View File
@@ -0,0 +1,62 @@
.. SPDX-License-Identifier: GPL-2.0
================================
CEDT - CXL Early Discovery Table
================================
The CXL Early Discovery Table is generated by BIOS to describe the CXL memory
regions configured at boot by the BIOS.
CHBS
====
The CXL Host Bridge Structure describes CXL host bridges. Other than describing
device register information, it reports the specific host bridge UID for this
host bridge. These host bridge ID's will be referenced in other tables.
Example ::
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000007 <- Host bridge _UID
Specification version : 00000001
Reserved : 00000000
Register base : 0000010370400000
Register length : 0000000000010000
CFMWS
=====
The CXL Fixed Memory Window structure describes a memory region associated
with one or more CXL host bridges (as described by the CHBS). It additionally
describes any inter-host-bridge interleave configuration that may have been
programmed by BIOS.
Example ::
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 000000C050000000 <- Memory Region
Window size : 0000003CA0000000
Interleave Members (2^n) : 01 <- Interleave configuration
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007 <- Host Bridge _UID
Next Target : 00000006 <- Host Bridge _UID
The restriction field dictates what this SPA range may be used for (memory type,
voltile vs persistent, etc). One or more bits may be set. ::
Bit[0]: CXL Type 2 Memory
Bit[1]: CXL Type 3 Memory
Bit[2]: Volatile Memory
Bit[3]: Persistent Memory
Bit[4]: Fixed Config (HPA cannot be reused)
INTRA-host-bridge interleave (multiple devices on one host bridge) is NOT
reported in this structure, and is solely defined via CXL device decoder
programming (host bridge and endpoint decoders).
linux-7.2.3/Documentation/driver-api/cxl/platform/acpi/dsdt.rst
+28
View File
@@ -0,0 +1,28 @@
.. SPDX-License-Identifier: GPL-2.0
==============================================
DSDT - Differentiated system Description Table
==============================================
This table describes what peripherals a machine has.
This table's UIDs for CXL devices - specifically host bridges, must be
consistent with the contents of the CEDT, otherwise the CXL driver will
fail to probe correctly.
Example Compute Express Link Host Bridge ::
Scope (_SB)
{
Device (S0D0)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
Name (_CID, Package (0x02) // _CID: Compatible ID
{
EisaId ("PNP0A08") /* PCI Express Bus */,
EisaId ("PNP0A03") /* PCI Bus */
})
...
Name (_UID, 0x05) // _UID: Unique ID
...
}
linux-7.2.3/Documentation/driver-api/cxl/platform/acpi/hmat.rst
+32
View File
@@ -0,0 +1,32 @@
.. SPDX-License-Identifier: GPL-2.0
===========================================
HMAT - Heterogeneous Memory Attribute Table
===========================================
The Heterogeneous Memory Attributes Table contains information such as cache
attributes and bandwidth and latency details for memory proximity domains.
For the purpose of this document, we will only discuss the SSLIB entry.
SLLBI
=====
The System Locality Latency and Bandwidth Information records latency and
bandwidth information for proximity domains.
This table is used by Linux to configure interleave weights and memory tiers.
Example (Heavily truncated for brevity) ::
Structure Type : 0001 [SLLBI]
Data Type : 00 <- Latency
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Entry : 0080 <- DRAM LTC
Entry : 0100 <- CXL LTC
Structure Type : 0001 [SLLBI]
Data Type : 03 <- Bandwidth
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Entry : 1200 <- DRAM BW
Entry : 0200 <- CXL BW
linux-7.2.3/Documentation/driver-api/cxl/platform/acpi/slit.rst
+21
View File
@@ -0,0 +1,21 @@
.. SPDX-License-Identifier: GPL-2.0
========================================
SLIT - System Locality Information Table
========================================
The system locality information table provides "abstract distances" between
accessor and memory nodes. Node without initiators (cpus) are infinitely (FF)
distance away from all other nodes.
The abstract distance described in this table does not describe any real
latency of bandwidth information.
Example ::
Signature : "SLIT" [System Locality Information Table]
Localities : 0000000000000004
Locality 0 : 10 20 20 30
Locality 1 : 20 10 30 20
Locality 2 : FF FF 0A FF
Locality 3 : FF FF FF 0A
linux-7.2.3/Documentation/driver-api/cxl/platform/acpi/srat.rst
+71
View File
@@ -0,0 +1,71 @@
.. SPDX-License-Identifier: GPL-2.0
=====================================
SRAT - Static Resource Affinity Table
=====================================
The System/Static Resource Affinity Table describes resource (CPU, Memory)
affinity to "Proximity Domains". This table is technically optional, but for
performance information (see "HMAT") to be enumerated by linux it must be
present.
There is a careful dance between the CEDT and SRAT tables and how NUMA nodes are
created. If things don't look quite the way you expect - check the SRAT Memory
Affinity entries and CEDT CFMWS to determine what your platform actually
supports in terms of flexible topologies.
The SRAT may statically assign portions of a CFMWS SPA range to a specific
proximity domains. See linux numa creation for more information about how
this presents in the NUMA topology.
Proximity Domain
================
A proximity domain is ROUGHLY equivalent to "NUMA Node" - though a 1-to-1
mapping is not guaranteed. There are scenarios where "Proximity Domain 4" may
map to "NUMA Node 3", for example. (See "NUMA Node Creation")
Memory Affinity
===============
Generally speaking, if a host does any amount of CXL fabric (decoder)
programming in BIOS - an SRAT entry for that memory needs to be present.
Example ::
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000001 <- NUMA Node 1
Reserved1 : 0000
Base Address : 000000C050000000 <- Physical Memory Region
Address Length : 0000003CA0000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Generic Port Affinity
=====================
The Generic Port Affinity subtable provides an association between a proximity
domain and a device handle representing a Generic Port such as a CXL host
bridge. With the association, latency and bandwidth numbers can be retrieved
from the SRAT for the path between CPU(s) (initiator) and the Generic Port.
This is used to construct performance coordinates for hotplugged CXL DEVICES,
which cannot be enumerated at boot by platform firmware.
Example ::
Subtable Type : 06 [Generic Port Affinity]
Length : 20 <- 32d, length of table
Reserved : 00
Device Handle Type : 00 <- 0 - ACPI, 1 - PCI
Proximity Domain : 00000001
Device Handle : ACPI0016:01
Flags : 00000001 <- Bit 0 (Enabled)
Reserved : 00000000
The Proximity Domain is matched up to the :doc:`HMAT <hmat>` SSLBI Target
Proximity Domain List for the related latency or bandwidth numbers. Those
performance numbers are tied to a CXL host bridge via the Device Handle.
The driver uses the association to retrieve the Generic Port performance
numbers for the whole CXL path access coordinates calculation.
linux-7.2.3/Documentation/driver-api/cxl/platform/bios-and-efi.rst
+285
View File
@@ -0,0 +1,285 @@
.. SPDX-License-Identifier: GPL-2.0
======================
BIOS/EFI Configuration
======================
BIOS and EFI are largely responsible for configuring static information about
devices (or potential future devices) such that Linux can build the appropriate
logical representations of these devices.
At a high level, this is what occurs during this phase of configuration.
* The bootloader starts the BIOS/EFI.
* BIOS/EFI do early device probe to determine static configuration
* BIOS/EFI creates ACPI Tables that describe static config for the OS
* BIOS/EFI create the system memory map (EFI Memory Map, E820, etc)
* BIOS/EFI calls :code:`start_kernel` and begins the Linux Early Boot process.
Much of what this section is concerned with is ACPI Table production and
static memory map configuration. More detail on these tables can be found
at :doc:`ACPI Tables <acpi>`.
.. note::
Platform Vendors should read carefully, as this sections has recommendations
on physical memory region size and alignment, memory holes, HDM interleave,
and what linux expects of HDM decoders trying to work with these features.
Linux Expectations of BIOS/EFI Software
=======================================
Linux expects BIOS/EFI software to construct sufficient ACPI tables (such as
CEDT, SRAT, HMAT, etc) and platform-specific configurations (such as HPA spaces
and host-bridge interleave configurations) to allow the Linux driver to
subsequently configure the devices in the CXL fabric at runtime.
Programming of HDM decoders and switch ports is not required, and may be
deferred to the CXL driver based on admin policy (e.g. udev rules).
Some platforms may require pre-programming HDM decoders and locking them
due to quirks (see: Zen5 address translation), but this is not the normal,
"expected" configuration path. This should be avoided if possible.
Some platforms may wish to pre-configure these resources to bring memory
up without requiring CXL driver support. These platform vendors should
test their configurations with the existing CXL driver and provide driver
support for their auto-configurations if features like RAS are required.
Platforms requiring boot-time programming and/or locking of CXL fabric
components may prevent features, such as device hot-plug, from working.
UEFI Settings
=============
If your platform supports it, the :code:`uefisettings` command can be used to
read/write EFI settings. Changes will be reflected on the next reboot. Kexec
is not a sufficient reboot.
One notable configuration here is the EFI_MEMORY_SP (Specific Purpose) bit.
When this is enabled, this bit tells linux to defer management of a memory
region to a driver (in this case, the CXL driver). Otherwise, the memory is
treated as "normal memory", and is exposed to the page allocator during
:code:`__init`.
uefisettings examples
---------------------
:code:`uefisettings identify` ::
uefisettings identify
bios_vendor: xxx
bios_version: xxx
bios_release: xxx
bios_date: xxx
product_name: xxx
product_family: xxx
product_version: xxx
On some AMD platforms, the :code:`EFI_MEMORY_SP` bit is set via the :code:`CXL
Memory Attribute` field. This may be called something else on your platform.
:code:`uefisettings get "CXL Memory Attribute"` ::
selector: xxx
...
question: Question {
name: "CXL Memory Attribute",
answer: "Enabled",
...
}
Physical Memory Map
===================
Physical Address Region Alignment
---------------------------------
As of Linux v6.14, the hotplug memory system requires memory regions to be
uniform in size and alignment. While the CXL specification allows for memory
regions as small as 256MB, the supported memory block size and alignment for
hotplugged memory is architecture-defined.
A Linux memory blocks may be as small as 128MB and increase in powers of two.
* On ARM, the default block size and alignment is either 128MB or 256MB.
* On x86, the default block size is 256MB, and increases to 2GB as the
capacity of the system increases up to 64GB.
For best support across versions, platform vendors should place CXL memory at
a 2GB aligned base address, and regions should be 2GB aligned. This also helps
prevent the creating thousands of memory devices (one per block).
Memory Holes
------------
Holes in the memory map are tricky. Consider a 4GB device located at base
address 0x100000000, but with the following memory map ::
---------------------
| 0x100000000 |
| CXL |
| 0x1BFFFFFFF |
---------------------
| 0x1C0000000 |
| MEMORY HOLE |
| 0x1FFFFFFFF |
---------------------
| 0x200000000 |
| CXL CONT. |
| 0x23FFFFFFF |
---------------------
There are two issues to consider:
* decoder programming, and
* memory block alignment.
If your architecture requires 2GB uniform size and aligned memory blocks, the
only capacity Linux is capable of mapping (as of v6.14) would be the capacity
from `0x100000000-0x180000000`. The remaining capacity will be stranded, as
they are not of 2GB aligned length.
Assuming your architecture and memory configuration allows 1GB memory blocks,
this memory map is supported and this should be presented as multiple CFMWS
in the CEDT that describe each side of the memory hole separately - along with
matching decoders.
Multiple decoders can (and should) be used to manage such a memory hole (see
below), but each chunk of a memory hole should be aligned to a reasonable block
size (larger alignment is always better). If you intend to have memory holes
in the memory map, expect to use one decoder per contiguous chunk of host
physical memory.
As of v6.14, Linux does provide support for memory hotplug of multiple
physical memory regions separated by a memory hole described by a single
HDM decoder.
Decoder Programming
===================
If BIOS/EFI intends to program the decoders to be statically configured,
there are a few things to consider to avoid major pitfalls that will
prevent Linux compatibility. Some of these recommendations are not
required "per the specification", but Linux makes no guarantees of support
otherwise.
Translation Point
-----------------
Per the specification, the only decoders which **TRANSLATE** Host Physical
Address (HPA) to Device Physical Address (DPA) are the **Endpoint Decoders**.
All other decoders in the fabric are intended to route accesses without
translating the addresses.
This is heavily implied by the specification, see: ::
CXL Specification 3.1
8.2.4.20: CXL HDM Decoder Capability Structure
- Implementation Note: CXL Host Bridge and Upstream Switch Port Decoder Flow
- Implementation Note: Device Decoder Logic
Given this, Linux makes a strong assumption that decoders between CPU and
endpoint will all be programmed with addresses ranges that are subsets of
their parent decoder.
Due to some ambiguity in how Architecture, ACPI, PCI, and CXL specifications
"hand off" responsibility between domains, some early adopting platforms
attempted to do translation at the originating memory controller or host
bridge. This configuration requires a platform specific extension to the
driver and is not officially endorsed - despite being supported.
It is *highly recommended* **NOT** to do this; otherwise, you are on your own
to implement driver support for your platform.
Interleave and Configuration Flexibility
----------------------------------------
If providing cross-host-bridge interleave, a CFMWS entry in the :doc:`CEDT
<acpi/cedt>` must be presented with target host-bridges for the interleaved
device sets (there may be multiple behind each host bridge).
If providing intra-host-bridge interleaving, only 1 CFMWS entry in the CEDT is
required for that host bridge - if it covers the entire capacity of the devices
behind the host bridge.
If intending to provide users flexibility in programming decoders beyond the
root, you may want to provide multiple CFMWS entries in the CEDT intended for
different purposes. For example, you may want to consider adding:
1) A CFMWS entry to cover all interleavable host bridges.
2) A CFMWS entry to cover all devices on a single host bridge.
3) A CFMWS entry to cover each device.
A platform may choose to add all of these, or change the mode based on a BIOS
setting. For each CFMWS entry, Linux expects descriptions of the described
memory regions in the :doc:`SRAT <acpi/srat>` to determine the number of
NUMA nodes it should reserve during early boot / init.
As of v6.14, Linux will create a NUMA node for each CEDT CFMWS entry, even if
a matching SRAT entry does not exist; however, this is not guaranteed in the
future and such a configuration should be avoided.
Memory Holes
------------
If your platform includes memory holes interspersed between your CXL memory, it
is recommended to utilize multiple decoders to cover these regions of memory,
rather than try to program the decoders to accept the entire range and expect
Linux to manage the overlap.
For example, consider the Memory Hole described above ::
---------------------
| 0x100000000 |
| CXL |
| 0x1BFFFFFFF |
---------------------
| 0x1C0000000 |
| MEMORY HOLE |
| 0x1FFFFFFFF |
---------------------
| 0x200000000 |
| CXL CONT. |
| 0x23FFFFFFF |
---------------------
Assuming this is provided by a single device attached directly to a host bridge,
Linux would expect the following decoder programming ::
----------------------- -----------------------
| root-decoder-0 | | root-decoder-1 |
| base: 0x100000000 | | base: 0x200000000 |
| size: 0xC0000000 | | size: 0x40000000 |
----------------------- -----------------------
| |
----------------------- -----------------------
| HB-decoder-0 | | HB-decoder-1 |
| base: 0x100000000 | | base: 0x200000000 |
| size: 0xC0000000 | | size: 0x40000000 |
----------------------- -----------------------
| |
----------------------- -----------------------
| ep-decoder-0 | | ep-decoder-1 |
| base: 0x100000000 | | base: 0x200000000 |
| size: 0xC0000000 | | size: 0x40000000 |
----------------------- -----------------------
With a CEDT configuration with two CFMWS describing the above root decoders.
Linux makes no guarantee of support for strange memory hole situations.
Multi-Media Devices
-------------------
The CFMWS field of the CEDT has special restriction bits which describe whether
the described memory region allows volatile or persistent memory (or both). If
the platform intends to support either:
1) A device with multiple media, or
2) Using a persistent memory device as normal memory
A platform may wish to create multiple CEDT CFMWS entries to describe the same
memory, with the intent of allowing the end user flexibility in how that memory
is configured. Linux does not presently have strong requirements in this area.
linux-7.2.3/Documentation/driver-api/cxl/platform/cdat.rst
+118
View File
@@ -0,0 +1,118 @@
.. SPDX-License-Identifier: GPL-2.0
======================================
Coherent Device Attribute Table (CDAT)
======================================
The CDAT provides functional and performance attributes of devices such
as CXL accelerators, switches, or endpoints. The table formatting is
similar to ACPI tables. CDAT data may be parsed by BIOS at boot or may
be enumerated at runtime (after device hotplug, for example).
Terminology:
DPA - Device Physical Address, used by the CXL device to denote the address
it supports for that device.
DSMADHandle - A device unique handle that is associated with a DPA range
defined by the DSMAS table.
===============================================
Device Scoped Memory Affinity Structure (DSMAS)
===============================================
The DSMAS contains information such as DSMADHandle, the DPA Base, and DPA
Length.
This table is used by Linux in conjunction with the Device Scoped Latency and
Bandwidth Information Structure (DSLBIS) to determine the performance
attributes of the CXL device itself.
Example ::
Structure Type : 00 [DSMAS]
Reserved : 00
Length : 0018 <- 24d, size of structure
DSMADHandle : 01
Flags : 00
Reserved : 0000
DPA Base : 0000000040000000 <- 1GiB base
DPA Length : 0000000080000000 <- 2GiB size
==================================================================
Device Scoped Latency and Bandwidth Information Structure (DSLBIS)
==================================================================
This table is used by Linux in conjunction with DSMAS to determine the
performance attributes of a CXL device. The DSLBIS contains latency
and bandwidth information based on DSMADHandle matching.
Example ::
Structure Type : 01 [DSLBIS]
Reserved : 00
Length : 18 <- 24d, size of structure
Handle : 0001 <- DSMAS handle
Flags : 00 <- Matches flag field for HMAT SLLBIS
Data Type : 00 <- Latency
Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS
Entry : 010000000000 <- First byte used here, CXL LTC
Reserved : 0000
Structure Type : 01 [DSLBIS]
Reserved : 00
Length : 18 <- 24d, size of structure
Handle : 0001 <- DSMAS handle
Flags : 00 <- Matches flag field for HMAT SLLBIS
Data Type : 03 <- Bandwidth
Entry Basee Unit : 0000000000001000 <- Entry Base Unit field in HMAT SSLBIS
Entry : 020000000000 <- First byte used here, CXL BW
Reserved : 0000
==================================================================
Switch Scoped Latency and Bandwidth Information Structure (SSLBIS)
==================================================================
The SSLBIS contains information about the latency and bandwidth of a switch.
The table is used by Linux to compute the performance coordinates of a CXL path
from the device to the root port where a switch is part of the path.
Example ::
Structure Type : 05 [SSLBIS]
Reserved : 00
Length : 20 <- 32d, length of record, including SSLB entries
Data Type : 00 <- Latency
Reserved : 000000
Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS
<- SSLB Entry 0
Port X ID : 0100 <- First port, 0100h represents an upstream port
Port Y ID : 0000 <- Second port, downstream port 0
Latency : 0100 <- Port latency
Reserved : 0000
<- SSLB Entry 1
Port X ID : 0100
Port Y ID : 0001
Latency : 0100
Reserved : 0000
Structure Type : 05 [SSLBIS]
Reserved : 00
Length : 18 <- 24d, length of record, including SSLB entry
Data Type : 03 <- Bandwidth
Reserved : 000000
Entry Base Unit : 00000000000000001000 <- Matches Entry Base Unit in HMAT SSLBIS
<- SSLB Entry 0
Port X ID : 0100 <- First port, 0100h represents an upstream port
Port Y ID : FFFF <- Second port, FFFFh indicates any port
Bandwidth : 1200 <- Port bandwidth
Reserved : 0000
The CXL driver uses a combination of CDAT, HMAT, SRAT, and other data to
generate "whole path performance" data for a CXL device.
linux-7.2.3/Documentation/driver-api/cxl/platform/device-hotplug.rst
+130
View File
@@ -0,0 +1,130 @@
.. SPDX-License-Identifier: GPL-2.0
==================
CXL Device Hotplug
==================
Device hotplug refers to *physical* hotplug of a device (addition or removal
of a physical device from the machine).
BIOS/EFI software is expected to configure sufficient resources **at boot
time** to allow hotplugged devices to be configured by software (such as
proximity domains, HPA regions, and host-bridge configurations).
BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
This document covers some examples of those resources, but should not
be considered exhaustive.
Hot-Remove
==========
Hot removal of a device typically requires careful removal of software
constructs (memory regions, associated drivers) which manage these devices.
Hard-removing a CXL.mem device without carefully tearing down driver stacks
is likely to cause the system to machine-check (or at least SIGBUS if memory
access is limited to user space).
Memory Device Hot-Add
=====================
A device present at boot may be associated with a CXL Fixed Memory Window
reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
device, but the construction of the CEDT CFMWS is platform-defined.
Hot-adding a memory device requires this pre-defined, **static** CFMWS to
have sufficient HPA space to describe that device.
There are a few common scenarios to consider.
Single-Endpoint Memory Device Present at Boot
---------------------------------------------
A device present at boot likely had its capacity reported in the
:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
the capacity of the new device will be limited to the original CFMWS capacity.
Adding capacity larger than the original device will cause memory region
creation to fail if the region size is greater than the CFMWS size.
The CFMWS is **static** and cannot be adjusted. Platforms which may expect
different sized devices to be hotplugged must allocate sufficient CFMWS space
**at boot time** to cover all future expected devices.
Multi-Endpoint Memory Device Present at Boot
--------------------------------------------
Non-switch-based Multi-Endpoint devices are outside the scope of what the
CXL specification describes, but they are technically possible. We describe
them here for instructive reasons only - this does not imply Linux support.
A hot-plug capable CXL memory device, such as one which presents multiple
expanders as a single large-capacity device, should report the **maximum
possible capacity** for the device at boot. ::
HB0
RP0
|
[Multi-Endpoint Memory Device]
_____|_____
| |
[Endpoint0] [Empty]
Limiting the size to the capacity preset at boot will limit hot-add support
to replacing capacity that was present at boot.
No CXL Device Present at Boot
-----------------------------
When no CXL memory device is present on boot, some platforms omit the CFMWS
in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible.
This describes the base case for any given device not being present at boot.
If a future possible device is not described in the CEDT at boot, hot-add
of that device is either limited or not possible.
For a platform to support hot-add of a full memory device, it must allocate
a CEDT CFMWS region with sufficient memory capacity to cover all future
potentially added capacity (along with any relevant CEDT CHBS entry).
To support memory hotplug directly on the host bridge/root port, or on a switch
downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
with sufficient resources to support the max possible (or expected) hotplug
memory capacity. ::
HB0 HB1
RP0 RP1 RP2
| | |
Empty Empty USP
________|________
| | | |
DSP DSP DSP DSP
| | | |
All Empty
For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
a pre-configured amount of memory capacity (per host bridge, or host bridge
interleave set), even if no device is attached to Root Ports or Downstream
Ports at boot (as depicted in the figure above).
Interleave Sets
===============
Host Bridge Interleave
----------------------
Host-bridge interleaved memory regions are defined **statically** in the
:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry
describing that interleave must have been provided **at boot**. Hotplugged
devices cannot add host-bridge interleave capabilities at hotplug time.
See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
example to see how a platform can provide this kind of flexibility regarding
hotplugged memory devices. BIOS/EFI software should consider options to
present flexible CEDT configurations with hotplug support.
HDM Interleave
--------------
Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
can be re-programmed after hotplug.
To add or remove a device to/from an existing HDM-applied interleaved region,
that region must be torn down an re-created.
linux-7.2.3/Documentation/driver-api/cxl/platform/example-configs.rst
+13
View File
@@ -0,0 +1,13 @@
.. SPDX-License-Identifier: GPL-2.0
Example Platform Configurations
###############################
.. toctree::
:maxdepth: 1
:caption: Contents
example-configurations/one-dev-per-hb.rst
example-configurations/multi-dev-per-hb.rst
example-configurations/hb-interleave.rst
example-configurations/flexible.rst
linux-7.2.3/Documentation/driver-api/cxl/platform/example-configurations/flexible.rst
+296
View File
@@ -0,0 +1,296 @@
.. SPDX-License-Identifier: GPL-2.0
=====================
Flexible Presentation
=====================
This system has a single socket with two CXL host bridges. Each host bridge
has two CXL memory expanders with a 4GB of memory (32GB total).
On this system, the platform designer wanted to provide the user flexibility
to configure the memory devices in various interleave or NUMA node
configurations. So they provided every combination.
Things to note:
* Cross-Bridge interleave is described in one CFMWS that covers all capacity.
* One CFMWS is also described per-host bridge.
* One CFMWS is also described per-device.
* This SRAT describes one node for each of the above CFMWS.
* The HMAT describes performance for each node in the SRAT.
:doc:`CEDT <../acpi/cedt>`::
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000007
Specification version : 00000001
Reserved : 00000000
Register base : 0000010370400000
Register length : 0000000000010000
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000006
Specification version : 00000001
Reserved : 00000000
Register base : 0000010380800000
Register length : 0000000000010000
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000001000000000
Window size : 0000000400000000
Interleave Members (2^n) : 01
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007
Second Target : 00000006
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000002000000000
Window size : 0000000200000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000002200000000
Window size : 0000000200000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000006
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000003000000000
Window size : 0000000100000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000003100000000
Window size : 0000000100000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000003200000000
Window size : 0000000100000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000006
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000003300000000
Window size : 0000000100000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000006
:doc:`SRAT <../acpi/srat>`::
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000001
Reserved1 : 0000
Base Address : 0000001000000000
Address Length : 0000000400000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000002
Reserved1 : 0000
Base Address : 0000002000000000
Address Length : 0000000200000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000003
Reserved1 : 0000
Base Address : 0000002200000000
Address Length : 0000000200000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000004
Reserved1 : 0000
Base Address : 0000003000000000
Address Length : 0000000100000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000005
Reserved1 : 0000
Base Address : 0000003100000000
Address Length : 0000000100000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000006
Reserved1 : 0000
Base Address : 0000003200000000
Address Length : 0000000100000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000007
Reserved1 : 0000
Base Address : 0000003300000000
Address Length : 0000000100000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
:doc:`HMAT <../acpi/hmat>`::
Structure Type : 0001 [SLLBI]
Data Type : 00 [Latency]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Target Proximity Domain List : 00000002
Target Proximity Domain List : 00000003
Target Proximity Domain List : 00000004
Target Proximity Domain List : 00000005
Target Proximity Domain List : 00000006
Target Proximity Domain List : 00000007
Entry : 0080
Entry : 0100
Entry : 0100
Entry : 0100
Entry : 0100
Entry : 0100
Entry : 0100
Entry : 0100
Structure Type : 0001 [SLLBI]
Data Type : 03 [Bandwidth]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Target Proximity Domain List : 00000002
Target Proximity Domain List : 00000003
Target Proximity Domain List : 00000004
Target Proximity Domain List : 00000005
Target Proximity Domain List : 00000006
Target Proximity Domain List : 00000007
Entry : 1200
Entry : 0400
Entry : 0200
Entry : 0200
Entry : 0100
Entry : 0100
Entry : 0100
Entry : 0100
:doc:`SLIT <../acpi/slit>`::
Signature : "SLIT" [System Locality Information Table]
Localities : 0000000000000003
Locality 0 : 10 20 20 20 20 20 20 20
Locality 1 : FF 0A FF FF FF FF FF FF
Locality 2 : FF FF 0A FF FF FF FF FF
Locality 3 : FF FF FF 0A FF FF FF FF
Locality 4 : FF FF FF FF 0A FF FF FF
Locality 5 : FF FF FF FF FF 0A FF FF
Locality 6 : FF FF FF FF FF FF 0A FF
Locality 7 : FF FF FF FF FF FF FF 0A
:doc:`DSDT <../acpi/dsdt>`::
Scope (_SB)
{
Device (S0D0)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
...
Name (_UID, 0x07) // _UID: Unique ID
}
...
Device (S0D5)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
...
Name (_UID, 0x06) // _UID: Unique ID
}
}
linux-7.2.3/Documentation/driver-api/cxl/platform/example-configurations/hb-interleave.rst
+107
View File
@@ -0,0 +1,107 @@
.. SPDX-License-Identifier: GPL-2.0
============================
Cross-Host-Bridge Interleave
============================
This system has a single socket with two CXL host bridges. Each host bridge
has a single CXL memory expander with a 4GB of memory.
Things to note:
* Cross-Bridge interleave is described.
* The expanders are described by a single CFMWS.
* This SRAT describes one node for both host bridges.
* The HMAT describes a single node's performance.
:doc:`CEDT <../acpi/cedt>`::
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000007
Specification version : 00000001
Reserved : 00000000
Register base : 0000010370400000
Register length : 0000000000010000
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000006
Specification version : 00000001
Reserved : 00000000
Register base : 0000010380800000
Register length : 0000000000010000
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000001000000000
Window size : 0000000200000000
Interleave Members (2^n) : 01
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007
Second Target : 00000006
:doc:`SRAT <../acpi/srat>`::
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000001
Reserved1 : 0000
Base Address : 0000001000000000
Address Length : 0000000200000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
:doc:`HMAT <../acpi/hmat>`::
Structure Type : 0001 [SLLBI]
Data Type : 00 [Latency]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Target Proximity Domain List : 00000002
Entry : 0080
Entry : 0100
Structure Type : 0001 [SLLBI]
Data Type : 03 [Bandwidth]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Target Proximity Domain List : 00000002
Entry : 1200
Entry : 0400
:doc:`SLIT <../acpi/slit>`::
Signature : "SLIT" [System Locality Information Table]
Localities : 0000000000000003
Locality 0 : 10 20
Locality 1 : FF 0A
:doc:`DSDT <../acpi/dsdt>`::
Scope (_SB)
{
Device (S0D0)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
...
Name (_UID, 0x07) // _UID: Unique ID
}
...
Device (S0D5)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
...
Name (_UID, 0x06) // _UID: Unique ID
}
}
linux-7.2.3/Documentation/driver-api/cxl/platform/example-configurations/multi-dev-per-hb.rst
+90
View File
@@ -0,0 +1,90 @@
.. SPDX-License-Identifier: GPL-2.0
================================
Multiple Devices per Host Bridge
================================
In this example system we will have a single socket and one CXL host bridge.
There are two CXL memory expanders with 4GB attached to the host bridge.
Things to note:
* Intra-Bridge interleave is not described here.
* The expanders are described by a single CEDT/CFMWS.
* This CEDT/SRAT describes one node for both devices.
* There is only one proximity domain the HMAT for both devices.
:doc:`CEDT <../acpi/cedt>`::
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000007
Specification version : 00000001
Reserved : 00000000
Register base : 0000010370400000
Register length : 0000000000010000
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000001000000000
Window size : 0000000200000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007
:doc:`SRAT <../acpi/srat>`::
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000001
Reserved1 : 0000
Base Address : 0000001000000000
Address Length : 0000000200000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
:doc:`HMAT <../acpi/hmat>`::
Structure Type : 0001 [SLLBI]
Data Type : 00 [Latency]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Entry : 0080
Entry : 0100
Structure Type : 0001 [SLLBI]
Data Type : 03 [Bandwidth]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Entry : 1200
Entry : 0200
:doc:`SLIT <../acpi/slit>`::
Signature : "SLIT" [System Locality Information Table]
Localities : 0000000000000003
Locality 0 : 10 20
Locality 1 : FF 0A
:doc:`DSDT <../acpi/dsdt>`::
Scope (_SB)
{
Device (S0D0)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
...
Name (_UID, 0x07) // _UID: Unique ID
}
...
}
linux-7.2.3/Documentation/driver-api/cxl/platform/example-configurations/one-dev-per-hb.rst
+136
View File
@@ -0,0 +1,136 @@
.. SPDX-License-Identifier: GPL-2.0
==========================
One Device per Host Bridge
==========================
This system has a single socket with two CXL host bridges. Each host bridge
has a single CXL memory expander with a 4GB of memory.
Things to note:
* Cross-Bridge interleave is not being used.
* The expanders are in two separate but adjacent memory regions.
* This CEDT/SRAT describes one node per device
* The expanders have the same performance and will be in the same memory tier.
:doc:`CEDT <../acpi/cedt>`::
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000007
Specification version : 00000001
Reserved : 00000000
Register base : 0000010370400000
Register length : 0000000000010000
Subtable Type : 00 [CXL Host Bridge Structure]
Reserved : 00
Length : 0020
Associated host bridge : 00000006
Specification version : 00000001
Reserved : 00000000
Register base : 0000010380800000
Register length : 0000000000010000
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000001000000000
Window size : 0000000100000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000007
Subtable Type : 01 [CXL Fixed Memory Window Structure]
Reserved : 00
Length : 002C
Reserved : 00000000
Window base address : 0000001100000000
Window size : 0000000100000000
Interleave Members (2^n) : 00
Interleave Arithmetic : 00
Reserved : 0000
Granularity : 00000000
Restrictions : 0006
QtgId : 0001
First Target : 00000006
:doc:`SRAT <../acpi/srat>`::
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000001
Reserved1 : 0000
Base Address : 0000001000000000
Address Length : 0000000100000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
Subtable Type : 01 [Memory Affinity]
Length : 28
Proximity Domain : 00000002
Reserved1 : 0000
Base Address : 0000001100000000
Address Length : 0000000100000000
Reserved2 : 00000000
Flags (decoded below) : 0000000B
Enabled : 1
Hot Pluggable : 1
Non-Volatile : 0
:doc:`HMAT <../acpi/hmat>`::
Structure Type : 0001 [SLLBI]
Data Type : 00 [Latency]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Target Proximity Domain List : 00000002
Entry : 0080
Entry : 0100
Entry : 0100
Structure Type : 0001 [SLLBI]
Data Type : 03 [Bandwidth]
Target Proximity Domain List : 00000000
Target Proximity Domain List : 00000001
Target Proximity Domain List : 00000002
Entry : 1200
Entry : 0200
Entry : 0200
:doc:`SLIT <../acpi/slit>`::
Signature : "SLIT" [System Locality Information Table]
Localities : 0000000000000003
Locality 0 : 10 20 20
Locality 1 : FF 0A FF
Locality 2 : FF FF 0A
:doc:`DSDT <../acpi/dsdt>`::
Scope (_SB)
{
Device (S0D0)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
...
Name (_UID, 0x07) // _UID: Unique ID
}
...
Device (S0D5)
{
Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) // _HID: Hardware ID
...
Name (_UID, 0x06) // _UID: Unique ID
}
}