523 lines
22 KiB
ReStructuredText
523 lines
22 KiB
ReStructuredText
=====================================================================
|
|
NVIDIA Tegra410 SoC Uncore Performance Monitoring Unit (PMU)
|
|
=====================================================================
|
|
|
|
The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance
|
|
metrics like memory bandwidth, latency, and utilization:
|
|
|
|
* Unified Coherence Fabric (UCF)
|
|
* PCIE
|
|
* PCIE-TGT
|
|
* CPU Memory (CMEM) Latency
|
|
* NVLink-C2C
|
|
* NV-CLink
|
|
* NV-DLink
|
|
|
|
PMU Driver
|
|
----------
|
|
|
|
The PMU driver describes the available events and configuration of each PMU in
|
|
sysfs. Please see the sections below to get the sysfs path of each PMU. Like
|
|
other uncore PMU drivers, the driver provides "cpumask" sysfs attribute to show
|
|
the CPU id used to handle the PMU event. There is also "associated_cpus"
|
|
sysfs attribute, which contains a list of CPUs associated with the PMU instance.
|
|
|
|
UCF PMU
|
|
-------
|
|
|
|
The Unified Coherence Fabric (UCF) in the NVIDIA Tegra410 SoC serves as a
|
|
distributed cache, last level for CPU Memory and CXL Memory, and cache coherent
|
|
interconnect that supports hardware coherence across multiple coherently caching
|
|
agents, including:
|
|
|
|
* CPU clusters
|
|
* GPU
|
|
* PCIe Ordering Controller Unit (OCU)
|
|
* Other IO-coherent requesters
|
|
|
|
The events and configuration options of this PMU device are described in sysfs,
|
|
see /sys/bus/event_source/devices/nvidia_ucf_pmu_<socket-id>.
|
|
|
|
Some of the events available in this PMU can be used to measure bandwidth and
|
|
utilization:
|
|
|
|
* slc_access_rd: count the number of read requests to SLC.
|
|
* slc_access_wr: count the number of write requests to SLC.
|
|
* slc_bytes_rd: count the number of bytes transferred by slc_access_rd.
|
|
* slc_bytes_wr: count the number of bytes transferred by slc_access_wr.
|
|
* mem_access_rd: count the number of read requests to local or remote memory.
|
|
* mem_access_wr: count the number of write requests to local or remote memory.
|
|
* mem_bytes_rd: count the number of bytes transferred by mem_access_rd.
|
|
* mem_bytes_wr: count the number of bytes transferred by mem_access_wr.
|
|
* cycles: counts the UCF cycles.
|
|
|
|
The average bandwidth is calculated as::
|
|
|
|
AVG_SLC_READ_BANDWIDTH_IN_GBPS = SLC_BYTES_RD / ELAPSED_TIME_IN_NS
|
|
AVG_SLC_WRITE_BANDWIDTH_IN_GBPS = SLC_BYTES_WR / ELAPSED_TIME_IN_NS
|
|
AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD / ELAPSED_TIME_IN_NS
|
|
AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR / ELAPSED_TIME_IN_NS
|
|
|
|
The average request rate is calculated as::
|
|
|
|
AVG_SLC_READ_REQUEST_RATE = SLC_ACCESS_RD / CYCLES
|
|
AVG_SLC_WRITE_REQUEST_RATE = SLC_ACCESS_WR / CYCLES
|
|
AVG_MEM_READ_REQUEST_RATE = MEM_ACCESS_RD / CYCLES
|
|
AVG_MEM_WRITE_REQUEST_RATE = MEM_ACCESS_WR / CYCLES
|
|
|
|
More details about what other events are available can be found in Tegra410 SoC
|
|
technical reference manual.
|
|
|
|
The events can be filtered based on source or destination. The source filter
|
|
indicates the traffic initiator to the SLC, e.g local CPU, non-CPU device, or
|
|
remote socket. The destination filter specifies the destination memory type,
|
|
e.g. local system memory (CMEM), local GPU memory (GMEM), or remote memory. The
|
|
local/remote classification of the destination filter is based on the home
|
|
socket of the address, not where the data actually resides. The available
|
|
filters are described in
|
|
/sys/bus/event_source/devices/nvidia_ucf_pmu_<socket-id>/format/.
|
|
|
|
The list of UCF PMU event filters:
|
|
|
|
* Source filter:
|
|
|
|
* src_loc_cpu: if set, count events from local CPU
|
|
* src_loc_noncpu: if set, count events from local non-CPU device
|
|
* src_rem: if set, count events from CPU, GPU, PCIE devices of remote socket
|
|
|
|
* Destination filter:
|
|
|
|
* dst_loc_cmem: if set, count events to local system memory (CMEM) address
|
|
* dst_loc_gmem: if set, count events to local GPU memory (GMEM) address
|
|
* dst_loc_other: if set, count events to local CXL memory address
|
|
* dst_rem: if set, count events to CPU, GPU, and CXL memory address of remote socket
|
|
|
|
If the source is not specified, the PMU will count events from all sources. If
|
|
the destination is not specified, the PMU will count events to all destinations.
|
|
|
|
Example usage:
|
|
|
|
* Count event id 0x0 in socket 0 from all sources and to all destinations::
|
|
|
|
perf stat -a -e nvidia_ucf_pmu_0/event=0x0/
|
|
|
|
* Count event id 0x0 in socket 0 with source filter = local CPU and destination
|
|
filter = local system memory (CMEM)::
|
|
|
|
perf stat -a -e nvidia_ucf_pmu_0/event=0x0,src_loc_cpu=0x1,dst_loc_cmem=0x1/
|
|
|
|
* Count event id 0x0 in socket 1 with source filter = local non-CPU device and
|
|
destination filter = remote memory::
|
|
|
|
perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/
|
|
|
|
PCIE PMU
|
|
--------
|
|
|
|
This PMU is located in the SOC fabric connecting the PCIE root complex (RC) and
|
|
the memory subsystem. It monitors all read/write traffic from the root port(s)
|
|
or a particular BDF in a PCIE RC to local or remote memory. There is one PMU per
|
|
PCIE RC in the SoC. Each RC can have up to 16 lanes that can be bifurcated into
|
|
up to 8 root ports. The traffic from each root port can be filtered using RP or
|
|
BDF filter. For example, specifying "src_rp_mask=0xFF" means the PMU counter will
|
|
capture traffic from all RPs. Please see below for more details.
|
|
|
|
The events and configuration options of this PMU device are described in sysfs,
|
|
see /sys/bus/event_source/devices/nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>.
|
|
|
|
The events in this PMU can be used to measure bandwidth, utilization, and
|
|
latency:
|
|
|
|
* rd_req: count the number of read requests by PCIE device.
|
|
* wr_req: count the number of write requests by PCIE device.
|
|
* rd_bytes: count the number of bytes transferred by rd_req.
|
|
* wr_bytes: count the number of bytes transferred by wr_req.
|
|
* rd_cum_outs: count outstanding rd_req each cycle.
|
|
* cycles: count the clock cycles of SOC fabric connected to the PCIE interface.
|
|
|
|
The average bandwidth is calculated as::
|
|
|
|
AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS
|
|
AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS
|
|
|
|
The average request rate is calculated as::
|
|
|
|
AVG_RD_REQUEST_RATE = RD_REQ / CYCLES
|
|
AVG_WR_REQUEST_RATE = WR_REQ / CYCLES
|
|
|
|
|
|
The average latency is calculated as::
|
|
|
|
FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS
|
|
AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ
|
|
AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ
|
|
|
|
The PMU events can be filtered based on the traffic source and destination.
|
|
The source filter indicates the PCIE devices that will be monitored. The
|
|
destination filter specifies the destination memory type, e.g. local system
|
|
memory (CMEM), local GPU memory (GMEM), or remote memory. The local/remote
|
|
classification of the destination filter is based on the home socket of the
|
|
address, not where the data actually resides. These filters can be found in
|
|
/sys/bus/event_source/devices/nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>/format/.
|
|
|
|
The list of event filters:
|
|
|
|
* Source filter:
|
|
|
|
* src_rp_mask: bitmask of root ports that will be monitored. Each bit in this
|
|
bitmask represents the RP index in the RC. If the bit is set, all devices under
|
|
the associated RP will be monitored. E.g "src_rp_mask=0xF" will monitor
|
|
devices in root port 0 to 3.
|
|
* src_bdf: the BDF that will be monitored. This is a 16-bit value that
|
|
follows formula: (bus << 8) + (device << 3) + (function). For example, the
|
|
value of BDF 27:01.1 is 0x2781.
|
|
* src_bdf_en: enable the BDF filter. If this is set, the BDF filter value in
|
|
"src_bdf" is used to filter the traffic.
|
|
|
|
Note that Root-Port and BDF filters are mutually exclusive and the PMU in
|
|
each RC can only have one BDF filter for the whole counters. If BDF filter
|
|
is enabled, the BDF filter value will be applied to all events.
|
|
|
|
* Destination filter:
|
|
|
|
* dst_loc_cmem: if set, count events to local system memory (CMEM) address
|
|
* dst_loc_gmem: if set, count events to local GPU memory (GMEM) address
|
|
* dst_loc_pcie_p2p: if set, count events to local PCIE peer address
|
|
* dst_loc_pcie_cxl: if set, count events to local CXL memory address
|
|
* dst_rem: if set, count events to remote memory address
|
|
|
|
If the source filter is not specified, the PMU will count events from all root
|
|
ports. If the destination filter is not specified, the PMU will count events
|
|
to all destinations.
|
|
|
|
Example usage:
|
|
|
|
* Count event id 0x0 from root port 0 of PCIE RC-0 on socket 0 targeting all
|
|
destinations::
|
|
|
|
perf stat -a -e nvidia_pcie_pmu_0_rc_0/event=0x0,src_rp_mask=0x1/
|
|
|
|
* Count event id 0x1 from root port 0 and 1 of PCIE RC-1 on socket 0 and
|
|
targeting just local CMEM of socket 0::
|
|
|
|
perf stat -a -e nvidia_pcie_pmu_0_rc_1/event=0x1,src_rp_mask=0x3,dst_loc_cmem=0x1/
|
|
|
|
* Count event id 0x2 from root port 0 of PCIE RC-2 on socket 1 targeting all
|
|
destinations::
|
|
|
|
perf stat -a -e nvidia_pcie_pmu_1_rc_2/event=0x2,src_rp_mask=0x1/
|
|
|
|
* Count event id 0x3 from root port 0 and 1 of PCIE RC-3 on socket 1 and
|
|
targeting just local CMEM of socket 1::
|
|
|
|
perf stat -a -e nvidia_pcie_pmu_1_rc_3/event=0x3,src_rp_mask=0x3,dst_loc_cmem=0x1/
|
|
|
|
* Count event id 0x4 from BDF 01:01.0 of PCIE RC-4 on socket 0 targeting all
|
|
destinations::
|
|
|
|
perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/
|
|
|
|
.. _NVIDIA_T410_PCIE_PMU_RC_Mapping_Section:
|
|
|
|
Mapping the RC# to lspci segment number
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA
|
|
Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space
|
|
for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register
|
|
contains the following information to map PCIE devices under the RP back to its RC# :
|
|
|
|
- Bus# (byte 0xc) : bus number as reported by the lspci output
|
|
- Segment# (byte 0xd) : segment number as reported by the lspci output
|
|
- RP# (byte 0xe) : port number as reported by LnkCap attribute from lspci for a device with Root Port capability
|
|
- RC# (byte 0xf): root complex number associated with the RP
|
|
- Socket# (byte 0x10): socket number associated with the RP
|
|
|
|
Example script for mapping lspci BDF to RC# and socket#::
|
|
|
|
#!/bin/bash
|
|
while read bdf rest; do
|
|
dvsec4_reg=$(lspci -vv -s $bdf | awk '
|
|
/Designated Vendor-Specific: Vendor=10de ID=0004/ {
|
|
match($0, /\[([0-9a-fA-F]+)/, arr);
|
|
print "0x" arr[1];
|
|
exit
|
|
}
|
|
')
|
|
if [ -n "$dvsec4_reg" ]; then
|
|
bus=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xc))).b)
|
|
segment=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xd))).b)
|
|
rp=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xe))).b)
|
|
rc=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xf))).b)
|
|
socket=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0x10))).b)
|
|
echo "$bdf: Bus=$bus, Segment=$segment, RP=$rp, RC=$rc, Socket=$socket"
|
|
fi
|
|
done < <(lspci -d 10de:)
|
|
|
|
Example output::
|
|
|
|
0001:00:00.0: Bus=00, Segment=01, RP=00, RC=00, Socket=00
|
|
0002:80:00.0: Bus=80, Segment=02, RP=01, RC=01, Socket=00
|
|
0002:a0:00.0: Bus=a0, Segment=02, RP=02, RC=01, Socket=00
|
|
0002:c0:00.0: Bus=c0, Segment=02, RP=03, RC=01, Socket=00
|
|
0002:e0:00.0: Bus=e0, Segment=02, RP=04, RC=01, Socket=00
|
|
0003:00:00.0: Bus=00, Segment=03, RP=00, RC=02, Socket=00
|
|
0004:00:00.0: Bus=00, Segment=04, RP=00, RC=03, Socket=00
|
|
0005:00:00.0: Bus=00, Segment=05, RP=00, RC=04, Socket=00
|
|
0005:40:00.0: Bus=40, Segment=05, RP=01, RC=04, Socket=00
|
|
0005:c0:00.0: Bus=c0, Segment=05, RP=02, RC=04, Socket=00
|
|
0006:00:00.0: Bus=00, Segment=06, RP=00, RC=05, Socket=00
|
|
0009:00:00.0: Bus=00, Segment=09, RP=00, RC=00, Socket=01
|
|
000a:80:00.0: Bus=80, Segment=0a, RP=01, RC=01, Socket=01
|
|
000a:a0:00.0: Bus=a0, Segment=0a, RP=02, RC=01, Socket=01
|
|
000a:e0:00.0: Bus=e0, Segment=0a, RP=03, RC=01, Socket=01
|
|
000b:00:00.0: Bus=00, Segment=0b, RP=00, RC=02, Socket=01
|
|
000c:00:00.0: Bus=00, Segment=0c, RP=00, RC=03, Socket=01
|
|
000d:00:00.0: Bus=00, Segment=0d, RP=00, RC=04, Socket=01
|
|
000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01
|
|
000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01
|
|
000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01
|
|
|
|
PCIE-TGT PMU
|
|
------------
|
|
|
|
This PMU is located in the SOC fabric connecting the PCIE root complex (RC) and
|
|
the memory subsystem. It monitors traffic targeting PCIE BAR and CXL HDM ranges.
|
|
There is one PCIE-TGT PMU per PCIE RC in the SoC. Each RC in Tegra410 SoC can
|
|
have up to 16 lanes that can be bifurcated into up to 8 root ports (RP). The PMU
|
|
provides RP filter to count PCIE BAR traffic to each RP and address filter to
|
|
count access to PCIE BAR or CXL HDM ranges. The details of the filters are
|
|
described in the following sections.
|
|
|
|
Mapping the RC# to lspci segment number is similar to the PCIE PMU. Please see
|
|
:ref:`NVIDIA_T410_PCIE_PMU_RC_Mapping_Section` for more info.
|
|
|
|
The events and configuration options of this PMU device are available in sysfs,
|
|
see /sys/bus/event_source/devices/nvidia_pcie_tgt_pmu_<socket-id>_rc_<pcie-rc-id>.
|
|
|
|
The events in this PMU can be used to measure bandwidth and utilization:
|
|
|
|
* rd_req: count the number of read requests to PCIE.
|
|
* wr_req: count the number of write requests to PCIE.
|
|
* rd_bytes: count the number of bytes transferred by rd_req.
|
|
* wr_bytes: count the number of bytes transferred by wr_req.
|
|
* cycles: count the clock cycles of SOC fabric connected to the PCIE interface.
|
|
|
|
The average bandwidth is calculated as::
|
|
|
|
AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS
|
|
AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS
|
|
|
|
The average request rate is calculated as::
|
|
|
|
AVG_RD_REQUEST_RATE = RD_REQ / CYCLES
|
|
AVG_WR_REQUEST_RATE = WR_REQ / CYCLES
|
|
|
|
The PMU events can be filtered based on the destination root port or target
|
|
address range. Filtering based on RP is only available for PCIE BAR traffic.
|
|
Address filter works for both PCIE BAR and CXL HDM ranges. These filters can be
|
|
found in sysfs, see
|
|
/sys/bus/event_source/devices/nvidia_pcie_tgt_pmu_<socket-id>_rc_<pcie-rc-id>/format/.
|
|
|
|
Destination filter settings:
|
|
|
|
* dst_rp_mask: bitmask to select the root port(s) to monitor. E.g. "dst_rp_mask=0xFF"
|
|
corresponds to all root ports (from 0 to 7) in the PCIE RC. Note that this filter is
|
|
only available for PCIE BAR traffic.
|
|
* dst_addr_base: BAR or CXL HDM filter base address.
|
|
* dst_addr_mask: BAR or CXL HDM filter address mask.
|
|
* dst_addr_en: enable BAR or CXL HDM address range filter. If this is set, the
|
|
address range specified by "dst_addr_base" and "dst_addr_mask" will be used to filter
|
|
the PCIE BAR and CXL HDM traffic address. The PMU uses the following comparison
|
|
to determine if the traffic destination address falls within the filter range::
|
|
|
|
(txn's addr & dst_addr_mask) == (dst_addr_base & dst_addr_mask)
|
|
|
|
If the comparison succeeds, then the event will be counted.
|
|
|
|
If the destination filter is not specified, the RP filter will be configured by default
|
|
to count PCIE BAR traffic to all root ports.
|
|
|
|
Example usage:
|
|
|
|
* Count event id 0x0 to root port 0 and 1 of PCIE RC-0 on socket 0::
|
|
|
|
perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_0/event=0x0,dst_rp_mask=0x3/
|
|
|
|
* Count event id 0x1 for accesses to PCIE BAR or CXL HDM address range
|
|
0x10000 to 0x100FF on socket 0's PCIE RC-1::
|
|
|
|
perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/
|
|
|
|
CPU Memory (CMEM) Latency PMU
|
|
-----------------------------
|
|
|
|
This PMU monitors latency events of memory read requests from the edge of the
|
|
Unified Coherence Fabric (UCF) to local CPU DRAM:
|
|
|
|
* RD_REQ counters: count read requests (32B per request).
|
|
* RD_CUM_OUTS counters: accumulated outstanding request counter, which track
|
|
how many cycles the read requests are in flight.
|
|
* CYCLES counter: counts the number of elapsed cycles.
|
|
|
|
The average latency is calculated as::
|
|
|
|
FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS
|
|
AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ
|
|
AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ
|
|
|
|
The events and configuration options of this PMU device are described in sysfs,
|
|
see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_<socket-id>.
|
|
|
|
Example usage::
|
|
|
|
perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}'
|
|
|
|
NVLink-C2C PMU
|
|
--------------
|
|
|
|
This PMU monitors latency events of memory read/write requests that pass through
|
|
the NVIDIA Chip-to-Chip (C2C) interface. Bandwidth events are not available
|
|
in this PMU, unlike the C2C PMU in Grace (Tegra241 SoC).
|
|
|
|
The events and configuration options of this PMU device are available in sysfs,
|
|
see /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_<socket-id>.
|
|
|
|
The list of events:
|
|
|
|
* IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests.
|
|
* IN_RD_REQ: the number of incoming read requests.
|
|
* IN_WR_CUM_OUTS: accumulated outstanding request (in cycles) of incoming write requests.
|
|
* IN_WR_REQ: the number of incoming write requests.
|
|
* OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests.
|
|
* OUT_RD_REQ: the number of outgoing read requests.
|
|
* OUT_WR_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing write requests.
|
|
* OUT_WR_REQ: the number of outgoing write requests.
|
|
* CYCLES: NVLink-C2C interface cycle counts.
|
|
|
|
The incoming events count the reads/writes from remote device to the SoC.
|
|
The outgoing events count the reads/writes from the SoC to remote device.
|
|
|
|
The sysfs /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_<socket-id>/peer
|
|
contains the information about the connected device.
|
|
|
|
When the C2C interface is connected to GPU(s), the user can use the
|
|
"gpu_mask" parameter to filter traffic to/from specific GPU(s). Each bit represents the GPU
|
|
index, e.g. "gpu_mask=0x1" corresponds to GPU 0 and "gpu_mask=0x3" is for GPU 0 and 1.
|
|
The PMU will monitor all GPUs by default if not specified.
|
|
|
|
When connected to another SoC, only the read events are available.
|
|
|
|
The events can be used to calculate the average latency of the read/write requests::
|
|
|
|
C2C_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS
|
|
|
|
IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ
|
|
IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ
|
|
|
|
IN_WR_AVG_LATENCY_IN_CYCLES = IN_WR_CUM_OUTS / IN_WR_REQ
|
|
IN_WR_AVG_LATENCY_IN_NS = IN_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ
|
|
|
|
OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ
|
|
OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ
|
|
|
|
OUT_WR_AVG_LATENCY_IN_CYCLES = OUT_WR_CUM_OUTS / OUT_WR_REQ
|
|
OUT_WR_AVG_LATENCY_IN_NS = OUT_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ
|
|
|
|
Example usage:
|
|
|
|
* Count incoming traffic from all GPUs connected via NVLink-C2C::
|
|
|
|
perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_req/
|
|
|
|
* Count incoming traffic from GPU 0 connected via NVLink-C2C::
|
|
|
|
perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x1/
|
|
|
|
* Count incoming traffic from GPU 1 connected via NVLink-C2C::
|
|
|
|
perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x2/
|
|
|
|
* Count outgoing traffic to all GPUs connected via NVLink-C2C::
|
|
|
|
perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_req/
|
|
|
|
* Count outgoing traffic to GPU 0 connected via NVLink-C2C::
|
|
|
|
perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x1/
|
|
|
|
* Count outgoing traffic to GPU 1 connected via NVLink-C2C::
|
|
|
|
perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x2/
|
|
|
|
NV-CLink PMU
|
|
------------
|
|
|
|
This PMU monitors latency events of memory read requests that pass through
|
|
the NV-CLINK interface. Bandwidth events are not available in this PMU.
|
|
In Tegra410 SoC, the NV-CLink interface is used to connect to another Tegra410
|
|
SoC and this PMU only counts read traffic.
|
|
|
|
The events and configuration options of this PMU device are available in sysfs,
|
|
see /sys/bus/event_source/devices/nvidia_nvclink_pmu_<socket-id>.
|
|
|
|
The list of events:
|
|
|
|
* IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests.
|
|
* IN_RD_REQ: the number of incoming read requests.
|
|
* OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests.
|
|
* OUT_RD_REQ: the number of outgoing read requests.
|
|
* CYCLES: NV-CLINK interface cycle counts.
|
|
|
|
The incoming events count the reads from remote device to the SoC.
|
|
The outgoing events count the reads from the SoC to remote device.
|
|
|
|
The events can be used to calculate the average latency of the read requests::
|
|
|
|
CLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS
|
|
|
|
IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ
|
|
IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ
|
|
|
|
OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ
|
|
OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ
|
|
|
|
Example usage:
|
|
|
|
* Count incoming read traffic from remote SoC connected via NV-CLINK::
|
|
|
|
perf stat -a -e nvidia_nvclink_pmu_0/in_rd_req/
|
|
|
|
* Count outgoing read traffic to remote SoC connected via NV-CLINK::
|
|
|
|
perf stat -a -e nvidia_nvclink_pmu_0/out_rd_req/
|
|
|
|
NV-DLink PMU
|
|
------------
|
|
|
|
This PMU monitors latency events of memory read requests that pass through
|
|
the NV-DLINK interface. Bandwidth events are not available in this PMU.
|
|
In Tegra410 SoC, this PMU only counts CXL memory read traffic.
|
|
|
|
The events and configuration options of this PMU device are available in sysfs,
|
|
see /sys/bus/event_source/devices/nvidia_nvdlink_pmu_<socket-id>.
|
|
|
|
The list of events:
|
|
|
|
* IN_RD_CUM_OUTS: accumulated outstanding read requests (in cycles) to CXL memory.
|
|
* IN_RD_REQ: the number of read requests to CXL memory.
|
|
* CYCLES: NV-DLINK interface cycle counts.
|
|
|
|
The events can be used to calculate the average latency of the read requests::
|
|
|
|
DLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS
|
|
|
|
IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ
|
|
IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / DLINK_FREQ_IN_GHZ
|
|
|
|
Example usage:
|
|
|
|
* Count read events to CXL memory::
|
|
|
|
perf stat -a -e '{nvidia_nvdlink_pmu_0/in_rd_req/,nvidia_nvdlink_pmu_0/in_rd_cum_outs/}'
|