admin-guide/perf/nvidia-pmu.rst

9 * NVLink-C2C0
10 * NVLink-C2C1
12 * PCIE
15 ----------
19 PMUs are managed by a common driver "arm-cs-arch-pmu". This driver describes
22 the driver provides "cpumask" sysfs attribute to show the CPU id used to handle
29 -------
31 The SCF PMU monitors system level cache events, CPU traffic, and
32 strongly-ordered (SO) PCIE write traffic to local/remote memory. Please see
37 see /sys/bus/event_sources/devices/nvidia_scf_pmu_<socket-id>.
43    perf stat -a -e nvidia_scf_pmu_0/event=0x0/
47    perf stat -a -e nvidia_scf_pmu_1/event=0x0/
49 NVLink-C2C0 PMU
50 --------------------
52 The NVLink-C2C0 PMU monitors incoming traffic from a GPU/CPU connected with
53 NVLink-C2C (Chip-2-Chip) interconnect. The type of traffic captured by this PMU
60 * NVIDIA Grace CPU Superchip: two Grace CPU SoCs are connected.
63   PCIE device of the remote SoC.
69 see /sys/bus/event_sources/devices/nvidia_nvlink_c2c0_pmu_<socket-id>.
73 * Count event id 0x0 from the GPU/CPU connected with socket 0::
75    perf stat -a -e nvidia_nvlink_c2c0_pmu_0/event=0x0/
77 * Count event id 0x0 from the GPU/CPU connected with socket 1::
79    perf stat -a -e nvidia_nvlink_c2c0_pmu_1/event=0x0/
81 * Count event id 0x0 from the GPU/CPU connected with socket 2::
83    perf stat -a -e nvidia_nvlink_c2c0_pmu_2/event=0x0/
85 * Count event id 0x0 from the GPU/CPU connected with socket 3::
87    perf stat -a -e nvidia_nvlink_c2c0_pmu_3/event=0x0/
89 NVLink-C2C1 PMU
90 -------------------
92 The NVLink-C2C1 PMU monitors incoming traffic from a GPU connected with
93 NVLink-C2C (Chip-2-Chip) interconnect. This PMU captures untranslated GPU
94 traffic, in contrast with NvLink-C2C0 PMU that captures ATS translated traffic.
99 see /sys/bus/event_sources/devices/nvidia_nvlink_c2c1_pmu_<socket-id>.
105    perf stat -a -e nvidia_nvlink_c2c1_pmu_0/event=0x0/
109    perf stat -a -e nvidia_nvlink_c2c1_pmu_1/event=0x0/
113    perf stat -a -e nvidia_nvlink_c2c1_pmu_2/event=0x0/
117    perf stat -a -e nvidia_nvlink_c2c1_pmu_3/event=0x0/
120 ---------------
122 The CNVLink PMU monitors traffic from GPU and PCIE device on remote sockets
123 to local memory. For PCIE traffic, this PMU captures read and relaxed ordered
128 see /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>.
134 /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>/format/rem_socket
139 traffic from remote GPU and PCIE devices.
145    perf stat -a -e nvidia_cnvlink_pmu_0/event=0x0,rem_socket=0xE/
149    perf stat -a -e nvidia_cnvlink_pmu_1/event=0x0,rem_socket=0xD/
153    perf stat -a -e nvidia_cnvlink_pmu_2/event=0x0,rem_socket=0xB/
157    perf stat -a -e nvidia_cnvlink_pmu_3/event=0x0,rem_socket=0x7/
160 PCIE PMU
161 ------------
163 The PCIE PMU monitors all read/write traffic from PCIE root ports to
168 see /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>.
173 /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>/format/root_port
180    perf stat -a -e nvidia_pcie_pmu_0/event=0x0,root_port=0x3/
184    perf stat -a -e nvidia_pcie_pmu_1/event=0x0,root_port=0x3/
189 ----------------
198    * SOCKET-A                      *          * SOCKET-B                      *
201    *                     : PCIE :  *          *  : PCIE :                     *
207    *  : GPU :<--NVLink-->: Grace :<---CNVLink--->: Grace :<--NVLink-->: GPU : *
219    CMEM = CPU Memory (e.g. LPDDR5X)
222   | Following table contains traffic coverage of Grace SoC PMU in socket-A:
226    +--------------+-------+-----------+-----------+-----+----------+----------+
228    +              +-------+-----------+-----------+-----+----------+----------+
229    | Destination  |       |GPU ATS    |GPU Not-ATS|     | Socket-B | Socket-B |
230    |              |PCI R/W|Translated,|Translated | CPU | CPU/PCIE1| GPU/PCIE2|
233    | Local        | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
235    +--------------+-------+-----------+-----------+-----+----------+----------+
236    | Local GMEM   | PCIE  |    N/A    |NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
238    +--------------+-------+-----------+-----------+-----+----------+----------+
239    | Remote       | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
242    +--------------+-------+-----------+-----------+-----+----------+----------+
243    | Remote GMEM  | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
245    +--------------+-------+-----------+-----------+-----+----------+----------+
250 * **NVIDIA Grace CPU Superchip**: two Grace CPU SoCs are connected.
255    * SOCKET-A        *             * SOCKET-B        *
258    *    : PCIE :     *             *    : PCIE :     *
264    *    : Grace :<--------NVLink------->: Grace :    *
276    CMEM = CPU Memory (e.g. LPDDR5X)
279   | Following table contains traffic coverage of Grace SoC PMU in socket-A:
283    +-----------------+-----------+---------+----------+-------------+
285    +                 +-----------+---------+----------+-------------+
286    | Destination     |           |         | Socket-B | Socket-B    |
287    |                 |  PCI R/W  |   CPU   | CPU/PCIE1| PCIE2       |
290    | Local           |  PCIE PMU | SCF PMU | SCF PMU  | NVLink-C2C0 |
292    +-----------------+-----------+---------+----------+-------------+
294    | SYSRAM/CMEM     |  PCIE PMU | SCF PMU |   N/A    |     N/A     |
295    | over NVLink-C2C |           |         |          |             |
296    +-----------------+-----------+---------+----------+-------------+