admin-guide/perf/nvidia-pmu.rst

9 * NVLink-C2C0
10 * NVLink-C2C1
15 ----------
19 PMUs are managed by a common driver "arm-cs-arch-pmu". This driver describes
22 the driver provides "cpumask" sysfs attribute to show the CPU id used to handle
29 -------
32 strongly-ordered (SO) PCIE write traffic to local/remote memory. Please see
37 see /sys/bus/event_sources/devices/nvidia_scf_pmu_<socket-id>.
41 * Count event id 0x0 in socket 0::
43    perf stat -a -e nvidia_scf_pmu_0/event=0x0/
45 * Count event id 0x0 in socket 1::
47    perf stat -a -e nvidia_scf_pmu_1/event=0x0/
49 NVLink-C2C0 PMU
50 --------------------
52 The NVLink-C2C0 PMU monitors incoming traffic from a GPU/CPU connected with
53 NVLink-C2C (Chip-2-Chip) interconnect. The type of traffic captured by this PMU
56 * NVIDIA Grace Hopper Superchip: Hopper GPU is connected with Grace SoC.
58   In this config, the PMU captures GPU ATS translated or EGM traffic from the GPU.
69 see /sys/bus/event_sources/devices/nvidia_nvlink_c2c0_pmu_<socket-id>.
73 * Count event id 0x0 from the GPU/CPU connected with socket 0::
75    perf stat -a -e nvidia_nvlink_c2c0_pmu_0/event=0x0/
77 * Count event id 0x0 from the GPU/CPU connected with socket 1::
79    perf stat -a -e nvidia_nvlink_c2c0_pmu_1/event=0x0/
81 * Count event id 0x0 from the GPU/CPU connected with socket 2::
83    perf stat -a -e nvidia_nvlink_c2c0_pmu_2/event=0x0/
85 * Count event id 0x0 from the GPU/CPU connected with socket 3::
87    perf stat -a -e nvidia_nvlink_c2c0_pmu_3/event=0x0/
89 NVLink-C2C1 PMU
90 -------------------
92 The NVLink-C2C1 PMU monitors incoming traffic from a GPU connected with
93 NVLink-C2C (Chip-2-Chip) interconnect. This PMU captures untranslated GPU
94 traffic, in contrast with NvLink-C2C0 PMU that captures ATS translated traffic.
99 see /sys/bus/event_sources/devices/nvidia_nvlink_c2c1_pmu_<socket-id>.
103 * Count event id 0x0 from the GPU connected with socket 0::
105    perf stat -a -e nvidia_nvlink_c2c1_pmu_0/event=0x0/
107 * Count event id 0x0 from the GPU connected with socket 1::
109    perf stat -a -e nvidia_nvlink_c2c1_pmu_1/event=0x0/
111 * Count event id 0x0 from the GPU connected with socket 2::
113    perf stat -a -e nvidia_nvlink_c2c1_pmu_2/event=0x0/
115 * Count event id 0x0 from the GPU connected with socket 3::
117    perf stat -a -e nvidia_nvlink_c2c1_pmu_3/event=0x0/
120 ---------------
122 The CNVLink PMU monitors traffic from GPU and PCIE device on remote sockets
128 see /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>.
134 /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>/format/rem_socket
139 traffic from remote GPU and PCIE devices.
143 * Count event id 0x0 for the traffic from remote socket 1, 2, and 3 to socket 0::
145    perf stat -a -e nvidia_cnvlink_pmu_0/event=0x0,rem_socket=0xE/
147 * Count event id 0x0 for the traffic from remote socket 0, 2, and 3 to socket 1::
149    perf stat -a -e nvidia_cnvlink_pmu_1/event=0x0,rem_socket=0xD/
151 * Count event id 0x0 for the traffic from remote socket 0, 1, and 3 to socket 2::
153    perf stat -a -e nvidia_cnvlink_pmu_2/event=0x0,rem_socket=0xB/
155 * Count event id 0x0 for the traffic from remote socket 0, 1, and 2 to socket 3::
157    perf stat -a -e nvidia_cnvlink_pmu_3/event=0x0,rem_socket=0x7/
161 ------------
168 see /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>.
173 /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>/format/root_port
178 * Count event id 0x0 from root port 0 and 1 of socket 0::
180    perf stat -a -e nvidia_pcie_pmu_0/event=0x0,root_port=0x3/
182 * Count event id 0x0 from root port 0 and 1 of socket 1::
184    perf stat -a -e nvidia_pcie_pmu_1/event=0x0,root_port=0x3/
189 ----------------
193 * **NVIDIA Grace Hopper Superchip**: Hopper GPU is connected with Grace SoC.
198    * SOCKET-A                      *          * SOCKET-B                      *
207    *  : GPU :<--NVLink-->: Grace :<---CNVLink--->: Grace :<--NVLink-->: GPU : *
218    GMEM = GPU Memory (e.g. HBM)
222   | Following table contains traffic coverage of Grace SoC PMU in socket-A:
226    +--------------+-------+-----------+-----------+-----+----------+----------+
228    +              +-------+-----------+-----------+-----+----------+----------+
229    | Destination  |       |GPU ATS    |GPU Not-ATS|     | Socket-B | Socket-B |
230    |              |PCI R/W|Translated,|Translated | CPU | CPU/PCIE1| GPU/PCIE2|
233    | Local        | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
235    +--------------+-------+-----------+-----------+-----+----------+----------+
236    | Local GMEM   | PCIE  |    N/A    |NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
238    +--------------+-------+-----------+-----------+-----+----------+----------+
239    | Remote       | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
242    +--------------+-------+-----------+-----------+-----+----------+----------+
243    | Remote GMEM  | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
245    +--------------+-------+-----------+-----------+-----+----------+----------+
255    * SOCKET-A        *             * SOCKET-B        *
264    *    : Grace :<--------NVLink------->: Grace :    *
275    GMEM = GPU Memory (e.g. HBM)
279   | Following table contains traffic coverage of Grace SoC PMU in socket-A:
283    +-----------------+-----------+---------+----------+-------------+
285    +                 +-----------+---------+----------+-------------+
286    | Destination     |           |         | Socket-B | Socket-B    |
290    | Local           |  PCIE PMU | SCF PMU | SCF PMU  | NVLink-C2C0 |
292    +-----------------+-----------+---------+----------+-------------+
295    | over NVLink-C2C |           |         |          |             |
296    +-----------------+-----------+---------+----------+-------------+