1.. SPDX-License-Identifier: GPL-2.0 2 3================ 4CoreSight - Perf 5================ 6 7 :Author: Carsten Haitzler <carsten.haitzler@arm.com> 8 :Date: June 29th, 2022 9 10Perf is able to locally access CoreSight trace data and store it to the 11output perf data files. This data can then be later decoded to give the 12instructions that were traced for debugging or profiling purposes. You 13can log such data with a perf record command like:: 14 15 perf record -e cs_etm//u testbinary 16 17This would run some test binary (testbinary) until it exits and record 18a perf.data trace file. That file would have AUX sections if CoreSight 19is working correctly. You can dump the content of this file as 20readable text with a command like:: 21 22 perf report --stdio --dump -i perf.data 23 24You should find some sections of this file have AUX data blocks like:: 25 26 0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0 offset: 0 ref: 0x1b614fc1061b0ad1 idx: 0 tid: 531230 cpu: -1 27 28 . ... CoreSight ETM Trace data: size 73168 bytes 29 Idx:0; ID:10; I_ASYNC : Alignment Synchronisation. 30 Idx:12; ID:10; I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 } 31 Idx:17; ID:10; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000; 32 Idx:26; ID:10; I_TRACE_ON : Trace On. 33 Idx:27; ID:10; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS; 34 Idx:38; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE 35 Idx:39; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE 36 Idx:40; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE 37 Idx:41; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN 38 ... 39 40If you see these above, then your system is tracing CoreSight data 41correctly. 42 43To compile perf with CoreSight support in the tools/perf directory do:: 44 45 make CORESIGHT=1 46 47This requires OpenCSD to build. You may install distribution packages 48for the support such as libopencsd and libopencsd-dev or download it 49and build yourself. Upstream OpenCSD is located at: 50 51 https://github.com/Linaro/OpenCSD 52 53For complete information on building perf with CoreSight support and 54more extensive usage look at: 55 56 https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md 57 58 59Kernel CoreSight Support 60------------------------ 61 62You will also want CoreSight support enabled in your kernel config. 63Ensure it is enabled with:: 64 65 CONFIG_CORESIGHT=y 66 67There are various other CoreSight options you probably also want 68enabled like:: 69 70 CONFIG_CORESIGHT_LINKS_AND_SINKS=y 71 CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y 72 CONFIG_CORESIGHT_CATU=y 73 CONFIG_CORESIGHT_SINK_TPIU=y 74 CONFIG_CORESIGHT_SINK_ETBV10=y 75 CONFIG_CORESIGHT_SOURCE_ETM4X=y 76 CONFIG_CORESIGHT_CTI=y 77 CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y 78 79Please refer to the kernel configuration help for more information. 80 81Fine-grained tracing with AUX pause and resume 82---------------------------------------------- 83 84Arm CoreSight may generate a large amount of hardware trace data, which 85will lead to overhead in recording and distract users when reviewing 86profiling result. To mitigate the issue of excessive trace data, Perf 87provides AUX pause and resume functionality for fine-grained tracing. 88 89The AUX pause and resume can be triggered by associated events. These 90events can be ftrace tracepoints (including static and dynamic 91tracepoints) or PMU events (e.g. CPU PMU cycle event). To create a perf 92session with AUX pause / resume, three configuration terms are 93introduced: 94 95- "aux-action=start-paused": it is specified for the cs_etm PMU event to 96 launch in a paused state. 97- "aux-action=pause": an associated event is specified with this term 98 to pause AUX trace. 99- "aux-action=resume": an associated event is specified with this term 100 to resume AUX trace. 101 102Example for triggering AUX pause and resume with ftrace tracepoints:: 103 104 perf record -e cs_etm/aux-action=start-paused/k,syscalls:sys_enter_openat/aux-action=resume/,syscalls:sys_exit_openat/aux-action=pause/ ls 105 106Example for triggering AUX pause and resume with PMU event:: 107 108 perf record -a -e cs_etm/aux-action=start-paused/k \ 109 -e cycles/aux-action=pause,period=10000000/ \ 110 -e cycles/aux-action=resume,period=1050000/ -- sleep 1 111 112Perf test - Verify kernel and userspace perf CoreSight work 113----------------------------------------------------------- 114 115When you run perf test, it will do a lot of self tests. Some of those 116tests will cover CoreSight (only if enabled and on ARM64). You 117generally would run perf test from the tools/perf directory in the 118kernel tree. Some tests will check some internal perf support like: 119 120 Check Arm CoreSight trace data recording and synthesized samples 121 Check Arm SPE trace data recording and synthesized samples 122 123Some others will actually use perf record and some test binaries that 124are in tests/shell/coresight and will collect traces to ensure a 125minimum level of functionality is met. The scripts that launch these 126tests are in the same directory. These will all look like: 127 128 CoreSight / ASM Pure Loop 129 CoreSight / Memcpy 16k 10 Threads 130 CoreSight / Thread Loop 10 Threads - Check TID 131 etc. 132 133These perf record tests will not run if the tool binaries do not exist 134in tests/shell/coresight/\*/ and will be skipped. If you do not have 135CoreSight support in hardware then either do not build perf with 136CoreSight support or remove these binaries in order to not have these 137tests fail and have them skip instead. 138 139These tests will log historical results in the current working 140directory (e.g. tools/perf) and will be named stats-\*.csv like: 141 142 stats-asm_pure_loop-out.csv 143 stats-memcpy_thread-16k_10.csv 144 ... 145 146These statistic files log some aspects of the AUX data sections in 147the perf data output counting some numbers of certain encodings (a 148good way to know that it's working in a very simple way). One problem 149with CoreSight is that given a large enough amount of data needing to 150be logged, some of it can be lost due to the processor not waking up 151in time to read out all the data from buffers etc.. You will notice 152that the amount of data collected can vary a lot per run of perf test. 153If you wish to see how this changes over time, simply run perf test 154multiple times and all these csv files will have more and more data 155appended to it that you can later examine, graph and otherwise use to 156figure out if things have become worse or better. 157 158This means sometimes these tests fail as they don't capture all the 159data needed. This is about tracking quality and amount of data 160produced over time and to see when changes to the Linux kernel improve 161quality of traces. 162 163Be aware that some of these tests take quite a while to run, specifically 164in processing the perf data file and dumping contents to then examine what 165is inside. 166 167You can change where these csv logs are stored by setting the 168PERF_TEST_CORESIGHT_STATDIR environment variable before running perf 169test like:: 170 171 export PERF_TEST_CORESIGHT_STATDIR=/var/tmp 172 perf test 173 174They will also store resulting perf output data in the current 175directory for later inspection like:: 176 177 perf-asm_pure_loop-out.data 178 perf-memcpy_thread-16k_10.data 179 ... 180 181You can alter where the perf data files are stored by setting the 182PERF_TEST_CORESIGHT_DATADIR environment variable such as:: 183 184 PERF_TEST_CORESIGHT_DATADIR=/var/tmp 185 perf test 186 187You may wish to set these above environment variables if you wish to 188keep the output of tests outside of the current working directory for 189longer term storage and examination. 190