Documentation/admin-guide/workload-tracing.rst

1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
7 :Authors: - Shuah Khan <skhan@linuxfoundation.org>
8           - Shefali Sharma <sshefali021@gmail.com>
9 :maintained-by: Shuah Khan <skhan@linuxfoundation.org>
14  * Understanding system resources necessary to build and run a workload
21    `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_,
22    `stress-ng <https://www.mankier.com/1/stress-ng>`_,
23    `paxtest <https://github.com/opntr/paxtest-freebsd>`_.
30 `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_ is a
50 How do we gather fine-grained system information?
65 to gather fine-grained information on a workload's usage of system resources.
67 We used strace to trace the perf, stress-ng, paxtest workloads to illustrate
77 install other tools that aren’t usually included to build Linux kernel.
78 Please note that the following works on Debian based distributions. You
81 Install tools to build Linux kernel and tools in kernel repository.
85   sudo apt-get install build-essential flex bison yacc
86   sudo apt install libelf-dev systemtap-sdt-dev libslang2-dev libperl-dev libdw-dev
90   sudo apt-get install cscope
92 Install stress-ng and paxtest::
94   apt-get install stress-ng
95   apt-get install paxtest
100 As mentioned earlier, we used strace to trace perf bench, stress-ng and
107 -------------------------
109 The perf bench command contains multiple multi-threaded microkernel
114 integrate transparently, and use performance-rich tooling subsystems.
116 Stress-ng netdev stressor workload
117 ----------------------------------
119 stress-ng is used for performing stress testing on the kernel. It allows
121 interfaces of the OS kernel, using "stressor-s". They are available for
124 to the `stress-ng man-page <https://www.mankier.com/1/stress-ng>`_ to
125 find the description of all the available stressor-s. The netdev stressor
130 -----------------------
153 In addition, strace can generate run-time statistics on times, calls, and
163 Running strace -c generates a report of the percentage of time spent in each
169  * Verbose mode usage: strace -v <command>
170  * Gather statistics: strace -c <command>
172 We used the “-c” option to gather fine-grained run-time statistics in use
176  * stress-ng
183 line tool for browsing C, C++ or Java code-bases. We can use it to find
192 Let’s checkout the latest Linux repository and build cscope database::
196   cscope -R -p10  # builds cscope.out database before starting browse session
197   cscope -d -p10  # starts browse session on cscope.out database
199 Note: Run "cscope -R -p10" to build the database and c"scope -d -p10" to
201 To get out of this mode press ctrl+d. -p option is used to specify the
202 number of file path components to display. -p10 is optimal for browsing
215 so and then build kernel and perf tool::
219   make -j3 all
228 the perf tool, run "perf -h".
231 ---------
238 ----------
239 The perf bench command contains multiple multi-threaded microkernel
244 integrate transparently, and use performance-rich tooling.
254 What is stress-ng and how do we use it?
257 As mentioned earlier, stress-ng is used for performing stress testing on
259 computer, as well as interfaces of the OS kernel, using stressor-s. They
274   stress-ng --netdev 1 -t 60 --metrics command.
284   perf record stress-ng --netdev 1 -t 60 --metrics command.
300 of the kernel testing for vulnerabilities. We focus on the kiddie mode here
306 The following command can be used to view resulting report in call-graph
309   perf record --call-graph dwarf paxtest kiddie
310   perf report --stdio
318 -------------------------------
322  strace -c perf bench all
329 +-------------------+-----------+-----------------+-------------------------+
333 +-------------------+-----------+-----------------+-------------------------+
335 +-------------------+-----------+-----------------+-------------------------+
337 +-------------------+-----------+-----------------+-------------------------+
339 +-------------------+-----------+-----------------+-------------------------+
341 +-------------------+-----------+-----------------+-------------------------+
343 +-------------------+-----------+-----------------+-------------------------+
345 +-------------------+-----------+-----------------+-------------------------+
347 +-------------------+-----------+-----------------+-------------------------+
349 +-------------------+-----------+-----------------+-------------------------+
351 +-------------------+-----------+-----------------+-------------------------+
353 +-------------------+-----------+-----------------+-------------------------+
355 +-------------------+-----------+-----------------+-------------------------+
357 +-------------------+-----------+-----------------+-------------------------+
359 +-------------------+-----------+-----------------+-------------------------+
361 +-------------------+-----------+-----------------+-------------------------+
363 +-------------------+-----------+-----------------+-------------------------+
365 +-------------------+-----------+-----------------+-------------------------+
367 +-------------------+-----------+-----------------+-------------------------+
369 +-------------------+-----------+-----------------+-------------------------+
371 +-------------------+-----------+-----------------+-------------------------+
373 +-------------------+-----------+-----------------+-------------------------+
375 +-------------------+-----------+-----------------+-------------------------+
377 +-------------------+-----------+-----------------+-------------------------+
379 +-------------------+-----------+-----------------+-------------------------+
381 +-------------------+-----------+-----------------+-------------------------+
383 +-------------------+-----------+-----------------+-------------------------+
385 +-------------------+-----------+-----------------+-------------------------+
387 +-------------------+-----------+-----------------+-------------------------+
389 +-------------------+-----------+-----------------+-------------------------+
391 +-------------------+-----------+-----------------+-------------------------+
393 +-------------------+-----------+-----------------+-------------------------+
395 +-------------------+-----------+-----------------+-------------------------+
397 +-------------------+-----------+-----------------+-------------------------+
399 +-------------------+-----------+-----------------+-------------------------+
401 +-------------------+-----------+-----------------+-------------------------+
403 +-------------------+-----------+-----------------+-------------------------+
405 +-------------------+-----------+-----------------+-------------------------+
407 +-------------------+-----------+-----------------+-------------------------+
409 +-------------------+-----------+-----------------+-------------------------+
411 +-------------------+-----------+-----------------+-------------------------+
413 +-------------------+-----------+-----------------+-------------------------+
415 +-------------------+-----------+-----------------+-------------------------+
417 +-------------------+-----------+-----------------+-------------------------+
419 +-------------------+-----------+-----------------+-------------------------+
421 +-------------------+-----------+-----------------+-------------------------+
423 +-------------------+-----------+-----------------+-------------------------+
425 +-------------------+-----------+-----------------+-------------------------+
427 Tracing stress-ng netdev stressor workload
428 ------------------------------------------
430 Run the following command to trace stress-ng netdev stressor workload::
432   strace -c  stress-ng --netdev 1 -t 60 --metrics
439 +-------------------+-----------+-----------------+-------------------------+
443 +-------------------+-----------+-----------------+-------------------------+
445 +-------------------+-----------+-----------------+-------------------------+
447 +-------------------+-----------+-----------------+-------------------------+
449 +-------------------+-----------+-----------------+-------------------------+
451 +-------------------+-----------+-----------------+-------------------------+
453 +-------------------+-----------+-----------------+-------------------------+
455 +-------------------+-----------+-----------------+-------------------------+
457 +-------------------+-----------+-----------------+-------------------------+
459 +-------------------+-----------+-----------------+-------------------------+
461 +-------------------+-----------+-----------------+-------------------------+
463 +-------------------+-----------+-----------------+-------------------------+
465 +-------------------+-----------+-----------------+-------------------------+
467 +-------------------+-----------+-----------------+-------------------------+
469 +-------------------+-----------+-----------------+-------------------------+
471 +-------------------+-----------+-----------------+-------------------------+
473 +-------------------+-----------+-----------------+-------------------------+
475 +-------------------+-----------+-----------------+-------------------------+
477 +-------------------+-----------+-----------------+-------------------------+
479 +-------------------+-----------+-----------------+-------------------------+
481 +-------------------+-----------+-----------------+-------------------------+
483 +-------------------+-----------+-----------------+-------------------------+
485 +-------------------+-----------+-----------------+-------------------------+
487 +-------------------+-----------+-----------------+-------------------------+
489 +-------------------+-----------+-----------------+-------------------------+
491 +-------------------+-----------+-----------------+-------------------------+
493 +-------------------+-----------+-----------------+-------------------------+
495 +-------------------+-----------+-----------------+-------------------------+
497 +-------------------+-----------+-----------------+-------------------------+
499 +-------------------+-----------+-----------------+-------------------------+
501 +-------------------+-----------+-----------------+-------------------------+
503 +-------------------+-----------+-----------------+-------------------------+
505 +-------------------+-----------+-----------------+-------------------------+
507 +-------------------+-----------+-----------------+-------------------------+
509 +-------------------+-----------+-----------------+-------------------------+
511 +-------------------+-----------+-----------------+-------------------------+
513 +-------------------+-----------+-----------------+-------------------------+
515 +-------------------+-----------+-----------------+-------------------------+
517 +-------------------+-----------+-----------------+-------------------------+
519 +-------------------+-----------+-----------------+-------------------------+
522 -------------------------------
526  strace -c paxtest kiddie
533 +-------------------+-----------+-----------------+----------------------+
537 +-------------------+-----------+-----------------+----------------------+
539 +-------------------+-----------+-----------------+----------------------+
541 +-------------------+-----------+-----------------+----------------------+
543 +-------------------+-----------+-----------------+----------------------+
545 +-------------------+-----------+-----------------+----------------------+
547 +-------------------+-----------+-----------------+----------------------+
549 +-------------------+-----------+-----------------+----------------------+
551 +-------------------+-----------+-----------------+----------------------+
553 +-------------------+-----------+-----------------+----------------------+
555 +-------------------+-----------+-----------------+----------------------+
557 +-------------------+-----------+-----------------+----------------------+
559 +-------------------+-----------+-----------------+----------------------+
561 +-------------------+-----------+-----------------+----------------------+
563 +-------------------+-----------+-----------------+----------------------+
565 +-------------------+-----------+-----------------+----------------------+
567 +-------------------+-----------+-----------------+----------------------+
569 +-------------------+-----------+-----------------+----------------------+
571 +-------------------+-----------+-----------------+----------------------+
573 +-------------------+-----------+-----------------+----------------------+
575 +-------------------+-----------+-----------------+----------------------+
577 +-------------------+-----------+-----------------+----------------------+
579 +-------------------+-----------+-----------------+----------------------+
581 +-------------------+-----------+-----------------+----------------------+
583 +-------------------+-----------+-----------------+----------------------+
585 +-------------------+-----------+-----------------+----------------------+
587 +-------------------+-----------+-----------------+----------------------+
589 +-------------------+-----------+-----------------+----------------------+
594 This document is intended to be used as a guide on how to gather fine-grained
600 …sed by OpenAPS <https://elisa.tech/blog/2022/02/02/discovery-linux-kernel-subsystems-used-by-opena…
601 …* `ELISA-White-Papers-Discovering Linux kernel subsystems used by a workload <https://github.com/e…
602  * `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_
603  * `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_
604  * `paxtest README <https://github.com/opntr/paxtest-freebsd/blob/hardenedbsd/0.9.14-hbsd/README>`_
605  * `stress-ng <https://www.mankier.com/1/stress-ng>`_
606 …ng system status and performance <https://access.redhat.com/documentation/en-us/red_hat_enterprise…