devel/testing/main.rst

12 Most (but not all) tests are also integrated as an automated test into
16   [./pyvenv/bin/]meson test --suite qemu:softfloat
20 An automated test is written with one of the test frameworks using its
21 generic test functions/classes. The test framework can run the tests and
24 An automated test has essentially three parts:
26 1. The test initialization of the parameters, where the expected parameters,
31    matches the expected result, the test has been successful; otherwise, it has
34 The rest of this document will cover the details for specific test
60 A unit test is responsible for exercising individual software components as a
64 structures. A test case of unit tests should be designed to uncover errors
72 If you are writing new code in QEMU, consider adding a unit test, especially
74 add a new unit test:
76 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
78 2. Write the test. Normally you would include the header file which exports
80    test. The test code should be organized with the glib testing framework.
81    Copying and modifying an existing test is usually a good idea.
83 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
85    dependencies to be linked with the test.  For a simple test whose source
86    is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
90        'foo-test': [],
95 a unit test failure is often directly invoking it or even running it under
105 and copy the actual command line which executes the unit test, then run
111 QTest is a device emulation testing framework.  It can be very useful to test
122 Writing portable test cases
125 Care must be taken when writing portable test cases that can be built and run
133   stdin/stdout/stderr and null devices. For example if your test case uses
137 * If your test cases uses the blkdebug feature, use relative path to pass
140 * Use double quotes in your extra QEMU command line in your test cases
145   test cases opens a file to write some data and later wants to compare the
148 * If a certain test case can only run on POSIX or Linux hosts, use a proper
149   #ifdef in the codes. If the whole test suite cannot run on Windows, disable
162 Each test case includes four files that have a common base name:
170 Consider adding a new QAPI schema test when you are making a change on the QAPI
177 2. Add the new test in ``tests/Makefile.include``. For example:
194 framework widely used to test block layer related features. It is higher level
197 test files are named with numbers.
209   # test with qcow2 format
211   # or test a different protocol
214 It's also possible to list test numbers explicitly:
227 Writing a new test case
232 test cases, so it is possible that extending one of them may achieve the goal
239 output. They are given the same number in file names. E.g. Test script ``055``
247 There isn't a hard rule about how to write a test script, but a new test is
249 commonly used ways to create a test:
270 Both Python and Bash frameworks in iotests provide helpers to manage test
271 images. They can be used to create and clean up images under the test
273 more convenient to use the pseudo block driver, ``null-co://``, as the test
278 test failure.  If using such devices are explicitly desired, consider adding
281 Debugging a test case
285 a failing test:
307 * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
311 Test case groups
314 "Tests may belong to one or more test groups, which are defined in the form
315 of a comment in the test source file. By convention, test groups are listed
316 in the second line of the test file, after the "#!/..." line, like this:
327 in upstream). This file may be used for defining some downstream test groups
343   our-ugly-workaround-test down ci
368 build and test QEMU in predefined and widely accessible Linux
369 environments. This makes it possible to expand the test coverage
432   make docker-test-build@debian
435 is downloaded and initialized automatically), in which the ``test-build`` job
510  * Commit the ``mappings.yml`` change together with the regenerated test
545    MAINTAINERS file for ``Build and test automation``.
571  * Commit the changes to ``lcitool/facts`` and the regenerated test
588 Different tests are added to cover various configurations to build and test
590 ``test-*``. They are typically shell scripts and are built on top of a shell
596 Debugging a Docker test failure
599 When CI tasks, maintainers or yourself report a Docker test failure, follow the
603    ``make docker-test-mingw@fedora-win64-cross J=8``.
624 * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
641 The test-tsan test will build using TSan and then run make check.
645   make docker-test-tsan@ubuntu2204
649 We recommend using DEBUG=1 to allow launching the test from inside the docker,
655 It is possible to build and test with TSan, with a few additional steps.
706 This allows for running the test and then checking the warnings afterwards.
728 at compile time for test or debug.
746 https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
795 This test suite contains scripts that bootstrap various guest images that have
803 command to run build test in an image. For example, ``make vm-build-freebsd``
806 not needed. The command will then generate the test image in ``./tests/vm/``
818 there isn't one, or if it is older than 2.10, the test won't work. In this case,
901   tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
909 A functional test focuses on the functional requirement of the software,
928 linux-user and softmmu TCG functionality. However to build test
950 environment that is only suitable for building test cases. Sometimes
952 for test cases that aren't yet in the LTS distros we support for QEMU
969 invoke QEMU for the test which is useful for debugging tests.
974 Tests can also be run directly from the test build directory. If you
975 run ``make help`` from the test build directory you will get a list of
979   make run-plugin-test-mmap-with-libinline.so
981 will run the mmap test with the ``libinline.so`` TCG plugin. The
982 gdbstub tests also re-use the test binaries but while exercising gdb.
984 TCG test dependencies
995 There are a number of out-of-tree test suites that are used for more
1004 as reporting test results via a special device::
1008 Linux Test Project
1013 exercise as many corner cases as possible. It is a useful test suite
1016   https://linux-test-project.github.io/
1025 If you want to gather coverage information on a single test the ``make
1027 information before running a single test.
1040 A flaky test is defined as a test that exhibits both a passing and a failing
1042 intermittent/flaky test are async wait, concurrency, and test order dependency