devel/testing/main.rst

8 the way to full blown functional tests. To get an overview of the
20 An automated test is written with one of the test frameworks using its
26 1. The test initialization of the parameters, where the expected parameters,
30    result set during the initialization of the parameters. If the result
34 The rest of this document will cover the details for specific test
40 The "make check" testing family includes most of the C based tests in QEMU.
49 Different sub-types of "make check" tests will be explained below.
62 within the boundaries of a component. The verification effort is in the
64 structures. A test case of unit tests should be designed to uncover errors
112 device models; it could also control certain aspects of QEMU (such as virtual
141   instead of single quotes, as Windows does not drop single quotes when
147   part of the mode string to fopen(), or O_BINARY flag for the open() call.
184 ``make check-block`` runs a subset of the block layer iotests (the tests that
195 than "make check" tests and 99% of the code is written in bash or Python
232 test cases, so it is possible that extending one of them may achieve the goal
234 reliable way to find a related one out of hundreds of tests.  One approach is
237 Usually an iotest case consists of two files. One is an executable that
248 usually a (copy and) modification of an existing case.  There are a few
251 * A Bash script. It will make use of several environmental variables related
252   to the testing procedure, and could source a group of ``common.*`` libraries
255 * A Python unittest script. Import ``iotests`` and create a subclass of
256   ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
263   execution. This is a combination of 1 and 2.
296   regardless of whether it is set or not.
308   instead of saving it into a log file in
315 of a comment in the test source file. By convention, test groups are listed
316 in the second line of the test file, after the "#!/..." line, like this:
325 Another way of defining groups is creating the tests/qemu-iotests/group.local
337   # Format of each line is:
353   work at least with the qcow2 file format, work with all kind of host
372 an alternative container runtime. Although many of the target
376 The container images are also used to augment the generation of tests
406 Note that any one of above configurations makes it possible for the user to
464 Most of the existing Dockerfiles were written by hand, simply by creating a
466 This has led to an inconsistent set of packages being present across the
474 ``libvirt-ci`` contains an ``lcitool`` program as well as a list of
475 mappings to distribution package names for a wide variety of third
476 party projects.  ``lcitool`` applies the mappings to a list of build
478 list of native packages to install on each distribution, and uses them
530 For enterprise distros that default to old, end-of-life versions of the
531 Python runtime, QEMU uses a separate set of mappings that work with more
548    cost/benefit tradeoff of adding new OS distros needs to be considered.
590 ``test-*``. They are typically shell scripts and are built on top of a shell
594 The full list of tests is printed in the ``make docker-help`` help.
621 * ``J=$N``: the number of parallel tasks in make commands in the container,
650 and to allow review of the warnings generated by TSan.
659 One particularity of sanitizers is that all the code, including shared objects
661 In the case of TSan, any synchronization primitive from glib (GMutex for
664 To build a tsan version of glib:
690 The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment
740 of the annotations themselves.  Annotations can be used to suppress
742 relationships between accesses of data.
750 The full set of annotations can be found here:
758 ``binfmt_misc`` to bootstrap a number of Debian's distros including
778 For example to setup the HPPA ports builds of Debian::
788 of the rootfs before it is copied into the container. The second stage
789 is run as part of the build. The final image will be tagged as
811 concerned about attackers taking control of the guest and potentially
865 Most importantly, create a subclass of BaseVM and implement ``build_image()``
870   predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
881     file of both root and the normal user
889   untars a raw virtio-blk block device, which is the tarball data blob of the
909 A functional test focuses on the functional requirement of the software,
927 The check-tcg tests are intended for simple smoke tests of both
936 Sometimes compilers have slightly odd names so the availability of
948 use the same container to build tests. However there are a number of
957 Running subset of tests
968 Adding ``V=1`` to the invocation will show the details of how to
975 run ``make help`` from the test build directory you will get a list of
995 There are a number of out-of-tree test suites that are used for more
996 extensive testing of processor features.
1011 The LTP is focused on exercising the syscall interface of a Linux
1057 .. [4] Luo, Qingzhou, et al. An empirical analysis of flaky tests.
1058        Proceedings of the 22nd ACM SIGSOFT International Symposium on
1059        Foundations of Software Engineering. 2014.