devel/testing/main.rst

8 the way to full blown functional tests. To get an overview of the
9 tests you can run ``make check-help`` from either the source or build
12 Most (but not all) tests are also integrated as an automated test into
18 will run just the softfloat tests.
21 generic test functions/classes. The test framework can run the tests and
40 The "make check" testing family includes most of the C based tests in QEMU.
42 The usual way to run these tests is:
48 which includes QAPI schema tests, unit tests, QTests and some iotests.
49 Different sub-types of "make check" tests will be explained below.
51 Before running tests, it is best to build QEMU programs first. Some tests
55 .. _unit-tests:
57 Unit tests
64 structures. A test case of unit tests should be designed to uncover errors
68 In QEMU, unit tests can be invoked with ``make check-unit``. They are
69 simple C tests that typically link to individual QEMU object files and
76 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
83 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
84    dictionary called ``tests``.  The values are any additional sources and
86    is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
94 Since unit tests don't require environment variables, the simplest way to debug
124 Both unit tests and qtests can run on POSIX hosts as well as Windows hosts.
152 .. _qapi-tests:
154 QAPI schema tests
157 The QAPI schema tests validate the QAPI parser used by QMP, by feeding
161 The input/output data is managed under the ``tests/qapi-schema`` directory.
175   ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
177 2. Add the new test in ``tests/Makefile.include``. For example:
184 ``make check-block`` runs a subset of the block layer iotests (the tests that
193 QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
195 than "make check" tests and 99% of the code is written in bash or Python
200 ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
203 By default, "raw" format and "file" protocol is used; all tests will be
230 Consider writing a tests case when you are making any changes to the block
234 reliable way to find a related one out of hundreds of tests.  One approach is
325 Another way of defining groups is creating the tests/qemu-iotests/group.local
328 or for temporarily disabling tests, like this:
334   # ci - tests to run on build
335   # down - our downstream tests, not for upstream
361 Container based tests
376 The container images are also used to augment the generation of tests
456 in ``tests/docker/dockerfiles/``, called ``debian.docker``. ``make docker-help``
465 a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
477 pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
488 ``tests/lcitool/projects/qemu.yml`` in order to make the dependency
494  * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
520    the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
532 recent versions.  These can be found in ``tests/lcitool/mappings.yml``.
588 Different tests are added to cover various configurations to build and test
589 QEMU.  Docker tests are the executables under ``tests/docker`` named
591 library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
594 The full list of tests is printed in the ``make docker-help`` help.
607    build QEMU and run tests from there, or press Ctrl-D to let the Docker
616 Various options can be used to affect how Docker tests are done. The full
701   export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \
721 tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
727 tests/tsan/ignore.tsan - Has TSan warnings we wish to disable
731 "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/ignore.tsan"
806 not needed. The command will then generate the test image in ``./tests/vm/``
840 For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
844     $ cd $QEMU_SRC/tests/vm
880     ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
901   tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
906 Functional tests using Python
913 The ``tests/functional`` directory hosts functional tests written in
914 Python. You can run the functional tests simply by executing:
927 The check-tcg tests are intended for simple smoke tests of both
948 use the same container to build tests. However there are a number of
957 Running subset of tests
960 You can build the tests for one architecture::
962   make build-tcg-tests-$TARGET
966   make run-tcg-tests-$TARGET
969 invoke QEMU for the test which is useful for debugging tests.
971 Running individual tests
976 all the tests that can be run. Please note that same binaries are used
977 in multiple tests, for example::
982 gdbstub tests also re-use the test binaries but while exercising gdb.
987 The TCG tests are deliberately very light on dependencies and are
988 either totally bare with minimal gcc lib support (for system-mode tests)
989 or just glibc (for linux-user tests). This is because getting a cross
1001 The KVM unit tests are designed to run as a Guest OS under KVM but
1006   https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
1023 ``--enable-gcov`` option and build. Then run the tests as usual.
1037 Flaky tests
1045 In QEMU, tests that are identified to be flaky are normally disabled by
1047 the tests to enable them.
1057 .. [4] Luo, Qingzhou, et al. An empirical analysis of flaky tests.