devel/testing/main.rst

7 everything from unit testing and exercising specific sub-systems all
9 tests you can run ``make check-help`` from either the source or build
16   [./pyvenv/bin/]meson test --suite qemu:softfloat
38 -------------------------
49 Different sub-types of "make check" tests will be explained below.
55 .. _unit-tests:
68 In QEMU, unit tests can be invoked with ``make check-unit``. They are
76 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
86    is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
90        'foo-test': [],
103   make check-unit V=1
120    make check-qtest
152 .. _qapi-tests:
161 The input/output data is managed under the ``tests/qapi-schema`` directory.
164   * ``${casename}.json`` - the file contains the JSON input for feeding the
166   * ``${casename}.out`` - the file contains the expected stdout from the parser
167   * ``${casename}.err`` - the file contains the expected stderr from the parser
168   * ``${casename}.exit`` - the expected error code
175   ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
179   ``qapi-schema += foo.json``
181 check-block
184 ``make check-block`` runs a subset of the block layer iotests (the tests that
188 .. _qemu-iotests:
191 ------------
193 QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
200 ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
210   ./check -qcow2
212   ./check -nbd
219   ./check -qcow2 001 030 153
221 Cache mode can be selected with the "-c" option, which may help reveal bugs
224 More options are supported by the ``./check`` script, run ``./check -h`` for
273 more convenient to use the pseudo block driver, ``null-co://``, as the test
274 image, which doesn't require image creation or cleaning up. Avoid system-wide
287 * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
293   ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
295   If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
298 * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
301   The final command line will be ``valgrind --log-file=$TEST_DIR/
302   <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
304 * ``-d`` (debug) just increases the logging verbosity, showing
307 * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
309   ``$TEST_DIR/qemu-machine-<random_string>``.
325 Another way of defining groups is creating the tests/qemu-iotests/group.local
334   # ci - tests to run on build
335   # down - our downstream tests, not for upstream
343   our-ugly-workaround-test down ci
347 - quick: Tests in this group should finish within a few seconds.
349 - auto: Tests in this group are used during "make check" and should be
351   (also non-x86), with every QEMU configuration (i.e. must not fail if
352   an optional feature is not compiled in - but reporting a "skip" is ok),
357 - disabled: Tests in this group are disabled and ignored by check.
359 .. _container-ref:
362 ---------------------
377 for testing TCG. See :ref:`checktcg-ref` for more details.
390   $ # or `apt-get install docker` for Ubuntu, etc.
403   $ sudo usermod $USER -a -G docker
425 From source tree, type ``make docker-help`` to see the help. Testing
432   make docker-test-build@debian
435 is downloaded and initialized automatically), in which the ``test-build`` job
442 ``registry.gitlab.com/qemu-project/qemu`` which will automatically be
443 used to pull in pre-built layers. This avoids unnecessary strain on
450    make docker-image-debian-arm64-cross NOCACHE=1
456 in ``tests/docker/dockerfiles/``, called ``debian.docker``. ``make docker-help``
462 for example, to make qemu-user powered cross build containers work.
470 using the ``lcitool`` program provided by the ``libvirt-ci`` project:
472   https://gitlab.com/libvirt/libvirt-ci
474 ``libvirt-ci`` contains an ``lcitool`` program as well as a list of
477 pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
483 Adding new build pre-requisites
487 pre-requisite to QEMU, the prerequisites should to be added to
491 In the simple case where the pre-requisite is already known to ``libvirt-ci``
494  * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
496  * Run ``make lcitool-refresh`` to re-generate all relevant build environment
499 It may be that ``libvirt-ci`` does not know about the new pre-requisite.
501 first to contribute the mapping to the ``libvirt-ci`` project:
503  * Fork the ``libvirt-ci`` project on gitlab
507    many OS distros as practical.  Run ``python -m pytest --regenerate-output``
511    files, and submit a merge request to the ``libvirt-ci`` project.
512    Please note in the description that this is a new build pre-requisite
517    all OS distributions supported by ``libvirt-ci``.
520    the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
522    run ``make lcitool-refresh``.
526    obvious breakages when adding the new pre-requisite. Please see
530 For enterprise distros that default to old, end-of-life versions of the
533 Modifying this file should not be necessary unless the new pre-requisite
540 In some cases ``libvirt-ci`` will not know about the OS distro that is
544  * Send a mail to qemu-devel, copying people listed in the
550  * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues
551    pointing to the qemu-devel mail thread in the archives.
554    to avoid duplication, as well as to get feedback from libvirt-ci
559  * Fork the ``libvirt-ci`` project on gitlab
563    uses a package format not currently known. The ``libvirt-ci``
568    as practical.  Run ``python -m pytest --regenerate-output`` and
572    files, and submit a merge request to the ``libvirt-ci`` project.
573    Please note in the description that this is a new build pre-requisite
578    all OS distributions supported by ``libvirt-ci``.
581    the ``libvirt-ci`` submodule to point to a commit that contains
590 ``test-*``. They are typically shell scripts and are built on top of a shell
594 The full list of tests is printed in the ``make docker-help`` help.
603    ``make docker-test-mingw@fedora-win64-cross J=8``.
607    build QEMU and run tests from there, or press Ctrl-D to let the Docker
609 4. If you press Ctrl-D, the same building and testing procedure will begin, and
622   similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
628 ----------------
641 The test-tsan test will build using TSan and then run make check.
645   make docker-test-tsan@ubuntu2204
668    $ git clone --depth=1 --branch=2.81.0 https://github.com/GNOME/glib.git
670    $ CFLAGS="-O2 -g -fsanitize=thread" meson build
671    $ ninja -C build
677   ../configure --enable-tsan \
678                --disable-werror --extra-cflags="-O0"
687    $ ldd build/qemu-x86_64 | grep glib
688    $ qemu-system-x86_64 ...
709 .. _tsan-suppressions:
721 tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
727 tests/tsan/ignore.tsan - Has TSan warnings we wish to disable
731 "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/ignore.tsan"
746 https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
752 https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
754 docker-binfmt-image-debian-% targets
755 ------------------------------------
766 You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU
772 executable on setup and avoids the need to find and re-open in the
773 chroot environment. This is triggered with the ``--persistent`` flag.
780   make docker-binfmt-image-debian-sid-hppa \
782     DEB_URL=http://ftp.ports.debian.org/debian-ports/ \
783     DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
784     EXECUTABLE=(pwd)/qemu-hppa V=1
787 ``debian-bootstrap.pre`` which is called to do the initial debootstrap
790 ``qemu/debian-sid-hppa``.
793 ----------
797 help which is displayed with ``make vm-help``.
802 Run ``make vm-help`` to list available make targets. Invoke a specific make
803 command to run build test in an image. For example, ``make vm-build-freebsd``
809 Note: images created by the scripts accept a well-known RSA key pair for SSH
817 By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If
819 provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
821 Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable.
826 The ``-j$X`` option in the make command line is not propagated into the VM,
847     $ ./netbsd --build-image --image /var/tmp/netbsd.img
851     # --debug is added)
852     $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
855     $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
858     $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
876   - Root password set to ``BaseVM.ROOT_PASS``
877   - User ``BaseVM.GUEST_USER`` is created, and password set to
879   - SSH service is enabled and started on boot,
880     ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
882   - DHCP client service is enabled and started on boot, so that it can
883     automatically configure the virtio-net-pci NIC and communicate with QEMU
885   - Necessary packages are installed to untar the source tarball and build
889   untars a raw virtio-blk block device, which is the tarball data blob of the
894 --------------------
901   tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
903 Alternatively, some command different from ``qemu-img info`` can be tested, by
904 changing the ``-c`` option.
907 -----------------------------
918   make check-functional
920 See :ref:`checkfunctional-ref` for more details.
922 .. _checktcg-ref:
924 Testing with "make check-tcg"
925 -----------------------------
927 The check-tcg tests are intended for simple smoke tests of both
928 linux-user and softmmu TCG functionality. However to build test
933   apt install gcc-aarch64-linux-gnu
940   $(configure) --cross-cc-aarch64=aarch64-cc
942 There is also a ``--cross-cc-cflags-ARCH`` flag in case additional
949 additional containers defined that have a minimal cross-build
955 See :ref:`container-ref` for more details.
962   make build-tcg-tests-$TARGET
966   make run-tcg-tests-$TARGET
979   make run-plugin-test-mmap-with-libinline.so
982 gdbstub tests also re-use the test binaries but while exercising gdb.
988 either totally bare with minimal gcc lib support (for system-mode tests)
989 or just glibc (for linux-user tests). This is because getting a cross
993 ---------------
995 There are a number of out-of-tree test suites that are used for more
1006   https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
1014 to run to exercise QEMU's linux-user code::
1016   https://linux-test-project.github.io/
1019 ----------------
1023 ``--enable-gcov`` option and build. Then run the tests as usual.
1026 clean-gcda`` target can be used to delete any existing coverage
1030 coverage-html`` which will create
1031 ``meson-logs/coveragereport/index.html``.
1038 -----------
1050 ----------