Lines Matching +full:docker +full:- +full:test +full:- +full:debug
7 everything from unit testing and exercising specific sub-systems all
9 tests you can run ``make check-help`` from either the source or build
12 Most (but not all) tests are also integrated as an automated test into
16 [./pyvenv/bin/]meson test --suite qemu:softfloat
20 An automated test is written with one of the test frameworks using its
21 generic test functions/classes. The test framework can run the tests and
24 An automated test has essentially three parts:
26 1. The test initialization of the parameters, where the expected parameters,
31 matches the expected result, the test has been successful; otherwise, it has
34 The rest of this document will cover the details for specific test
38 -------------------------
49 Different sub-types of "make check" tests will be explained below.
55 .. _unit-tests:
60 A unit test is responsible for exercising individual software components as a
64 structures. A test case of unit tests should be designed to uncover errors
68 In QEMU, unit tests can be invoked with ``make check-unit``. They are
72 If you are writing new code in QEMU, consider adding a unit test, especially
74 add a new unit test:
76 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
78 2. Write the test. Normally you would include the header file which exports
80 test. The test code should be organized with the glib testing framework.
81 Copying and modifying an existing test is usually a good idea.
83 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
85 dependencies to be linked with the test. For a simple test whose source
86 is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
90 'foo-test': [],
94 Since unit tests don't require environment variables, the simplest way to debug
95 a unit test failure is often directly invoking it or even running it under
103 make check-unit V=1
105 and copy the actual command line which executes the unit test, then run
111 QTest is a device emulation testing framework. It can be very useful to test
120 make check-qtest
122 Writing portable test cases
125 Care must be taken when writing portable test cases that can be built and run
133 stdin/stdout/stderr and null devices. For example if your test case uses
137 * If your test cases uses the blkdebug feature, use relative path to pass
140 * Use double quotes in your extra QEMU command line in your test cases
145 test cases opens a file to write some data and later wants to compare the
148 * If a certain test case can only run on POSIX or Linux hosts, use a proper
149 #ifdef in the codes. If the whole test suite cannot run on Windows, disable
152 .. _qapi-tests:
161 The input/output data is managed under the ``tests/qapi-schema`` directory.
162 Each test case includes four files that have a common base name:
164 * ``${casename}.json`` - the file contains the JSON input for feeding the
166 * ``${casename}.out`` - the file contains the expected stdout from the parser
167 * ``${casename}.err`` - the file contains the expected stderr from the parser
168 * ``${casename}.exit`` - the expected error code
170 Consider adding a new QAPI schema test when you are making a change on the QAPI
175 ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
177 2. Add the new test in ``tests/Makefile.include``. For example:
179 ``qapi-schema += foo.json``
181 check-block
184 ``make check-block`` runs a subset of the block layer iotests (the tests that
188 .. _qemu-iotests:
191 ------------
193 QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
194 framework widely used to test block layer related features. It is higher level
197 test files are named with numbers.
200 ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
209 # test with qcow2 format
210 ./check -qcow2
211 # or test a different protocol
212 ./check -nbd
214 It's also possible to list test numbers explicitly:
219 ./check -qcow2 001 030 153
221 Cache mode can be selected with the "-c" option, which may help reveal bugs
224 More options are supported by the ``./check`` script, run ``./check -h`` for
227 Writing a new test case
232 test cases, so it is possible that extending one of them may achieve the goal
239 output. They are given the same number in file names. E.g. Test script ``055``
247 There isn't a hard rule about how to write a test script, but a new test is
249 commonly used ways to create a test:
258 harder to debug.
270 Both Python and Bash frameworks in iotests provide helpers to manage test
271 images. They can be used to create and clean up images under the test
273 more convenient to use the pseudo block driver, ``null-co://``, as the test
274 image, which doesn't require image creation or cleaning up. Avoid system-wide
278 test failure. If using such devices are explicitly desired, consider adding
281 Debugging a test case
285 a failing test:
287 * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
293 ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
295 If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
298 * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
301 The final command line will be ``valgrind --log-file=$TEST_DIR/
302 <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
304 * ``-d`` (debug) just increases the logging verbosity, showing
307 * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
309 ``$TEST_DIR/qemu-machine-<random_string>``.
311 Test case groups
314 "Tests may belong to one or more test groups, which are defined in the form
315 of a comment in the test source file. By convention, test groups are listed
316 in the second line of the test file, after the "#!/..." line, like this:
325 Another way of defining groups is creating the tests/qemu-iotests/group.local
327 in upstream). This file may be used for defining some downstream test groups
334 # ci - tests to run on build
335 # down - our downstream tests, not for upstream
343 our-ugly-workaround-test down ci
347 - quick: Tests in this group should finish within a few seconds.
349 - auto: Tests in this group are used during "make check" and should be
351 (also non-x86), with every QEMU configuration (i.e. must not fail if
352 an optional feature is not compiled in - but reporting a "skip" is ok),
357 - disabled: Tests in this group are disabled and ignored by check.
359 .. _container-ref:
362 ---------------------
368 build and test QEMU in predefined and widely accessible Linux
369 environments. This makes it possible to expand the test coverage
371 was originally written for Docker although we also support Podman as
373 names and scripts are prefixed with "docker" the system will
377 for testing TCG. See :ref:`checktcg-ref` for more details.
379 Docker Prerequisites
382 Install "docker" with the system package manager and start the Docker service
384 Docker commands. Typically it means setting up passwordless ``sudo docker``
389 $ sudo yum install docker
390 $ # or `apt-get install docker` for Ubuntu, etc.
391 $ sudo systemctl start docker
392 $ sudo docker ps
397 "docker" group and making the docker daemon socket file (by default
398 ``/var/run/docker.sock``) accessible to the group:
402 $ sudo groupadd docker
403 $ sudo usermod $USER -a -G docker
404 $ sudo chown :docker /var/run/docker.sock
407 exploit the whole host with Docker bind mounting or other privileged
425 From source tree, type ``make docker-help`` to see the help. Testing
432 make docker-test-build@debian
435 is downloaded and initialized automatically), in which the ``test-build`` job
442 ``registry.gitlab.com/qemu-project/qemu`` which will automatically be
443 used to pull in pre-built layers. This avoids unnecessary strain on
450 make docker-image-debian-arm64-cross NOCACHE=1
456 in ``tests/docker/dockerfiles/``, called ``debian.docker``. ``make docker-help``
459 A ``.pre`` script can be added beside the ``.docker`` file, which will be
462 for example, to make qemu-user powered cross build containers work.
465 a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
470 using the ``lcitool`` program provided by the ``libvirt-ci`` project:
472 https://gitlab.com/libvirt/libvirt-ci
474 ``libvirt-ci`` contains an ``lcitool`` program as well as a list of
477 pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
483 Adding new build pre-requisites
487 pre-requisite to QEMU, the prerequisites should to be added to
491 In the simple case where the pre-requisite is already known to ``libvirt-ci``
494 * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
496 * Run ``make lcitool-refresh`` to re-generate all relevant build environment
499 It may be that ``libvirt-ci`` does not know about the new pre-requisite.
501 first to contribute the mapping to the ``libvirt-ci`` project:
503 * Fork the ``libvirt-ci`` project on gitlab
507 many OS distros as practical. Run ``python -m pytest --regenerate-output``
510 * Commit the ``mappings.yml`` change together with the regenerated test
511 files, and submit a merge request to the ``libvirt-ci`` project.
512 Please note in the description that this is a new build pre-requisite
517 all OS distributions supported by ``libvirt-ci``.
520 the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
522 run ``make lcitool-refresh``.
526 obvious breakages when adding the new pre-requisite. Please see
530 For enterprise distros that default to old, end-of-life versions of the
533 Modifying this file should not be necessary unless the new pre-requisite
540 In some cases ``libvirt-ci`` will not know about the OS distro that is
544 * Send a mail to qemu-devel, copying people listed in the
545 MAINTAINERS file for ``Build and test automation``.
550 * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues
551 pointing to the qemu-devel mail thread in the archives.
554 to avoid duplication, as well as to get feedback from libvirt-ci
559 * Fork the ``libvirt-ci`` project on gitlab
563 uses a package format not currently known. The ``libvirt-ci``
568 as practical. Run ``python -m pytest --regenerate-output`` and
571 * Commit the changes to ``lcitool/facts`` and the regenerated test
572 files, and submit a merge request to the ``libvirt-ci`` project.
573 Please note in the description that this is a new build pre-requisite
578 all OS distributions supported by ``libvirt-ci``.
581 the ``libvirt-ci`` submodule to point to a commit that contains
588 Different tests are added to cover various configurations to build and test
589 QEMU. Docker tests are the executables under ``tests/docker`` named
590 ``test-*``. They are typically shell scripts and are built on top of a shell
591 library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
594 The full list of tests is printed in the ``make docker-help`` help.
596 Debugging a Docker test failure
599 When CI tasks, maintainers or yourself report a Docker test failure, follow the
600 below steps to debug it:
603 ``make docker-test-mingw@fedora-win64-cross J=8``.
605 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
607 build QEMU and run tests from there, or press Ctrl-D to let the Docker
609 4. If you press Ctrl-D, the same building and testing procedure will begin, and
611 the prompt for debug.
616 Various options can be used to affect how Docker tests are done. The full
617 list is in the ``make docker`` help text. The frequently used ones are:
622 similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
624 * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
628 ----------------
637 Thread Sanitizer in Docker
639 TSan is currently supported in the ubuntu2204 docker.
641 The test-tsan test will build using TSan and then run make check.
645 make docker-test-tsan@ubuntu2204
647 TSan warnings under docker are placed in files located at build/tsan/.
649 We recommend using DEBUG=1 to allow launching the test from inside the docker,
655 It is possible to build and test with TSan, with a few additional steps.
656 These steps are normally done automatically in the docker.
668 $ git clone --depth=1 --branch=2.81.0 https://github.com/GNOME/glib.git
670 $ CFLAGS="-O2 -g -fsanitize=thread" meson build
671 $ ninja -C build
677 ../configure --enable-tsan \
678 --disable-werror --extra-cflags="-O0"
687 $ ldd build/qemu-x86_64 | grep glib
688 $ qemu-system-x86_64 ...
706 This allows for running the test and then checking the warnings afterwards.
709 .. _tsan-suppressions:
721 tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
727 tests/tsan/ignore.tsan - Has TSan warnings we wish to disable
728 at compile time for test or debug.
731 "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/ignore.tsan"
746 https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
752 https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
754 docker-binfmt-image-debian-% targets
755 ------------------------------------
760 simplify setting up a rootfs by using docker to contain the foreign
766 You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU
770 potential complications when copying into the docker image. Modern
772 executable on setup and avoids the need to find and re-open in the
773 chroot environment. This is triggered with the ``--persistent`` flag.
780 make docker-binfmt-image-debian-sid-hppa \
782 DEB_URL=http://ftp.ports.debian.org/debian-ports/ \
783 DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
784 EXECUTABLE=(pwd)/qemu-hppa V=1
787 ``debian-bootstrap.pre`` which is called to do the initial debootstrap
790 ``qemu/debian-sid-hppa``.
793 ----------
795 This test suite contains scripts that bootstrap various guest images that have
797 help which is displayed with ``make vm-help``.
802 Run ``make vm-help`` to list available make targets. Invoke a specific make
803 command to run build test in an image. For example, ``make vm-build-freebsd``
806 not needed. The command will then generate the test image in ``./tests/vm/``
809 Note: images created by the scripts accept a well-known RSA key pair for SSH
817 By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If
818 there isn't one, or if it is older than 2.10, the test won't work. In this case,
819 provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
821 Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable.
826 The ``-j$X`` option in the make command line is not propagated into the VM,
832 Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
847 $ ./netbsd --build-image --image /var/tmp/netbsd.img
851 # --debug is added)
852 $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
855 $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
858 $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
876 - Root password set to ``BaseVM.ROOT_PASS``
877 - User ``BaseVM.GUEST_USER`` is created, and password set to
879 - SSH service is enabled and started on boot,
882 - DHCP client service is enabled and started on boot, so that it can
883 automatically configure the virtio-net-pci NIC and communicate with QEMU
885 - Necessary packages are installed to untar the source tarball and build
889 untars a raw virtio-blk block device, which is the tarball data blob of the
894 --------------------
901 tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
903 Alternatively, some command different from ``qemu-img info`` can be tested, by
904 changing the ``-c`` option.
907 -----------------------------
909 A functional test focuses on the functional requirement of the software,
918 make check-functional
920 See :ref:`checkfunctional-ref` for more details.
922 .. _checktcg-ref:
924 Testing with "make check-tcg"
925 -----------------------------
927 The check-tcg tests are intended for simple smoke tests of both
928 linux-user and softmmu TCG functionality. However to build test
933 apt install gcc-aarch64-linux-gnu
940 $(configure) --cross-cc-aarch64=aarch64-cc
942 There is also a ``--cross-cc-cflags-ARCH`` flag in case additional
949 additional containers defined that have a minimal cross-build
950 environment that is only suitable for building test cases. Sometimes
952 for test cases that aren't yet in the LTS distros we support for QEMU
955 See :ref:`container-ref` for more details.
962 make build-tcg-tests-$TARGET
966 make run-tcg-tests-$TARGET
969 invoke QEMU for the test which is useful for debugging tests.
974 Tests can also be run directly from the test build directory. If you
975 run ``make help`` from the test build directory you will get a list of
979 make run-plugin-test-mmap-with-libinline.so
981 will run the mmap test with the ``libinline.so`` TCG plugin. The
982 gdbstub tests also re-use the test binaries but while exercising gdb.
984 TCG test dependencies
988 either totally bare with minimal gcc lib support (for system-mode tests)
989 or just glibc (for linux-user tests). This is because getting a cross
993 ---------------
995 There are a number of out-of-tree test suites that are used for more
1004 as reporting test results via a special device::
1006 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
1008 Linux Test Project
1013 exercise as many corner cases as possible. It is a useful test suite
1014 to run to exercise QEMU's linux-user code::
1016 https://linux-test-project.github.io/
1019 ----------------
1023 ``--enable-gcov`` option and build. Then run the tests as usual.
1025 If you want to gather coverage information on a single test the ``make
1026 clean-gcda`` target can be used to delete any existing coverage
1027 information before running a single test.
1030 coverage-html`` which will create
1031 ``meson-logs/coveragereport/index.html``.
1038 -----------
1040 A flaky test is defined as a test that exhibits both a passing and a failing
1042 intermittent/flaky test are async wait, concurrency, and test order dependency
1050 ----------