Lines Matching full:the
8 the way to full blown functional tests. To get an overview of the
9 tests you can run ``make check-help`` from either the source or build
13 the meson build system so can be run directly from the build tree,
18 will run just the softfloat tests.
20 An automated test is written with one of the test frameworks using its
21 generic test functions/classes. The test framework can run the tests and
26 1. The test initialization of the parameters, where the expected parameters,
28 2. The call to the code that should be tested;
29 3. An assertion, comparing the result from the previous call with the expected
30 result set during the initialization of the parameters. If the result
31 matches the expected result, the test has been successful; otherwise, it has
34 The rest of this document will cover the details for specific test
40 The "make check" testing family includes most of the C based tests in QEMU.
42 The usual way to run these tests is:
52 expect the executables to exist and will fail with obscure messages if they
62 within the boundaries of a component. The verification effort is in the
63 smallest software unit and focuses on the internal processing logic and data
78 2. Write the test. Normally you would include the header file which exports
79 the module API, then verify the interface behaves as expected from your
80 test. The test code should be organized with the glib testing framework.
83 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
84 dictionary called ``tests``. The values are any additional sources and
85 dependencies to be linked with the test. For a simple test whose source
94 Since unit tests don't require environment variables, the simplest way to debug
105 and copy the actual command line which executes the unit test, then run
106 it from the command line.
126 successfully on various hosts. The following list shows some best practices:
137 * If your test cases uses the blkdebug feature, use relative path to pass
138 the config and image file paths in the command line as Windows absolute
139 path contains the delimiter ":" which will confuse the blkdebug parser.
142 passing the command line to QEMU.
144 implementation treats text files and binary files the same. So if your
145 test cases opens a file to write some data and later wants to compare the
146 written data with the original one, be sure to pass the letter 'b' as
147 part of the mode string to fopen(), or O_BINARY flag for the open() call.
149 #ifdef in the codes. If the whole test suite cannot run on Windows, disable
150 the build in the meson.build file.
157 The QAPI schema tests validate the QAPI parser used by QMP, by feeding
158 predefined input to the parser and comparing the result with the reference
161 The input/output data is managed under the ``tests/qapi-schema`` directory.
164 * ``${casename}.json`` - the file contains the JSON input for feeding the
166 * ``${casename}.out`` - the file contains the expected stdout from the parser
167 * ``${casename}.err`` - the file contains the expected stderr from the parser
168 * ``${casename}.exit`` - the expected error code
170 Consider adding a new QAPI schema test when you are making a change on the QAPI
171 parser (either fixing a bug or extending/modifying the syntax). To do this:
173 1. Add four files for the new case as explained above. For example:
177 2. Add the new test in ``tests/Makefile.include``. For example:
184 ``make check-block`` runs a subset of the block layer iotests (the tests that
185 are in the "auto" group).
186 See the "QEMU iotests" section below for more information.
193 QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
195 than "make check" tests and 99% of the code is written in bash or Python
196 scripts. The testing success criteria is golden output comparison, and the
199 To run iotests, make sure QEMU is built successfully, then switch to the
200 ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
204 executed, except the unsupported ones. You can override the format and protocol
221 Cache mode can be selected with the "-c" option, which may help reveal bugs
224 More options are supported by the ``./check`` script, run ``./check -h`` for
230 Consider writing a tests case when you are making any changes to the block
231 layer. An iotest case is usually the choice for that. There are already many
232 test cases, so it is possible that extending one of them may achieve the goal
233 and save the boilerplate to create one. (Unfortunately, there isn't a 100%
238 produces output to stdout and stderr, the other is the expected reference
239 output. They are given the same number in file names. E.g. Test script ``055``
244 image formats, more than one ``.out`` files are created ending with the
252 to the testing procedure, and could source a group of ``common.*`` libraries
256 ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
257 this approach is that the output is too scarce, and the script is considered
262 from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
265 Pick the language per your preference since both Bash and Python have
271 images. They can be used to create and clean up images under the test
273 more convenient to use the pseudo block driver, ``null-co://``, as the test
277 another application on the host may have locked the file, possibly leading to a
284 The following options to the ``check`` script can be useful when debugging
288 connection from a gdb client. The options given to ``gdbserver`` (e.g. the
289 address on which to listen for connections) are taken from the ``$GDB_OPTIONS``
293 ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
295 If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
299 warnings, it will print and save the log in
301 The final command line will be ``valgrind --log-file=$TEST_DIR/
304 * ``-d`` (debug) just increases the logging verbosity, showing
305 for example the QMP commands and answers.
307 * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
314 "Tests may belong to one or more test groups, which are defined in the form
315 of a comment in the test source file. By convention, test groups are listed
316 in the second line of the test file, after the "#!/..." line, like this:
325 Another way of defining groups is creating the tests/qemu-iotests/group.local
345 Note that the following group names have a special meaning:
353 work at least with the qcow2 file format, work with all kind of host
367 The container testing framework in QEMU utilizes public images to
369 environments. This makes it possible to expand the test coverage
370 across distros, toolchain flavors and library versions. The support
372 an alternative container runtime. Although many of the target
373 names and scripts are prefixed with "docker" the system will
376 The container images are also used to augment the generation of tests
382 Install "docker" with the system package manager and start the Docker service
383 on your development machine, then make sure you have the privilege to run
394 The last command should print an empty table, to verify the system is ready.
396 An alternative method to set up permissions is by adding the current user to
397 "docker" group and making the docker daemon socket file (by default
398 ``/var/run/docker.sock``) accessible to the group:
406 Note that any one of above configurations makes it possible for the user to
407 exploit the whole host with Docker bind mounting or other privileged
413 Install "podman" with the system package manager.
420 The last command should print an empty table, to verify the system is ready.
425 From source tree, type ``make docker-help`` to see the help. Testing
427 ``make`` are done in the container, with parameters defined by the
434 This will create a container instance using the ``debian`` image (the image
435 is downloaded and initialized automatically), in which the ``test-build`` job
441 The QEMU project has a container registry hosted by GitLab at
444 the distro archives created by multiple developers running the same
446 locally by using the ``NOCACHE`` build option:
455 Along with many other images, the ``debian`` image is defined in a Dockerfile
457 command will list all the available images.
459 A ``.pre`` script can be added beside the ``.docker`` file, which will be
460 executed before building the image under the build context directory. This is
464 Most of the existing Dockerfiles were written by hand, simply by creating a
465 a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
466 This has led to an inconsistent set of packages being present across the
469 Thus going forward, QEMU is aiming to automatically generate the Dockerfiles
470 using the ``lcitool`` program provided by the ``libvirt-ci`` project:
476 party projects. ``lcitool`` applies the mappings to a list of build
477 pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
487 pre-requisite to QEMU, the prerequisites should to be added to
488 ``tests/lcitool/projects/qemu.yml`` in order to make the dependency
489 available in the CI build environments.
491 In the simple case where the pre-requisite is already known to ``libvirt-ci``
492 the following steps are needed:
494 * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
499 It may be that ``libvirt-ci`` does not know about the new pre-requisite.
500 If that is the case, some extra preparation steps will be required
501 first to contribute the mapping to the ``libvirt-ci`` project:
503 * Fork the ``libvirt-ci`` project on gitlab
505 * Add an entry for the new build prerequisite to
508 and check that the changes are correct.
510 * Commit the ``mappings.yml`` change together with the regenerated test
511 files, and submit a merge request to the ``libvirt-ci`` project.
512 Please note in the description that this is a new build pre-requisite
515 * CI pipeline will run to validate that the changes to ``mappings.yml``
516 are correct, by attempting to install the newly listed package on
519 * Once the merge request is accepted, go back to QEMU and update
520 the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
521 contains the ``mappings.yml`` update. Then add the prerequisite and
526 obvious breakages when adding the new pre-requisite. Please see
530 For enterprise distros that default to old, end-of-life versions of the
533 Modifying this file should not be necessary unless the new pre-requisite
540 In some cases ``libvirt-ci`` will not know about the OS distro that is
541 desired to be tested. Before adding a new OS distro, discuss the proposed
544 * Send a mail to qemu-devel, copying people listed in the
547 There are limited CI compute resources available to QEMU, so the
551 pointing to the qemu-devel mail thread in the archives.
553 This alerts other people who might be interested in the work
555 maintainers on any tips to ease the addition
559 * Fork the ``libvirt-ci`` project on gitlab
561 * Add metadata under ``lcitool/facts/targets/`` for the new OS
562 distro. There might be code changes required if the OS distro
563 uses a package format not currently known. The ``libvirt-ci``
564 maintainers can advise on this when the issue is filed.
566 * Edit the ``lcitool/facts/mappings.yml`` change to add entries for
567 the new OS, listing the native package names for as many packages
569 check that the changes are correct.
571 * Commit the changes to ``lcitool/facts`` and the regenerated test
572 files, and submit a merge request to the ``libvirt-ci`` project.
573 Please note in the description that this is a new build pre-requisite
576 * CI pipeline will run to validate that the changes to ``mappings.yml``
577 are correct, by attempting to install the newly listed package on
580 * Once the merge request is accepted, go back to QEMU and update
581 the ``libvirt-ci`` submodule to point to a commit that contains
582 the ``mappings.yml`` update.
589 QEMU. Docker tests are the executables under ``tests/docker`` named
591 library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
594 The full list of tests is printed in the ``make docker-help`` help.
599 When CI tasks, maintainers or yourself report a Docker test failure, follow the
602 1. Locally reproduce the failure with the reported command line. E.g. run
604 2. Add "V=1" to the command line, try again, to see the verbose output.
605 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
606 in the container right before testing starts. You could either manually
607 build QEMU and run tests from there, or press Ctrl-D to let the Docker
609 4. If you press Ctrl-D, the same building and testing procedure will begin, and
610 will hopefully run into the error again. After that, you will be dropped to
611 the prompt for debug.
616 Various options can be used to affect how Docker tests are done. The full
617 list is in the ``make docker`` help text. The frequently used ones are:
619 * ``V=1``: the same as in top level ``make``. It will be propagated to the
621 * ``J=$N``: the number of parallel tasks in make commands in the container,
622 similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
623 top level ``make`` will not be propagated into the container.)
624 * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
639 TSan is currently supported in the ubuntu2204 docker.
641 The test-tsan test will build using TSan and then run make check.
649 We recommend using DEBUG=1 to allow launching the test from inside the docker,
650 and to allow review of the warnings generated by TSan.
656 These steps are normally done automatically in the docker.
659 One particularity of sanitizers is that all the code, including shared objects
661 In the case of TSan, any synchronization primitive from glib (GMutex for
673 To configure the build for TSan:
690 The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment
693 More information on the TSAN_OPTIONS can be found here:
705 The above exitcode=0 has TSan continue without error if any warnings are found.
706 This allows for running the test and then checking the warnings afterwards.
716 to fix the code if possible to eliminate the data race rather than suppress
717 the warning.
722 The comment on each suppression will typically indicate why we are
723 suppressing it. More information on the file format can be found here:
733 More information on the file format can be found here under "Blacklist Format":
740 of the annotations themselves. Annotations can be used to suppress
750 The full set of annotations can be found here:
760 simplify setting up a rootfs by using docker to contain the foreign
766 You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU
767 user binary to automatically run binaries for the foreign
768 architecture. While the scripts will try their best to work with
770 potential complications when copying into the docker image. Modern
771 kernels support the ``F`` (fix binary) flag which will open the QEMU
772 executable on setup and avoids the need to find and re-open in the
773 chroot environment. This is triggered with the ``--persistent`` flag.
778 For example to setup the HPPA ports builds of Debian::
786 The ``DEB_`` variables are substitutions used by
787 ``debian-bootstrap.pre`` which is called to do the initial debootstrap
788 of the rootfs before it is copied into the container. The second stage
789 is run as part of the build. The final image will be tagged as
796 necessary packages to build QEMU. The basic usage is documented in ``Makefile``
804 will build the source tree in the FreeBSD image. The command can be executed
805 from either the source tree or the build dir; if the former, ``./configure`` is
806 not needed. The command will then generate the test image in ``./tests/vm/``
807 under the working directory.
809 Note: images created by the scripts accept a well-known RSA key pair for SSH
811 concerned about attackers taking control of the guest and potentially
812 exploiting a QEMU security bug to compromise the host.
817 By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If
818 there isn't one, or if it is older than 2.10, the test won't work. In this case,
819 provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
821 Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable.
826 The ``-j$X`` option in the make command line is not propagated into the VM,
827 specify ``J=$X`` to control the make jobs in the guest.
832 Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
833 debugging and verbose output. If this is not enough, see the next section.
834 ``V=1`` will be propagated down into the make jobs in the guest.
839 Each guest script is an executable script with the same command line options.
840 For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
846 # To bootstrap the image
850 # To run an arbitrary command in guest (the output will not be echoed unless
867 the script's ``main()``.
870 predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
871 the checksum, so consider using it.
873 * Once the image is downloaded, users, SSH server and QEMU build deps should
881 file of both root and the normal user
883 automatically configure the virtio-net-pci NIC and communicate with QEMU
885 - Necessary packages are installed to untar the source tarball and build
889 untars a raw virtio-blk block device, which is the tarball data blob of the
897 supported. To start the fuzzer, run
904 changing the ``-c`` option.
909 A functional test focuses on the functional requirement of the software,
913 The ``tests/functional`` directory hosts functional tests written in
914 Python. You can run the functional tests simply by executing:
927 The check-tcg tests are intended for simple smoke tests of both
935 The configure script will automatically pick up their presence.
936 Sometimes compilers have slightly odd names so the availability of
937 them can be prompted by passing in the appropriate configure option
938 for the architecture in question, for example::
945 If you have the ability to run containers as the user the build system
948 use the same container to build tests. However there are a number of
952 for test cases that aren't yet in the LTS distros we support for QEMU
960 You can build the tests for one architecture::
968 Adding ``V=1`` to the invocation will show the details of how to
969 invoke QEMU for the test which is useful for debugging tests.
974 Tests can also be run directly from the test build directory. If you
975 run ``make help`` from the test build directory you will get a list of
976 all the tests that can be run. Please note that same binaries are used
981 will run the mmap test with the ``libinline.so`` TCG plugin. The
982 gdbstub tests also re-use the test binaries but while exercising gdb.
987 The TCG tests are deliberately very light on dependencies and are
1001 The KVM unit tests are designed to run as a Guest OS under KVM but
1002 there is no reason why they can't exercise the TCG as well. It
1003 provides a minimal OS kernel with hooks for enabling the MMU as well
1011 The LTP is focused on exercising the syscall interface of a Linux
1021 ``gcov`` is a GCC tool to analyze the testing coverage by
1022 instrumenting the tested code. To use it, configure QEMU with
1023 ``--enable-gcov`` option and build. Then run the tests as usual.
1025 If you want to gather coverage information on a single test the ``make
1033 Further analysis can be conducted by running the ``gcov`` command
1034 directly on the various .gcda output files. Please read the ``gcov``
1041 result with the same code on different runs. Some usual reasons for an
1046 default. Set the QEMU_TEST_FLAKY_TESTS environment variable before running
1047 the tests to enable them.
1058 Proceedings of the 22nd ACM SIGSOFT International Symposium on