devel/testing/functional.rst

6 The ``tests/functional`` directory hosts functional tests written in
10 The tests should be written in the style of the Python `unittest`_ framework,
11 using stdio for the TAP protocol. The folder ``tests/functional/qemu_test``
12 provides classes (e.g. the ``QemuBaseTest``, ``QemuUserTest`` and the
14 into the right shape, e.g. by replacing the 'stdout' python object to redirect
15 the normal output of your test to stderr instead.
17 Note that if you don't use one of the QemuBaseTest based classes for your
20 prefixing every line with a "# " to mark the output as a TAP comment, or
21 e.g. by capturing the stdout output of subprocesses (redirecting it to
26  * Customize the command line arguments given to the convenience
29  * Interact with the QEMU monitor, send QMP commands and check
32  * Interact with the guest OS, using the convenience console device
33    (which may be useful to assert the effectiveness and correctness of
42 You can run the functional tests simply by executing:
49 the following line will only run the tests for the x86_64 target:
55 To run a single test file without the meson test runner, you can also
56 execute the file directly by specifying two environment variables first,
57 the PYTHONPATH that has to include the python folder and the tests/functional
58 folder of the source tree, and QEMU_TEST_QEMU_BINARY that has to point
59 to the QEMU binary that should be used for the test. The current working
66 The test framework will automatically purge any scratch files created during
67 the tests. If needing to debug a failed test, it is possible to keep these
69 variable.  Any preserved files will be deleted the next time the test is run
75 The framework collects log files for each test in the build directory
76 in the following subfolder::
82 * ``base.log`` contains the generic logging information that is written
83   by the calls to the logging functions in the test code (e.g. by calling
84   the ``self.log.info()`` or ``self.log.debug()`` functions).
85 * ``console.log`` contains the output of the serial console of the guest.
86 * ``default.log`` contains the output of QEMU. This file could be named
87   differently if the test chooses to use a different identifier for
88   the guest VM (e.g. when the test spins up multiple VMs).
93 The ``tests/functional/qemu_test`` directory provides the ``qemu_test``
94 Python module, containing the ``qemu_test.QemuSystemTest`` class.
114 By providing the "hash bang" line at the beginning of the script, marking
115 the file as executable and by calling into QemuSystemTest.main(), the test
117 runner, the QemuSystemTest.main() function takes care of running the test
118 functions in the right fassion (e.g. with TAP output that is required by the
121 The ``qemu_test.QemuSystemTest`` base test class
124 The ``qemu_test.QemuSystemTest`` class has a number of characteristics
128 instance, available at ``self.vm``.  Because many tests will tweak the
129 QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
130 is left to the test writer.
132 The base test class has also support for tests with more than one
133 QEMUMachine. The way to get machines is through the ``self.get_vm()``
134 method which will return a QEMUMachine instance. The ``self.get_vm()``
135 method accepts arguments that will be passed to the QEMUMachine creation
137 machine and get it more than once through the tests methods. A simple
167 At test "tear down", ``qemu_test.QemuSystemTest`` handles all the QEMUMachines
173 The QEMUMachine API is already widely used in the Python iotests,
174 device-crash-test and other Python scripts.  It's a wrapper around the
177  * the ability to set command line arguments to be given to the QEMU
190 The QEMU binary used for the ``self.vm`` QEMUMachine instance will
191 primarily depend on the value of the ``qemu_bin`` instance attribute.
192 If it is not explicitly set by the test code, its default value will
193 be the result the QEMU_TEST_QEMU_BINARY environment variable.
199 process. While the QEMUMachine class owns the primary QMP monitor
201 by setting the ``QEMU_TEST_QMP_BACKDOOR`` env variable to refer
202 to a UNIX socket name. The ``qmp-shell`` command can then be
203 attached to the stalled QEMU to examine its live state.
211 The following attributes are available on any ``qemu_test.QemuBaseTest``
217 The target architecture of the QEMU binary.
220 A test may, for instance, use this value when selecting the architecture
226 The preserved value of the ``QEMU_TEST_QEMU_BINARY`` environment
232 The QemuUserTest class can be used for running an executable via the
238 The QemuSystemTest class can be used for running tests via one of the
244 A QEMUMachine instance, initially configured according to the given
250 The cpu model that will be set to all QEMUMachine instances created
251 by the test.
256 The machine type that will be set to all QEMUMachine instances created
257 by the test. By using the set_machine() function of the QemuSystemTest
258 class to set this attribute, you can automatically check whether the
259 machine is available to skip the test in case it is not built into the
266 firmware images, etc.) from the internet to be able to run tests with
267 them. This imposes additional challenges to the test framework.
269 First there is the problem that some people might not have an
272 the tests that download files should only be added to the "thorough"
273 speed mode in the meson.build file, while the "quick" speed mode is
275 ``make check`` then only runs the quick functional tests along with
276 the other quick tests from the other test suites. If you choose to
277 run only ``make check-functional``, the "thorough" tests will be
278 executed, too. And to run all functional tests along with the others,
283 The second problem with downloading files from the internet are time
284 constraints. The time for downloading files should not be taken into
285 account when the test is running and the timeout of the test is ticking
286 (since downloading can be very slow, depending on the network bandwidth).
287 This problem is solved by downloading the assets ahead of time, before
288 the tests are run. This pre-caching is done with the qemu_test.Asset
298 In your test function, you can then get the file name of the cached
304 The pre-caching will be done automatically when running
307 the assets without running the tests, you can do so by running::
311 The cache is populated in the ``~/.cache/qemu/download`` directory by
312 default, but the location can be changed by setting the
318 Since the test framework is based on the common Python unittest framework,
319 you can use the usual Python decorators which allow for easily skipping
320 tests running under certain conditions, for example, on the lack of a binary
321 on the test system or when the running environment is a CI system. For further
326 While the conditions for skipping tests are often specifics of each one, there
327 are recurring scenarios identified by the QEMU developers and the use of
330 Here is a list of the most used variables:
336 the environment.
338 The definition of *large* is a bit arbitrary here, but it usually means an
344 considered not safe to run on the developer's workstation, thus they are
345 skipped by default. The definition of *not safe* is also arbitrary but
349 You should export ``QEMU_TEST_ALLOW_UNTRUSTED_CODE=1`` on the environment in
357 due to the constraints of the CI environment. If you encounter a
358 similar situation then raise a bug and then mark the test as shown on
359 the code snippet below:
374 e.g. if the QEMU binary has been compiled with debugging options enabled.
377 the QEMU_TEST_ALLOW_SLOW environment variable has been set.