devel/testing/functional.rst

6 The ``tests/functional`` directory hosts functional tests written in
7 Python. They are usually higher level tests, and may interact with
10 The tests should be written in the style of the Python `unittest`_ framework,
11 using stdio for the TAP protocol. The folder ``tests/functional/qemu_test``
24 Tests based on ``qemu_test.QemuSystemTest`` can easily:
39 Running tests
42 You can run the functional tests simply by executing:
48 It is also possible to run tests for a certain target only, for example
49 the following line will only run the tests for the x86_64 target:
57 the PYTHONPATH that has to include the python folder and the tests/functional
62   $ export PYTHONPATH=../python:../tests/functional
64   $ pyvenv/bin/python3 ../tests/functional/test_file.py
67 the tests. If needing to debug a failed test, it is possible to keep these
78  <builddir>/tests/functional/<arch>/<fileid>.<classid>.<testname>/
90 Introduction to writing tests
93 The ``tests/functional/qemu_test`` directory provides the ``qemu_test``
128 instance, available at ``self.vm``.  Because many tests will tweak the
132 The base test class has also support for tests with more than one
137 machine and get it more than once through the tests methods. A simple
219 Tests are also free to use this attribute value, for their own needs.
238 The QemuSystemTest class can be used for running tests via one of the
265 Many functional tests download assets (e.g. Linux kernels, initrds,
266 firmware images, etc.) from the internet to be able to run tests with
270 unconstrained internet connection, so such tests should not be run by
272 the tests that download files should only be added to the "thorough"
274 fine for functional tests that can be run without downloading files.
275 ``make check`` then only runs the quick functional tests along with
276 the other quick tests from the other test suites. If you choose to
277 run only ``make check-functional``, the "thorough" tests will be
278 executed, too. And to run all functional tests along with the others,
288 the tests are run. This pre-caching is done with the qemu_test.Asset
307 the assets without running the tests, you can do so by running::
315 Skipping tests
320 tests running under certain conditions, for example, on the lack of a binary
324   https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures
326 While the conditions for skipping tests are often specifics of each one, there
328 environment variables became a kind of standard way to enable/disable tests.
334 Tests which are going to fetch or produce assets considered *large* are not
343 There are tests which will boot a kernel image or firmware that can be
350 order to allow tests which make use of those kind of assets.
354 Some tests are not working reliably and thus are disabled by default.
355 This includes tests that don't run reliably on GitLab's CI which
368 Tests should not live in this state forever and should either be fixed
373 Tests that have a very long runtime and might run into timeout issues
376 cycles during normal testing, such tests are disabled by default unless