atf_pytest_wrapper: fix use with pytest-7.4

As of pytest 7.4 it no longer walks all the way to the root directory
of the file system to find conftest files. As a result we don't find /
usr/tests/con

atf_pytest_wrapper: fix use with pytest-7.4

As of pytest 7.4 it no longer walks all the way to the root directory
of the file system to find conftest files. As a result we don't find /
usr/tests/conftest.py, and don't load atf_python. That in turn causes
atf_python tests to fail.

Explicitly set the confcutdir, as advised by the pytest changelog.

See also: https://github.com/pytest-dev/pytest/pull/11043
MFC after: 3 weeks
Sponsored by: Orange Business Services
Differential Revision: https://reviews.freebsd.org/D41064

show more ...

atf_pytest_wrapper: fix pytest output truncation

Pass `-vv` to pytest in order to always get the full output.
While here, enforce the modeline.

Differential Revision: https://reviews.freebsd.org/D3

atf_pytest_wrapper: fix pytest output truncation

Pass `-vv` to pytest in order to always get the full output.
While here, enforce the modeline.

Differential Revision: https://reviews.freebsd.org/D37894
MFC after: 2 weeks

show more ...

testing: provide meaningful error when pytest is not available

atf format does not provide any way of signalling any error message
back to the atf runner when listing tests. Work this around by
re

testing: provide meaningful error when pytest is not available

atf format does not provide any way of signalling any error message
back to the atf runner when listing tests. Work this around by
reporting "__test_cases_list_pytest_binary_not_found__" test instead.

Reviewed By: kp
Differential Revision: https://reviews.freebsd.org/D35721

show more ...

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spac

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spaces or just more than one --atf-var argument.
Switch wrapper to send these kv pairs as env variables. Specifically,
use _ATF_VAR_key=value format to distinguish from the other vars.

Add the `atf_vars` fixture returning all passed kv pairs as a dict.

Reviewed by: lwhsu
Differential Revision: https://reviews.freebsd.org/D35625
MFC after: 2 weeks

show more ...

testing: workaround pytest parser bug in pytest-atf-wrapper.

Reviewed by: lwhsu, kp
Differential Revision: https://reviews.freebsd.org/D35614
MFC after: 2 weeks

testing: move atf-pytest-wrapper to /usr/libexec

Move pytest wrapper to the collection of the other atf wrappers
in libexec. It solves the problem of combining bits & pieces from
bsd.test.mk and b

testing: move atf-pytest-wrapper to /usr/libexec

Move pytest wrapper to the collection of the other atf wrappers
in libexec. It solves the problem of combining bits & pieces from
bsd.test.mk and bgs.prog.mk to address "test binary, but not the
suite binary".

Reviewed by: kp
Differential Revision: https://reviews.freebsd.org/D35604
MFC after: 2 weeks

show more ...

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the format understandable by pytest. Each test has this wrapper specified
after the shebang. When kyua executes the test, wrapper calls pytest, which
loads atf plugin, does the work and returns the result. Additionally, a
separate python "package", `/usr/tests/atf_python` has been added to collect
code that may be useful across different tests.

Current limitations:
* Opaque metadata passing via X-Name properties. Require some fixtures to write
* `-s srcdir` parameter passed by the runner is ignored.
* No `atf-c-api(3)` or similar - relying on pytest framework & existing python libraries
* No support for `atf_tc_<get|has>_config_var()` & `atf_tc_set_md_var()`.
Can be probably implemented with env variables & autoload fixtures

Differential Revision: https://reviews.freebsd.org/D31084
Reviewed by: kp, ngie

show more ...