atf_python: Fix a typo in a type annotation

No functional change intended.

MFC after: 1 week
Sponsored by: Stormshield
Sponsored by: Klara, Inc.

atf_pytest: fix xfail detection from pytest report

The location of the 'wasxfail' attribute was moved from
the 'reason' attribute back to the parent 'report'. This
fixes an issue where xfails are wr

atf_pytest: fix xfail detection from pytest report

The location of the 'wasxfail' attribute was moved from
the 'reason' attribute back to the parent 'report'. This
fixes an issue where xfails are wrongly reported to ATF
as skipped tests.

Signed-off-by: Siva Mahadevan <me@svmhdvn.name>
Pull-request: https://github.com/freebsd/freebsd-src/pull/1849
Sponsored by: The FreeBSD Foundation

show more ...

Testing: add framework for the kernel unit tests.

This changes intends to reduce the bar to the kernel unit-testing by
introducing a new kernel-testing framework ("ktest") based on Netlink,
loadab

Testing: add framework for the kernel unit tests.

This changes intends to reduce the bar to the kernel unit-testing by
introducing a new kernel-testing framework ("ktest") based on Netlink,
loadable test modules and python test suite integration.

This framework provides the following features:
* Integration to the FreeBSD test suite
* Automatic test discovery
* Automatic test module loading
* Minimal boiler-plate code in both kernel and userland
* Passing any metadata to the test
* Convenient environment pre-setup using python testing framework
* Streaming messages from the kernel to the userland
* Running tests in the dedicated taskqueues
* Skipping or parametrizing tests

Differential Revision: https://reviews.freebsd.org/D39385
MFC after: 2 weeks

show more ...

tests: refactor atf_python a bit

* Move more logic from conftest.py to the actual atf_pytest handler
* Move nodeid_to_method_name() to the utils.py so it can be shared

MFC after: 2 weeks

testing: handling non-root users with VNETs in pytest-based tests.

Currently isolation and resource requirements are handled directly
by the kyua runner, based on the requirements specified by the

testing: handling non-root users with VNETs in pytest-based tests.

Currently isolation and resource requirements are handled directly
by the kyua runner, based on the requirements specified by the test.
It works well for simple tests, but may cause discrepancy with tests
doing complex pre-setups. For example, all tests that perform
VNET setups require root access to properly function.

This change adds additional handling of the "require_user" property
within the python testing framework. Specifically, it requests
root access if the test class signals its root requirements and
drops privileges to the desired user after performing the pre-setup.

Differential Revision: https://reviews.freebsd.org/D37923
MFC after: 2 weeks

show more ...

testing: allow custom test cleanup handlers in pytest

In order to provide more flexibility for the test writers,
add per-test-method cleanups in addition to the per-class cleanups.

Now the test 'te

testing: allow custom test cleanup handlers in pytest

In order to provide more flexibility for the test writers,
add per-test-method cleanups in addition to the per-class cleanups.

Now the test 'test_one' can perform cleanup by either defining
per-class 'cleanup' method (typically used in VNET classes) and
per-test method 'cleanup_test_one'. The latter has preference.
In order to handle paramatrization, testid is passed as a single
argument to both of the methods.

MFC after: 2 weeks

show more ...

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spac

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spaces or just more than one --atf-var argument.
Switch wrapper to send these kv pairs as env variables. Specifically,
use _ATF_VAR_key=value format to distinguish from the other vars.

Add the `atf_vars` fixture returning all passed kv pairs as a dict.

Reviewed by: lwhsu
Differential Revision: https://reviews.freebsd.org/D35625
MFC after: 2 weeks

show more ...

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the format understandable by pytest. Each test has this wrapper specified
after the shebang. When kyua executes the test, wrapper calls pytest, which
loads atf plugin, does the work and returns the result. Additionally, a
separate python "package", `/usr/tests/atf_python` has been added to collect
code that may be useful across different tests.

Current limitations:
* Opaque metadata passing via X-Name properties. Require some fixtures to write
* `-s srcdir` parameter passed by the runner is ignored.
* No `atf-c-api(3)` or similar - relying on pytest framework & existing python libraries
* No support for `atf_tc_<get|has>_config_var()` & `atf_tc_set_md_var()`.
Can be probably implemented with env variables & autoload fixtures

Differential Revision: https://reviews.freebsd.org/D31084
Reviewed by: kp, ngie

show more ...

atf_python: Fix a typo in a type annotation

No functional change intended.

MFC after: 1 week
Sponsored by: Stormshield
Sponsored by: Klara, Inc.

atf_pytest: fix xfail detection from pytest report

The location of the 'wasxfail' attribute was moved from
the 'reason' attribute back to the parent 'report'. This
fixes an issue where xfails are wr

atf_pytest: fix xfail detection from pytest report

The location of the 'wasxfail' attribute was moved from
the 'reason' attribute back to the parent 'report'. This
fixes an issue where xfails are wrongly reported to ATF
as skipped tests.

Signed-off-by: Siva Mahadevan <me@svmhdvn.name>
Pull-request: https://github.com/freebsd/freebsd-src/pull/1849
Sponsored by: The FreeBSD Foundation

show more ...

Testing: add framework for the kernel unit tests.

This changes intends to reduce the bar to the kernel unit-testing by
introducing a new kernel-testing framework ("ktest") based on Netlink,
loadab

Testing: add framework for the kernel unit tests.

This changes intends to reduce the bar to the kernel unit-testing by
introducing a new kernel-testing framework ("ktest") based on Netlink,
loadable test modules and python test suite integration.

This framework provides the following features:
* Integration to the FreeBSD test suite
* Automatic test discovery
* Automatic test module loading
* Minimal boiler-plate code in both kernel and userland
* Passing any metadata to the test
* Convenient environment pre-setup using python testing framework
* Streaming messages from the kernel to the userland
* Running tests in the dedicated taskqueues
* Skipping or parametrizing tests

Differential Revision: https://reviews.freebsd.org/D39385
MFC after: 2 weeks

show more ...

tests: refactor atf_python a bit

* Move more logic from conftest.py to the actual atf_pytest handler
* Move nodeid_to_method_name() to the utils.py so it can be shared

MFC after: 2 weeks

testing: handling non-root users with VNETs in pytest-based tests.

Currently isolation and resource requirements are handled directly
by the kyua runner, based on the requirements specified by the

testing: handling non-root users with VNETs in pytest-based tests.

Currently isolation and resource requirements are handled directly
by the kyua runner, based on the requirements specified by the test.
It works well for simple tests, but may cause discrepancy with tests
doing complex pre-setups. For example, all tests that perform
VNET setups require root access to properly function.

This change adds additional handling of the "require_user" property
within the python testing framework. Specifically, it requests
root access if the test class signals its root requirements and
drops privileges to the desired user after performing the pre-setup.

Differential Revision: https://reviews.freebsd.org/D37923
MFC after: 2 weeks

show more ...

testing: allow custom test cleanup handlers in pytest

In order to provide more flexibility for the test writers,
add per-test-method cleanups in addition to the per-class cleanups.

Now the test 'te

testing: allow custom test cleanup handlers in pytest

In order to provide more flexibility for the test writers,
add per-test-method cleanups in addition to the per-class cleanups.

Now the test 'test_one' can perform cleanup by either defining
per-class 'cleanup' method (typically used in VNET classes) and
per-test method 'cleanup_test_one'. The latter has preference.
In order to handle paramatrization, testid is passed as a single
argument to both of the methods.

MFC after: 2 weeks

show more ...

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spac

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spaces or just more than one --atf-var argument.
Switch wrapper to send these kv pairs as env variables. Specifically,
use _ATF_VAR_key=value format to distinguish from the other vars.

Add the `atf_vars` fixture returning all passed kv pairs as a dict.

Reviewed by: lwhsu
Differential Revision: https://reviews.freebsd.org/D35625
MFC after: 2 weeks

show more ...

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the format understandable by pytest. Each test has this wrapper specified
after the shebang. When kyua executes the test, wrapper calls pytest, which
loads atf plugin, does the work and returns the result. Additionally, a
separate python "package", `/usr/tests/atf_python` has been added to collect
code that may be useful across different tests.

Current limitations:
* Opaque metadata passing via X-Name properties. Require some fixtures to write
* `-s srcdir` parameter passed by the runner is ignored.
* No `atf-c-api(3)` or similar - relying on pytest framework & existing python libraries
* No support for `atf_tc_<get|has>_config_var()` & `atf_tc_set_md_var()`.
Can be probably implemented with env variables & autoload fixtures

Differential Revision: https://reviews.freebsd.org/D31084
Reviewed by: kp, ngie

show more ...

atf_pytest: fix xfail detection from pytest report

The location of the 'wasxfail' attribute was moved from
the 'reason' attribute back to the parent 'report'. This
fixes an issue where xfails are wr

atf_pytest: fix xfail detection from pytest report

The location of the 'wasxfail' attribute was moved from
the 'reason' attribute back to the parent 'report'. This
fixes an issue where xfails are wrongly reported to ATF
as skipped tests.

Signed-off-by: Siva Mahadevan <me@svmhdvn.name>
Pull-request: https://github.com/freebsd/freebsd-src/pull/1849
Sponsored by: The FreeBSD Foundation

show more ...

Testing: add framework for the kernel unit tests.

This changes intends to reduce the bar to the kernel unit-testing by
introducing a new kernel-testing framework ("ktest") based on Netlink,
loadab

Testing: add framework for the kernel unit tests.

This changes intends to reduce the bar to the kernel unit-testing by
introducing a new kernel-testing framework ("ktest") based on Netlink,
loadable test modules and python test suite integration.

This framework provides the following features:
* Integration to the FreeBSD test suite
* Automatic test discovery
* Automatic test module loading
* Minimal boiler-plate code in both kernel and userland
* Passing any metadata to the test
* Convenient environment pre-setup using python testing framework
* Streaming messages from the kernel to the userland
* Running tests in the dedicated taskqueues
* Skipping or parametrizing tests

Differential Revision: https://reviews.freebsd.org/D39385
MFC after: 2 weeks

show more ...

tests: refactor atf_python a bit

* Move more logic from conftest.py to the actual atf_pytest handler
* Move nodeid_to_method_name() to the utils.py so it can be shared

MFC after: 2 weeks

testing: handling non-root users with VNETs in pytest-based tests.

Currently isolation and resource requirements are handled directly
by the kyua runner, based on the requirements specified by the

testing: handling non-root users with VNETs in pytest-based tests.

Currently isolation and resource requirements are handled directly
by the kyua runner, based on the requirements specified by the test.
It works well for simple tests, but may cause discrepancy with tests
doing complex pre-setups. For example, all tests that perform
VNET setups require root access to properly function.

This change adds additional handling of the "require_user" property
within the python testing framework. Specifically, it requests
root access if the test class signals its root requirements and
drops privileges to the desired user after performing the pre-setup.

Differential Revision: https://reviews.freebsd.org/D37923
MFC after: 2 weeks

show more ...

testing: allow custom test cleanup handlers in pytest

In order to provide more flexibility for the test writers,
add per-test-method cleanups in addition to the per-class cleanups.

Now the test 'te

testing: allow custom test cleanup handlers in pytest

In order to provide more flexibility for the test writers,
add per-test-method cleanups in addition to the per-class cleanups.

Now the test 'test_one' can perform cleanup by either defining
per-class 'cleanup' method (typically used in VNET classes) and
per-test method 'cleanup_test_one'. The latter has preference.
In order to handle paramatrization, testid is passed as a single
argument to both of the methods.

MFC after: 2 weeks

show more ...

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spac

testing: pass ATF vars to pytest via env instead of arguments.

This change is a continuation of 9c42645a1e4d workaround.
Apparently pytest argument parser is not happy when parsing values
with spaces or just more than one --atf-var argument.
Switch wrapper to send these kv pairs as env variables. Specifically,
use _ATF_VAR_key=value format to distinguish from the other vars.

Add the `atf_vars` fixture returning all passed kv pairs as a dict.

Reviewed by: lwhsu
Differential Revision: https://reviews.freebsd.org/D35625
MFC after: 2 weeks

show more ...

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the

testing: Add basic atf support to pytest.

Implementation consists of the pytest plugin implementing ATF format and
a simple C++ wrapper, which reorders the provided arguments from ATF format
to the format understandable by pytest. Each test has this wrapper specified
after the shebang. When kyua executes the test, wrapper calls pytest, which
loads atf plugin, does the work and returns the result. Additionally, a
separate python "package", `/usr/tests/atf_python` has been added to collect
code that may be useful across different tests.

Current limitations:
* Opaque metadata passing via X-Name properties. Require some fixtures to write
* `-s srcdir` parameter passed by the runner is ignored.
* No `atf-c-api(3)` or similar - relying on pytest framework & existing python libraries
* No support for `atf_tc_<get|has>_config_var()` & `atf_tc_set_md_var()`.
Can be probably implemented with env variables & autoload fixtures

Differential Revision: https://reviews.freebsd.org/D31084
Reviewed by: kp, ngie

show more ...