kyua: Add "debug -x|--execute cmd" option

With execenv=jail specified, the "cmd" runs inside the test's jail.

Reviewed by: ngie, markj
Differential Revision: https://reviews.freebsd.org/D52642

kyua: Make "debug -p" skip writing stdout/stderr to tmp files

Reviewed by: kp, ngie
Differential Revision: https://reviews.freebsd.org/D54363

kyua: increase cleanup_timeout to 300s

Test cleanup times out after 60 seconds, causing the test to fail if the
cleanup hasn't finished in time. This timeout should be configurable,
but it currentl

kyua: increase cleanup_timeout to 300s

Test cleanup times out after 60 seconds, causing the test to fail if the
cleanup hasn't finished in time. This timeout should be configurable,
but it currently is not.

Several tests (e.g. bridge, pf) are now reaching this limit; as a
workaround, increase the timeout to 300 seconds.

Reviewed by: igoro, kevans, des
Approved by: des (mentor), kevans (mentor)
Differential Revision: https://reviews.freebsd.org/D50499

show more ...

kyua: Add "kyua debug -p" option

This allows the test engine to be paused right before the test cleanup
routine, simplifying debugging.

Reviewed by: ziaee, ngie
Differential Revision: https://revie

kyua: Add "kyua debug -p" option

This allows the test engine to be paused right before the test cleanup
routine, simplifying debugging.

Reviewed by: ziaee, ngie
Differential Revision: https://reviews.freebsd.org/D49463

show more ...

kyua: Switch from std::auto_ptr<> to std::unique_ptr<>

A few places were assigning auto_ptr<>s depending on the ownership
handoff. These now use an explicit std::move() as required by
unique_ptr<>.

kyua: Switch from std::auto_ptr<> to std::unique_ptr<>

A few places were assigning auto_ptr<>s depending on the ownership
handoff. These now use an explicit std::move() as required by
unique_ptr<>.

Reviewed by: ngie, emaste
Differential Revision: https://reviews.freebsd.org/D49790

show more ...

kyua: Pass unprivileged user config prop to ATF using all known names

Kyua and ATF speak different naming styles. In this case, the
unprivileged user property can be named with underscore on the Kyu

kyua: Pass unprivileged user config prop to ATF using all known names

Kyua and ATF speak different naming styles. In this case, the
unprivileged user property can be named with underscore on the Kyua
side, and with a hyphen on the ATF side. Sometimes it is not obvious
which style should be used in which situation. For instance, a test case
may require this configuration property being set using require.config.
Also, a test case may want to read the property using something like
atf_tc_get_config_var(). Which names should be used in these cases?
From the perspective of the original code, it is expected to be this:
require.config unprivileged-user
atf_tc_get_config_var(tc, "unprivileged-user")

But, as long as Kyua is the main interface, its users expect to work
with kyua.conf(5), which says that it must be named as unprivileged_user
(with underscore). As a result, test authors tend to do this instead:
require.config unprivileged_user
atf_tc_get_config_var(tc, "unprivileged_user")

Kyua already has hacks to understand both unprivileged_user and
unprivileged-user coming from require.config. And this patch covers the
missing second part -- make Kyua pass both names back to ATF as two
identical configuration properties named different ways.

Reviewed by: ngie, asomers
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D49039

show more ...

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle is extended respectively:
- execenv init (creates a jail or does nothing for default
execenv="host")
- test exec
- cleanup exec (optional)
- execenv cleanup (removes a jail or does nothing for default
execenv="host")

The following new functionality is provided, from bottom to top:

1 ATF based tests

- The new "execenv" metadata property can be set to explicitly ask for
an execution environment: "host" or "jail". If it's not defined, as
all existing tests do, then it implicitly means "host".

- The new "execenv.jail.params" metadata property can be optionally
defined to ask Kyua to use specific jail(8) parameters during creation
of a temporary jail. An example is "vnet allow.raw_sockets".

Kyua implicitly adds "children.max" to "execenv_jail_params"
parameters with the maximum possible value. A test case can override
it.

2 Kyuafile

- The same new metadata properties can be defined on Kyuafile level:
"execenv" and "execenv_jail_params".

- Note that historically ATF uses dotted style of metadata naming, while
Kyua uses underscore style. Hence "execenv.jail.params" vs.
"execenv_jail_params".

3 kyua.conf, kyua CLI

- The new "execenvs" engine configuration variable can be set to a list
of execution environments to run only tests designed for. Tests of not
listed environments are skipped.

- By default, this variable lists all execution environments supported
by a Kyua binary, e.g. execenvs="host jail".

- This variable can be changed via "kyua.conf" or via kyua CLI's "-v"
parameter. For example, "kyua -v execenvs=host test" will run only
host-based tests and skip jail-based ones.

- Current value of this variable can be examined with "kyua config".

[markj] This feature has not landed upstream yet.
See the discussion in https://github.com/freebsd/kyua/pull/224 .
Having the ability to automatically jail tests allows many network tests
to run in parallel, giving a drastic speedup. So, let's import the
feature and start using it in main.

Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Reviewed by: markj, kp
Tested by: markj, kp
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D45865

show more ...

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OU

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OUT)_TESTS_SUPPORT.

Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24103

show more ...

kyua: Add "debug -x|--execute cmd" option

With execenv=jail specified, the "cmd" runs inside the test's jail.

Reviewed by: ngie, markj
Differential Revision: https://reviews.freebsd.org/D52642

kyua: Make "debug -p" skip writing stdout/stderr to tmp files

Reviewed by: kp, ngie
Differential Revision: https://reviews.freebsd.org/D54363

kyua: increase cleanup_timeout to 300s

Test cleanup times out after 60 seconds, causing the test to fail if the
cleanup hasn't finished in time. This timeout should be configurable,
but it currentl

kyua: increase cleanup_timeout to 300s

Test cleanup times out after 60 seconds, causing the test to fail if the
cleanup hasn't finished in time. This timeout should be configurable,
but it currently is not.

Several tests (e.g. bridge, pf) are now reaching this limit; as a
workaround, increase the timeout to 300 seconds.

Reviewed by: igoro, kevans, des
Approved by: des (mentor), kevans (mentor)
Differential Revision: https://reviews.freebsd.org/D50499

show more ...

kyua: Add "kyua debug -p" option

This allows the test engine to be paused right before the test cleanup
routine, simplifying debugging.

Reviewed by: ziaee, ngie
Differential Revision: https://revie

kyua: Add "kyua debug -p" option

This allows the test engine to be paused right before the test cleanup
routine, simplifying debugging.

Reviewed by: ziaee, ngie
Differential Revision: https://reviews.freebsd.org/D49463

show more ...

kyua: Switch from std::auto_ptr<> to std::unique_ptr<>

A few places were assigning auto_ptr<>s depending on the ownership
handoff. These now use an explicit std::move() as required by
unique_ptr<>.

kyua: Switch from std::auto_ptr<> to std::unique_ptr<>

A few places were assigning auto_ptr<>s depending on the ownership
handoff. These now use an explicit std::move() as required by
unique_ptr<>.

Reviewed by: ngie, emaste
Differential Revision: https://reviews.freebsd.org/D49790

show more ...

kyua: Pass unprivileged user config prop to ATF using all known names

Kyua and ATF speak different naming styles. In this case, the
unprivileged user property can be named with underscore on the Kyu

kyua: Pass unprivileged user config prop to ATF using all known names

Kyua and ATF speak different naming styles. In this case, the
unprivileged user property can be named with underscore on the Kyua
side, and with a hyphen on the ATF side. Sometimes it is not obvious
which style should be used in which situation. For instance, a test case
may require this configuration property being set using require.config.
Also, a test case may want to read the property using something like
atf_tc_get_config_var(). Which names should be used in these cases?
From the perspective of the original code, it is expected to be this:
require.config unprivileged-user
atf_tc_get_config_var(tc, "unprivileged-user")

But, as long as Kyua is the main interface, its users expect to work
with kyua.conf(5), which says that it must be named as unprivileged_user
(with underscore). As a result, test authors tend to do this instead:
require.config unprivileged_user
atf_tc_get_config_var(tc, "unprivileged_user")

Kyua already has hacks to understand both unprivileged_user and
unprivileged-user coming from require.config. And this patch covers the
missing second part -- make Kyua pass both names back to ATF as two
identical configuration properties named different ways.

Reviewed by: ngie, asomers
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D49039

show more ...

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle is extended respectively:
- execenv init (creates a jail or does nothing for default
execenv="host")
- test exec
- cleanup exec (optional)
- execenv cleanup (removes a jail or does nothing for default
execenv="host")

The following new functionality is provided, from bottom to top:

1 ATF based tests

- The new "execenv" metadata property can be set to explicitly ask for
an execution environment: "host" or "jail". If it's not defined, as
all existing tests do, then it implicitly means "host".

- The new "execenv.jail.params" metadata property can be optionally
defined to ask Kyua to use specific jail(8) parameters during creation
of a temporary jail. An example is "vnet allow.raw_sockets".

Kyua implicitly adds "children.max" to "execenv_jail_params"
parameters with the maximum possible value. A test case can override
it.

2 Kyuafile

- The same new metadata properties can be defined on Kyuafile level:
"execenv" and "execenv_jail_params".

- Note that historically ATF uses dotted style of metadata naming, while
Kyua uses underscore style. Hence "execenv.jail.params" vs.
"execenv_jail_params".

3 kyua.conf, kyua CLI

- The new "execenvs" engine configuration variable can be set to a list
of execution environments to run only tests designed for. Tests of not
listed environments are skipped.

- By default, this variable lists all execution environments supported
by a Kyua binary, e.g. execenvs="host jail".

- This variable can be changed via "kyua.conf" or via kyua CLI's "-v"
parameter. For example, "kyua -v execenvs=host test" will run only
host-based tests and skip jail-based ones.

- Current value of this variable can be examined with "kyua config".

[markj] This feature has not landed upstream yet.
See the discussion in https://github.com/freebsd/kyua/pull/224 .
Having the ability to automatically jail tests allows many network tests
to run in parallel, giving a drastic speedup. So, let's import the
feature and start using it in main.

Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Reviewed by: markj, kp
Tested by: markj, kp
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D45865

show more ...

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OU

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OUT)_TESTS_SUPPORT.

Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24103

show more ...

kyua: Add "debug -x|--execute cmd" option

With execenv=jail specified, the "cmd" runs inside the test's jail.

Reviewed by: ngie, markj
Differential Revision: https://reviews.freebsd.org/D52642

kyua: Make "debug -p" skip writing stdout/stderr to tmp files

Reviewed by: kp, ngie
Differential Revision: https://reviews.freebsd.org/D54363

kyua: increase cleanup_timeout to 300s

Test cleanup times out after 60 seconds, causing the test to fail if the
cleanup hasn't finished in time. This timeout should be configurable,
but it currentl

kyua: increase cleanup_timeout to 300s

Test cleanup times out after 60 seconds, causing the test to fail if the
cleanup hasn't finished in time. This timeout should be configurable,
but it currently is not.

Several tests (e.g. bridge, pf) are now reaching this limit; as a
workaround, increase the timeout to 300 seconds.

Reviewed by: igoro, kevans, des
Approved by: des (mentor), kevans (mentor)
Differential Revision: https://reviews.freebsd.org/D50499

show more ...

kyua: Add "kyua debug -p" option

This allows the test engine to be paused right before the test cleanup
routine, simplifying debugging.

Reviewed by: ziaee, ngie
Differential Revision: https://revie

kyua: Add "kyua debug -p" option

This allows the test engine to be paused right before the test cleanup
routine, simplifying debugging.

Reviewed by: ziaee, ngie
Differential Revision: https://reviews.freebsd.org/D49463

show more ...

kyua: Switch from std::auto_ptr<> to std::unique_ptr<>

A few places were assigning auto_ptr<>s depending on the ownership
handoff. These now use an explicit std::move() as required by
unique_ptr<>.

kyua: Switch from std::auto_ptr<> to std::unique_ptr<>

A few places were assigning auto_ptr<>s depending on the ownership
handoff. These now use an explicit std::move() as required by
unique_ptr<>.

Reviewed by: ngie, emaste
Differential Revision: https://reviews.freebsd.org/D49790

show more ...

kyua: Pass unprivileged user config prop to ATF using all known names

Kyua and ATF speak different naming styles. In this case, the
unprivileged user property can be named with underscore on the Kyu

kyua: Pass unprivileged user config prop to ATF using all known names

Kyua and ATF speak different naming styles. In this case, the
unprivileged user property can be named with underscore on the Kyua
side, and with a hyphen on the ATF side. Sometimes it is not obvious
which style should be used in which situation. For instance, a test case
may require this configuration property being set using require.config.
Also, a test case may want to read the property using something like
atf_tc_get_config_var(). Which names should be used in these cases?
From the perspective of the original code, it is expected to be this:
require.config unprivileged-user
atf_tc_get_config_var(tc, "unprivileged-user")

But, as long as Kyua is the main interface, its users expect to work
with kyua.conf(5), which says that it must be named as unprivileged_user
(with underscore). As a result, test authors tend to do this instead:
require.config unprivileged_user
atf_tc_get_config_var(tc, "unprivileged_user")

Kyua already has hacks to understand both unprivileged_user and
unprivileged-user coming from require.config. And this patch covers the
missing second part -- make Kyua pass both names back to ATF as two
identical configuration properties named different ways.

Reviewed by: ngie, asomers
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D49039

show more ...

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle is extended respectively:
- execenv init (creates a jail or does nothing for default
execenv="host")
- test exec
- cleanup exec (optional)
- execenv cleanup (removes a jail or does nothing for default
execenv="host")

The following new functionality is provided, from bottom to top:

1 ATF based tests

- The new "execenv" metadata property can be set to explicitly ask for
an execution environment: "host" or "jail". If it's not defined, as
all existing tests do, then it implicitly means "host".

- The new "execenv.jail.params" metadata property can be optionally
defined to ask Kyua to use specific jail(8) parameters during creation
of a temporary jail. An example is "vnet allow.raw_sockets".

Kyua implicitly adds "children.max" to "execenv_jail_params"
parameters with the maximum possible value. A test case can override
it.

2 Kyuafile

- The same new metadata properties can be defined on Kyuafile level:
"execenv" and "execenv_jail_params".

- Note that historically ATF uses dotted style of metadata naming, while
Kyua uses underscore style. Hence "execenv.jail.params" vs.
"execenv_jail_params".

3 kyua.conf, kyua CLI

- The new "execenvs" engine configuration variable can be set to a list
of execution environments to run only tests designed for. Tests of not
listed environments are skipped.

- By default, this variable lists all execution environments supported
by a Kyua binary, e.g. execenvs="host jail".

- This variable can be changed via "kyua.conf" or via kyua CLI's "-v"
parameter. For example, "kyua -v execenvs=host test" will run only
host-based tests and skip jail-based ones.

- Current value of this variable can be examined with "kyua config".

[markj] This feature has not landed upstream yet.
See the discussion in https://github.com/freebsd/kyua/pull/224 .
Having the ability to automatically jail tests allows many network tests
to run in parallel, giving a drastic speedup. So, let's import the
feature and start using it in main.

Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Reviewed by: markj, kp
Tested by: markj, kp
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D45865

show more ...

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OU

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OUT)_TESTS_SUPPORT.

Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24103

show more ...