generate-ucl.lua: Add missing copyright header

Add a standard BSD license with a copyright claim for everyone who has
modified this file since it was added, excluding users who explicitly
declined a

generate-ucl.lua: Add missing copyright header

Add a standard BSD license with a copyright claim for everyone who has
modified this file since it was added, excluding users who explicitly
declined attribution.

MFC after: 3 days
Reviewed by: ifreund_freebsdfoundation.org, imp, bapt, markj, emaste
Differential Revision: https://reviews.freebsd.org/D52881

show more ...

packages: Set dependency origin in generate-ucl.lua

Commit 822fd5606748 changed the origin of packages from "base"
to "base/<package name>", but the old origin was still used for
dependencies.

Asid

packages: Set dependency origin in generate-ucl.lua

Commit 822fd5606748 changed the origin of packages from "base"
to "base/<package name>", but the old origin was still used for
dependencies.

Aside from being wrong, this confuses pkg <= 2.3.1 and causes
"pkg info -d" to show the dependencies as "<pkgname>-(null)"
instead of the correct version string.

Remove the explicit origin from dependencies in the UCL files,
and instead populate this during the build in generate-ucl.lua.
In the (very unlikely) case that an explicit origin is still
needed, don't overwrite one if it's already present.

MFC after: 1 day
Fixes: 822fd5606748 ("pkgbase: set origin for each packages as base/FreeBSD-*")
Reported by: bapt
Reviewed by: cperciva, bapt
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D52892

show more ...

release: packages: Fix typos

Reviewed by: cperciva
Approved by: emaste (mentor)
MFC after: 1 day
Differential Revision: https://reviews.freebsd.org/D52917

packages: Add minimal-jail set

This is minimal, but without bootloader, hardware and networking support
that's typically not required in jails.

This requires extending the 'set' annotation to be a

packages: Add minimal-jail set

This is minimal, but without bootloader, hardware and networking support
that's typically not required in jails.

This requires extending the 'set' annotation to be a comma-separated
list, so that packages can be in multiple sets.

MFC after: 3 seconds
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D52591

show more ...

packages: remove tests from base set

Create a new tests package set to hold the tests, which are not useful
to all users and don't belong in the base set.

Reviewed by: emaste, ivy
Sponsored by: The

packages: remove tests from base set

Create a new tests package set to hold the tests, which are not useful
to all users and don't belong in the base set.

Reviewed by: emaste, ivy
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D52574

show more ...

packages: Add meta-package sets

Add several metapackages which contain no files themselves, but depend
on other packages. This allows the user to install a system by using
a set like "FreeBSD-set-m

packages: Add meta-package sets

Add several metapackages which contain no files themselves, but depend
on other packages. This allows the user to install a system by using
a set like "FreeBSD-set-minimal" instead of having to know which
specific packages are required, and it means if we add more package
in the future, existing installations will get them automatically as
long as they were installed using a set.

The defined sets are significantly less granular than the actual
packages; the assumption is that users who want a very specific set of
packages installed can do that manually, while sets are intended more
for typical users who just want to install FreeBSD.

The following sets are defined:

* minimal: the set of packages required to boot the system and bring
up a multi-user UNIX system. This includes hardware support,
networking (wireless, DHCP), basic functionality like syslogd, cron
and periodic. It does not include a kernel, because the kernels are
quite large and the user might want to use a custom kernel.

sendmail and DMA are not included in minimal. Since we provide two
MTAs, the user should select which one they want, or install one from
ports. sshd is not included because it's not required for the base
system, and the user might want to install it from ports.

Some other significant subsystems (e.g. NFS, Kerberos) are also not
included.

The minimal set does not include any libraries itself, since we rely
on dependencies to pull these in. This implies it doesn't include
library manpages, which is fine, since users won't want manpages for
libraries unless they've also installed the development packages.

* devel: the compiler toolchain for building software, along with all
"-dev" packages which include header files and static libraries.

* lib32: 32-bit compatibility packages. This includes both runtime
and development packages.

* kernels: the kernel packages. For freebsd.org builds, this will be
GENERIC, GENERIC-DEBUG, etc., for custom builds it will be whatever
was set in $KERNCONF.

* base: everything else; this includes the entire base system.

For each set, also generate a <setname>-dbg set containing the debug
packages for the set.

The sets are built along with the rest of the packages using a new
script called create-sets.sh, which examines the "set" annotation in
each package and puts it in the appropriate set. This is in
anticipation of the later "groups" feature appearing in pkg itself, at
which point we can simply replace the set annotation with the group.

MFC after: 3 days
Reviewed by: imp, bapt
Differential Revision: https://reviews.freebsd.org/D52412

show more ...

packages: Improve handling of -lib packages

For some packages (OpenSSL, Kerberos) we want to ship runtime libraries
in a separate package, e.g. openssl and openssl-lib. Currently this is
done using

packages: Improve handling of -lib packages

For some packages (OpenSSL, Kerberos) we want to ship runtime libraries
in a separate package, e.g. openssl and openssl-lib. Currently this is
done using PACKAGE=openssl-lib, but that creates packages with strange
names like openssl-lib-lib32.

Instead, add a new LIB_PACKAGE option to bsd.lib.mk that causes runtime
libraries to be placed in a new -lib subpackage. This significantly
improves the set of packages we create; for example, OpenSSL goes from:

FreeBSD-openssl
FreeBSD-openssl-dbg
FreeBSD-openssl-lib
FreeBSD-openssl-lib-dbg
FreeBSD-openssl-lib-dbg-lib32
FreeBSD-openssl-lib-dev
FreeBSD-openssl-lib-dev-lib32
FreeBSD-openssl-lib-lib32
FreeBSD-openssl-lib-man
FreeBSD-openssl-man

to:

FreeBSD-openssl
FreeBSD-openssl-dbg
FreeBSD-openssl-dbg-lib32
FreeBSD-openssl-dev
FreeBSD-openssl-dev-lib32
FreeBSD-openssl-lib
FreeBSD-openssl-lib32
FreeBSD-openssl-man

While here, move /usr/bin/krb5-config and /usr/bin/compile_et into
the kerberos-dev package.

Reviewed by: des
Differential Revision: https://reviews.freebsd.org/D51925

show more ...

share/man: Move many manpages to more correct packages

At the moment, all the manpages in share/man are in the utilities-man
package. Move some of them to the package they should actually be in,
us

share/man: Move many manpages to more correct packages

At the moment, all the manpages in share/man are in the utilities-man
package. Move some of them to the package they should actually be in,
using the new MANGROUPS feature.

Move all of section 3 to clibs-man. Although some of these are from
/usr/include/sys rather than libc, you can't practically use them
without libc, and clibs-man is where the libc manpages live already.

Move all of sections 4 and 9 to a new kernel-man package, except for
atf-test-case.4 which goes to tests-man. atf-test-case.4 is in the
wrong section, but this needs to be fixed upstream.

kernel-man requires special handling in generate-ucl.lua since it's
got a -man suffix but doesn't want the ' (manual pages)' automatic
suffix. For now, fix this by adding a list of packages that don't
get automatic suffixes.

Reviewed by: ifreund_freebsdfoundation.org, manu, emaste
Differential Revision: https://reviews.freebsd.org/D51504

show more ...

packages: handle suffixes in generate-ucl.lua

Move handling of comment/desc suffixes from generate-ucl.sh to
generate-ucl.lua.

Reviewed by: des, bapt
Approved by: des (mentor)
Differential Revision

packages: handle suffixes in generate-ucl.lua

Move handling of comment/desc suffixes from generate-ucl.sh to
generate-ucl.lua.

Reviewed by: des, bapt
Approved by: des (mentor)
Differential Revision: https://reviews.freebsd.org/D50284

show more ...

packages: handle dependencies in ucl

For packages with specific dependencies, add the dependencies in their
UCL files instead of listing them in generate-ucl.sh.

generate-ucl.sh has logic to add au

packages: handle dependencies in ucl

For packages with specific dependencies, add the dependencies in their
UCL files instead of listing them in generate-ucl.sh.

generate-ucl.sh has logic to add automatic subpackage dependencies, so
e.g. "FreeBSD-foo-dev" depends on "FreeBSD-foo". Move this into
generate-ucl.lua instead.

This means we no longer need to pre-process the UCL in generate-ucl.sh,
and we can undo the hack where it calls generate-ucl.lua with the same
filename as both input and output file.

Reviewed by: des, bapt
Approved by: des (mentor)
Differential Revision: https://reviews.freebsd.org/D50283

show more ...

packages: move ucl files to a subdirectory

We want to have a UCL file for every package, to set fields like
comment, desc, license, etc.; to avoid cluttering release/packages with
~200 new files, mo

packages: move ucl files to a subdirectory

We want to have a UCL file for every package, to set fields like
comment, desc, license, etc.; to avoid cluttering release/packages with
~200 new files, move these to the ucl/ directory.

Add a better description for the yp package to demonstrate how this will
work in practice.

Clean up the handling of the package comment suffix for generated
packages (dev, lib32, etc.) and do the UCL modifications in
generate-ucl.lua, so we don't overwrite the ucl's comment.

Add a long-form suffix to the description for generated packages to
explain what they do.

Eventually, we should require the ucl file exists (try=false) to prevent
people accidentally creating new packages with a typo, but for now, keep
the ucl optional.

Reviewed by: kevans, des, bapt, emaste
Approved by: kevans (mentor), des (mentor)
Differential Revision: https://reviews.freebsd.org/D50160

show more ...

release: Sort keys in generate-ucl.lua

When writing out the UCL file containing a package's metadata, metadata
values consisting of key-value pairs were not sorted. For instance, in
the certctl pac

release: Sort keys in generate-ucl.lua

When writing out the UCL file containing a package's metadata, metadata
values consisting of key-value pairs were not sorted. For instance, in
the certctl package we could have either

"deps": {
"FreeBSD-openssl": {
...
},
"FreeBSD-caroot": {
...
},
}

or

"deps": {
"FreeBSD-caroot": {
...
},
"FreeBSD-openssl": {
...
},
}

This breaks reproducibility.

Use the undocumented third parameter to libucl's to_format() to request
recursive sorting of keys.

Reviewed by: manu, bapt
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D51021

show more ...

pkgbase: rework certctl package to only run rehash on the main package

Rework how ucl manifest are generated leveraging ucl features and flua

now the ucl generation is done via a lua script which u

pkgbase: rework certctl package to only run rehash on the main package

Rework how ucl manifest are generated leveraging ucl features and flua

now the ucl generation is done via a lua script which uses libucl to
ingest the template and use variables as defined in its command line.

the template will include only if it exist a ucl file named after the
package name which will complement the template or overwrite what was
defined in the template if defined in this specific ucl file

this allows to overwrite license, but add script only to the packages
who actually needs them.

As a results the post install scripts are now only added to the right
package and not also added to the subpackages like -man or -dev

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D44374

show more ...

generate-ucl.lua: Add missing copyright header

Add a standard BSD license with a copyright claim for everyone who has
modified this file since it was added, excluding users who explicitly
declined a

generate-ucl.lua: Add missing copyright header

Add a standard BSD license with a copyright claim for everyone who has
modified this file since it was added, excluding users who explicitly
declined attribution.

MFC after: 3 days
Reviewed by: ifreund_freebsdfoundation.org, imp, bapt, markj, emaste
Differential Revision: https://reviews.freebsd.org/D52881

show more ...

packages: Set dependency origin in generate-ucl.lua

Commit 822fd5606748 changed the origin of packages from "base"
to "base/<package name>", but the old origin was still used for
dependencies.

Asid

packages: Set dependency origin in generate-ucl.lua

Commit 822fd5606748 changed the origin of packages from "base"
to "base/<package name>", but the old origin was still used for
dependencies.

Aside from being wrong, this confuses pkg <= 2.3.1 and causes
"pkg info -d" to show the dependencies as "<pkgname>-(null)"
instead of the correct version string.

Remove the explicit origin from dependencies in the UCL files,
and instead populate this during the build in generate-ucl.lua.
In the (very unlikely) case that an explicit origin is still
needed, don't overwrite one if it's already present.

MFC after: 1 day
Fixes: 822fd5606748 ("pkgbase: set origin for each packages as base/FreeBSD-*")
Reported by: bapt
Reviewed by: cperciva, bapt
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D52892

show more ...

release: packages: Fix typos

Reviewed by: cperciva
Approved by: emaste (mentor)
MFC after: 1 day
Differential Revision: https://reviews.freebsd.org/D52917

packages: Add minimal-jail set

This is minimal, but without bootloader, hardware and networking support
that's typically not required in jails.

This requires extending the 'set' annotation to be a

packages: Add minimal-jail set

This is minimal, but without bootloader, hardware and networking support
that's typically not required in jails.

This requires extending the 'set' annotation to be a comma-separated
list, so that packages can be in multiple sets.

MFC after: 3 seconds
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D52591

show more ...

packages: remove tests from base set

Create a new tests package set to hold the tests, which are not useful
to all users and don't belong in the base set.

Reviewed by: emaste, ivy
Sponsored by: The

packages: remove tests from base set

Create a new tests package set to hold the tests, which are not useful
to all users and don't belong in the base set.

Reviewed by: emaste, ivy
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D52574

show more ...

packages: Add meta-package sets

Add several metapackages which contain no files themselves, but depend
on other packages. This allows the user to install a system by using
a set like "FreeBSD-set-m

packages: Add meta-package sets

Add several metapackages which contain no files themselves, but depend
on other packages. This allows the user to install a system by using
a set like "FreeBSD-set-minimal" instead of having to know which
specific packages are required, and it means if we add more package
in the future, existing installations will get them automatically as
long as they were installed using a set.

The defined sets are significantly less granular than the actual
packages; the assumption is that users who want a very specific set of
packages installed can do that manually, while sets are intended more
for typical users who just want to install FreeBSD.

The following sets are defined:

* minimal: the set of packages required to boot the system and bring
up a multi-user UNIX system. This includes hardware support,
networking (wireless, DHCP), basic functionality like syslogd, cron
and periodic. It does not include a kernel, because the kernels are
quite large and the user might want to use a custom kernel.

sendmail and DMA are not included in minimal. Since we provide two
MTAs, the user should select which one they want, or install one from
ports. sshd is not included because it's not required for the base
system, and the user might want to install it from ports.

Some other significant subsystems (e.g. NFS, Kerberos) are also not
included.

The minimal set does not include any libraries itself, since we rely
on dependencies to pull these in. This implies it doesn't include
library manpages, which is fine, since users won't want manpages for
libraries unless they've also installed the development packages.

* devel: the compiler toolchain for building software, along with all
"-dev" packages which include header files and static libraries.

* lib32: 32-bit compatibility packages. This includes both runtime
and development packages.

* kernels: the kernel packages. For freebsd.org builds, this will be
GENERIC, GENERIC-DEBUG, etc., for custom builds it will be whatever
was set in $KERNCONF.

* base: everything else; this includes the entire base system.

For each set, also generate a <setname>-dbg set containing the debug
packages for the set.

The sets are built along with the rest of the packages using a new
script called create-sets.sh, which examines the "set" annotation in
each package and puts it in the appropriate set. This is in
anticipation of the later "groups" feature appearing in pkg itself, at
which point we can simply replace the set annotation with the group.

MFC after: 3 days
Reviewed by: imp, bapt
Differential Revision: https://reviews.freebsd.org/D52412

show more ...

packages: Improve handling of -lib packages

For some packages (OpenSSL, Kerberos) we want to ship runtime libraries
in a separate package, e.g. openssl and openssl-lib. Currently this is
done using

packages: Improve handling of -lib packages

For some packages (OpenSSL, Kerberos) we want to ship runtime libraries
in a separate package, e.g. openssl and openssl-lib. Currently this is
done using PACKAGE=openssl-lib, but that creates packages with strange
names like openssl-lib-lib32.

Instead, add a new LIB_PACKAGE option to bsd.lib.mk that causes runtime
libraries to be placed in a new -lib subpackage. This significantly
improves the set of packages we create; for example, OpenSSL goes from:

FreeBSD-openssl
FreeBSD-openssl-dbg
FreeBSD-openssl-lib
FreeBSD-openssl-lib-dbg
FreeBSD-openssl-lib-dbg-lib32
FreeBSD-openssl-lib-dev
FreeBSD-openssl-lib-dev-lib32
FreeBSD-openssl-lib-lib32
FreeBSD-openssl-lib-man
FreeBSD-openssl-man

to:

FreeBSD-openssl
FreeBSD-openssl-dbg
FreeBSD-openssl-dbg-lib32
FreeBSD-openssl-dev
FreeBSD-openssl-dev-lib32
FreeBSD-openssl-lib
FreeBSD-openssl-lib32
FreeBSD-openssl-man

While here, move /usr/bin/krb5-config and /usr/bin/compile_et into
the kerberos-dev package.

Reviewed by: des
Differential Revision: https://reviews.freebsd.org/D51925

show more ...

share/man: Move many manpages to more correct packages

At the moment, all the manpages in share/man are in the utilities-man
package. Move some of them to the package they should actually be in,
us

share/man: Move many manpages to more correct packages

At the moment, all the manpages in share/man are in the utilities-man
package. Move some of them to the package they should actually be in,
using the new MANGROUPS feature.

Move all of section 3 to clibs-man. Although some of these are from
/usr/include/sys rather than libc, you can't practically use them
without libc, and clibs-man is where the libc manpages live already.

Move all of sections 4 and 9 to a new kernel-man package, except for
atf-test-case.4 which goes to tests-man. atf-test-case.4 is in the
wrong section, but this needs to be fixed upstream.

kernel-man requires special handling in generate-ucl.lua since it's
got a -man suffix but doesn't want the ' (manual pages)' automatic
suffix. For now, fix this by adding a list of packages that don't
get automatic suffixes.

Reviewed by: ifreund_freebsdfoundation.org, manu, emaste
Differential Revision: https://reviews.freebsd.org/D51504

show more ...

packages: handle suffixes in generate-ucl.lua

Move handling of comment/desc suffixes from generate-ucl.sh to
generate-ucl.lua.

Reviewed by: des, bapt
Approved by: des (mentor)
Differential Revision

packages: handle suffixes in generate-ucl.lua

Move handling of comment/desc suffixes from generate-ucl.sh to
generate-ucl.lua.

Reviewed by: des, bapt
Approved by: des (mentor)
Differential Revision: https://reviews.freebsd.org/D50284

show more ...

packages: handle dependencies in ucl

For packages with specific dependencies, add the dependencies in their
UCL files instead of listing them in generate-ucl.sh.

generate-ucl.sh has logic to add au

packages: handle dependencies in ucl

For packages with specific dependencies, add the dependencies in their
UCL files instead of listing them in generate-ucl.sh.

generate-ucl.sh has logic to add automatic subpackage dependencies, so
e.g. "FreeBSD-foo-dev" depends on "FreeBSD-foo". Move this into
generate-ucl.lua instead.

This means we no longer need to pre-process the UCL in generate-ucl.sh,
and we can undo the hack where it calls generate-ucl.lua with the same
filename as both input and output file.

Reviewed by: des, bapt
Approved by: des (mentor)
Differential Revision: https://reviews.freebsd.org/D50283

show more ...

packages: move ucl files to a subdirectory

We want to have a UCL file for every package, to set fields like
comment, desc, license, etc.; to avoid cluttering release/packages with
~200 new files, mo

packages: move ucl files to a subdirectory

We want to have a UCL file for every package, to set fields like
comment, desc, license, etc.; to avoid cluttering release/packages with
~200 new files, move these to the ucl/ directory.

Add a better description for the yp package to demonstrate how this will
work in practice.

Clean up the handling of the package comment suffix for generated
packages (dev, lib32, etc.) and do the UCL modifications in
generate-ucl.lua, so we don't overwrite the ucl's comment.

Add a long-form suffix to the description for generated packages to
explain what they do.

Eventually, we should require the ucl file exists (try=false) to prevent
people accidentally creating new packages with a typo, but for now, keep
the ucl optional.

Reviewed by: kevans, des, bapt, emaste
Approved by: kevans (mentor), des (mentor)
Differential Revision: https://reviews.freebsd.org/D50160

show more ...

release: Sort keys in generate-ucl.lua

When writing out the UCL file containing a package's metadata, metadata
values consisting of key-value pairs were not sorted. For instance, in
the certctl pac

release: Sort keys in generate-ucl.lua

When writing out the UCL file containing a package's metadata, metadata
values consisting of key-value pairs were not sorted. For instance, in
the certctl package we could have either

"deps": {
"FreeBSD-openssl": {
...
},
"FreeBSD-caroot": {
...
},
}

or

"deps": {
"FreeBSD-caroot": {
...
},
"FreeBSD-openssl": {
...
},
}

This breaks reproducibility.

Use the undocumented third parameter to libucl's to_format() to request
recursive sorting of keys.

Reviewed by: manu, bapt
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D51021

show more ...