vmimage.subr: skip -dbg sets for WITHOUT_DEBUG_FILES/WITHOUT_KERNEL_SYMBOLS

This matches non-pkgbase behavior as installworld/installkernel skip the
debug bits if these knobs are set.

MFC after: 1

vmimage.subr: skip -dbg sets for WITHOUT_DEBUG_FILES/WITHOUT_KERNEL_SYMBOLS

This matches non-pkgbase behavior as installworld/installkernel skip the
debug bits if these knobs are set.

MFC after: 1 week
Sponsored by: Chelsio Communications
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55572

show more ...

release: Remove not-NO_ROOT cases

We always use NO_ROOT for release artifact builds, so remove the
alternate code paths.

For the first step we set NO_ROOT unconditionally in cases that invoke
subma

release: Remove not-NO_ROOT cases

We always use NO_ROOT for release artifact builds, so remove the
alternate code paths.

For the first step we set NO_ROOT unconditionally in cases that invoke
submakes, and turn NO_ROOT being unset into an error in lover-level
targets so that we can catch potential out-of-tree build scripts (or
missed in-tree cases) that expect to run not-NO_ROOT builds. The second
step will be to remove those entirely.

Reviewed by: cperciva
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D54179

show more ...

vmimage.subr: Install pkg on pkgbase images

When building VM and Cloud images with a packaged base system, install
the pkg package; this makes it possible for the base system to be
updated without i

vmimage.subr: Install pkg on pkgbase images

When building VM and Cloud images with a packaged base system, install
the pkg package; this makes it possible for the base system to be
updated without installing pkg from the FreeBSD-ports repository.

Sponsored by: Amazon

show more ...

release: Add a MISSING_METALOGS hack to VMs

The packages for 15.0-RELEASE built without the bug fix needed to make
files created via @sample get properly listed in METALOG. Fix the
cloudware which

release: Add a MISSING_METALOGS hack to VMs

The packages for 15.0-RELEASE built without the bug fix needed to make
files created via @sample get properly listed in METALOG. Fix the
cloudware which contain @sample-using packages by adding the necessary
files to METALOG manually.

This should be reverted after the next full package build, and live on
only in releng/15.0.

Reviewed by: markj
MFC after: immediately (15.0-RC2)
Differential Revision: https://reviews.freebsd.org/D53797

show more ...

vmimage.subr: Log correct paths into METALOG

Paths all need to start with "./" because that's what newfs wants.

Fixes: e0c41af9256b ("vmimage.subr: Enable FreeBSD-base repo if pkgbase")
MFC after:

vmimage.subr: Log correct paths into METALOG

Paths all need to start with "./" because that's what newfs wants.

Fixes: e0c41af9256b ("vmimage.subr: Enable FreeBSD-base repo if pkgbase")
MFC after: immediately

show more ...

vmimage.subr: Don't log missing files from pkgs

When installing "extra" packages (aka those built from the ports tree),
we record everything being installed in METALOG.pkg; the contents of
that file

vmimage.subr: Don't log missing files from pkgs

When installing "extra" packages (aka those built from the ports tree),
we record everything being installed in METALOG.pkg; the contents of
that file is appended to METALOG before we generate the filesystem.

There are two cases when files recorded in METALOG.pkg will no longer
exist by the time we create the final disk image:
1. If a pkg bug results in false dependencies being installed which
are later removed by "pkg autoremove", and
2. If the pkg we build and install from /usr/ports is older than the
pkg on pkg.freebsd.org, and pkg gets upgraded automatically as part of
installing extra packages.

The ultimate issue in both cases is that there's no mechanism for
removing entries from METALOG when we run 'pkg delete'.

Address this build breakage by checking, line by line, if filesystem
objects mentioned in METALOG.pkg exist before appending them to METALOG.

Fixes: 6a13aeac3c1f ("vmimage.subr: pkg autoremove after pkg install")
MFC after: immediately (needed for 15.0-RC1)

show more ...

vmimage.subr: autoremove -y

Running 'pkg autoremove' without -y results in VM image builds failing
when (bogusly installed) packages are removed.

Fixes: 6a13aeac3c1f ("vmimage.subr: pkg autoremove

vmimage.subr: autoremove -y

Running 'pkg autoremove' without -y results in VM image builds failing
when (bogusly installed) packages are removed.

Fixes: 6a13aeac3c1f ("vmimage.subr: pkg autoremove after pkg install")
MFC after: immediately (needed for 15.0-RC1)

show more ...

vmimage.subr: Enable FreeBSD-base repo if pkgbase

When creating a VM image using pkgbase, create a configuration file in
/usr/local/etc/pkg/repos/FreeBSD.conf which enables the FreeBSD-base
reposito

vmimage.subr: Enable FreeBSD-base repo if pkgbase

When creating a VM image using pkgbase, create a configuration file in
/usr/local/etc/pkg/repos/FreeBSD.conf which enables the FreeBSD-base
repository. (This repository is defined in /etc/pkg/FreeBSD.conf as
being disabled by default.)

Reported by: Mark Millard
MFC after: immediately (needed for 15.0-RC1)

show more ...

vmimage.subr: Log pkg/local.sqlite if pkgbase

We were doing this in vm_extra_install_packages but VM images without
any extra packages installed would not get this installed. This
results in a pkgb

vmimage.subr: Log pkg/local.sqlite if pkgbase

We were doing this in vm_extra_install_packages but VM images without
any extra packages installed would not get this installed. This
results in a pkgbase system which thinks it doesn't have any packages
installed (even though all the files are right there).

Add a "metalog_add_data ./var/db/pkg/local.sqlite" call to the pkgbase
install code path, and make the call from vm_extra_install_packages
conditional on !PKGBASE.

Reported by: Michael Dexter
MFC after: immediately (needed for 15.0-RC1)

show more ...

release: Comply with Azure Marketplace requirements for aarch64

Azure requires the first 1 MB (2,048 sectors) of the OS disk to remain
empty for VM images:

https://learn.microsoft.com/partner-cen

release: Comply with Azure Marketplace requirements for aarch64

Azure requires the first 1 MB (2,048 sectors) of the OS disk to remain
empty for VM images:

https://learn.microsoft.com/partner-center/marketplace-offers/azure-vm-certification-faq#vm-images-must-have-1-mb-of-free-space

Also append the BOOTPARTSOFFSET suffix for aarch64 images, which only
has an ESP partition for booting.

Co-authored-by: Brad Davis <brd@FreeBSD.org>
Approved by: cperciva
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D53628

show more ...

vmimage.subr: pkg autoremove after pkg install

A bug in pkg, which somehow only surfaced as a consequence of pkgbase,
results in pkg install sometimes pulling in false dependencies. This
problem mi

vmimage.subr: pkg autoremove after pkg install

A bug in pkg, which somehow only surfaced as a consequence of pkgbase,
results in pkg install sometimes pulling in false dependencies. This
problem might be limited to cases when the lib32 pkgbase packages are
not installed. In the case of EC2 "small" images, installing the
ebsnvme-id package results in binutils, gcc12-devel, gmp, indexinfo,
liblz4, mpc, mpfr, and zstd packages being installed.

These false dependencies are however not recorded as dependencies --
at some level pkg does understand that they're not needed -- so running
pkg autoremove immediately after pkg install cleans them up.

Note: This does not remove lines from METALOG corresponding to these
packages, and makefs emits an error when it attempts to create the
filesystem but cannot find the files listed in METALOG -- but makefs
does seem to complete normally despite the error messages.

This change should be reverted once the pkg issue has been located and
fixed.

Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53543

show more ...

release: Use PKG_CMD when installing pkgbase packages in vmimage.subr

This defaults to plain "pkg", but being able to override it is useful
when testing pkg itself.

Reviewed by: cperciva
MFC after:

release: Use PKG_CMD when installing pkgbase packages in vmimage.subr

This defaults to plain "pkg", but being able to override it is useful
when testing pkg itself.

Reviewed by: cperciva
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53307

show more ...

vmimage.subr: Fix when/where we fix up METALOG

We only need to check for unMETALOGed directories and sort the METALOG
file if we're using it, i.e. if we're doing a NO_ROOT build. This
non-NO_ROOT b

vmimage.subr: Fix when/where we fix up METALOG

We only need to check for unMETALOGed directories and sort the METALOG
file if we're using it, i.e. if we're doing a NO_ROOT build. This
non-NO_ROOT builds by no longer bogusly writing to /METALOG*.

We only need to add databases (spwd.db etc) to METALOG if we're doing
a pkgbase-enabled NO_ROOT build; but we should always do this before
creating the filesystem, not only if we installed extra packages (in
vm_extra_install_packages, where that code was erroneously placed).
This fixes non-cloud VM images, which in 15.0-BETA2 shipped without
password databases.

Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53194

show more ...

vmimage.subr: Add missing directories to METALOG

Theoretically METALOG should include everything which needs to go
into disk images; unfortunately there are still a few bugs which
are resulting in d

vmimage.subr: Add missing directories to METALOG

Theoretically METALOG should include everything which needs to go
into disk images; unfortunately there are still a few bugs which
are resulting in directories not being listed -- and if METALOG
has files in unrecorded directories, the directories end up being
created with 000 permissions.

Oddly enough, systems where / has 000 permissions are not very
usable.

As a temporary hack, compare the staging tree against METALOG and
add entries for any unrecorded directories. This will hopefully
be reverted before 15.0-RELEASE.

Reviewed by: bapt, emaste, ivy
Sponsored by: https://www.patreon.com/cperciva
MFC after: 5 minutes
Differential Revision: https://reviews.freebsd.org/D53153

show more ...

vmimage.subr: Sort METALOG before running makefs

When creating VM images from pkgbase, the METALOG may not be in order;
in particular, files may be listed before the directories which contain
them.

vmimage.subr: Sort METALOG before running makefs

When creating VM images from pkgbase, the METALOG may not be in order;
in particular, files may be listed before the directories which contain
them. This causes makefs to create directories with 000 permissions.

Interestingly, such VM images boot just fine, since root ignores those
permissions; the first sign of trouble was sshd refusing logins with an
error message which said absolutely nothing about /etc/ having
incorrect permissions or being unable to read files inside it.

Immediately prior to running makefs, sort the METALOG file. While
we're here, uniquify as well; this does not guarantee that we do not
have duplicate paths, but if there are duplicate paths with different
settings something else has gone wrong and we don't really have any
good way of solving the problem anyway.

Reviewed by: ivy
Hint from: imp
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53046

show more ...

vmimage.subr: metalog_add generated databases

The files /etc/login.conf.db, /etc/passwd, /etc/pwd.db, /etc/spwd.db,
and /var/db/services.db are generated at install time. When VM images
are built u

vmimage.subr: metalog_add generated databases

The files /etc/login.conf.db, /etc/passwd, /etc/pwd.db, /etc/spwd.db,
and /var/db/services.db are generated at install time. When VM images
are built using installworld, there is Makefile code which emits the
necessary METALOG lines; but the pkgbase triggers do not do this. This
resulted in VM images which lacked these essential database files.

VM images are currently the only place where this matters, since this
issue is specific to non-root image builds uusing pkgbase. (In some
point in the future we will start building release ISO iamges from
pkgbase, at which point it will be a problem there as well.)

The long-term solution for this is probably to teach the relevant
database-creating tools to emit METALOG lines; but for now, just log
the necessary files from vmimage.subr.

Reviewed by: dch
MFC after: 5 minutes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53034

show more ...

release: Don't install missing lib32 into VMs

There is no FreeBSD-set-lib32 on riscv64 or powerpc64le.

Reviewed by: ivy, emaste
Fixes: 2d184465e8bb ("release: Rework VM package selection using sets

release: Don't install missing lib32 into VMs

There is no FreeBSD-set-lib32 on riscv64 or powerpc64le.

Reviewed by: ivy, emaste
Fixes: 2d184465e8bb ("release: Rework VM package selection using sets")
MFC after: 6 hours
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D52987

show more ...

release: Rework VM package selection using sets

Filtering the list of packages broke with the introduction of package
sets, since we excluded FreeBSD-src.* but still included
FreeBSD-set-src, which

release: Rework VM package selection using sets

Filtering the list of packages broke with the introduction of package
sets, since we excluded FreeBSD-src.* but still included
FreeBSD-set-src, which transitively included the FreeBSD-src and
FreeBSD-src-sys packages. This could be fixed by excluding package
sets, but that would be fragile, potentially breaking upgrades if
new packages are introduced.

Instead, start with an explicit set of package sets:
base, base-dbg
lib32, lib32-dbg
kernels, kernels-dbg
tests
and filter the package sets; the EC2 "small" and "builder" AMIs
filter out everything except "base" and "kernels".

Note that using FreeBSD-set-kernels may pose a problem in the future
if we start shipping packages for multiple differently-configured
kernels. That will be addressed if and when that problem arises.

Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D52922

show more ...

release: Set INSTALL_AS_USER=yes when running pkg clean in NO_ROOT mode

Otherwise pkg complains about unexpected ownership of the root
directory.

MFC after: 1 day
Fixes: 86221f965c06 ("vmimage.sub

release: Set INSTALL_AS_USER=yes when running pkg clean in NO_ROOT mode

Otherwise pkg complains about unexpected ownership of the root
directory.

MFC after: 1 day
Fixes: 86221f965c06 ("vmimage.subr: Fix NO_ROOT build in ..._rmcache")

show more ...

vmimage.subr: fix typo in checking NO_ROOT var

Reviewed by: imp
Fixes: 08b497dc6c4d ("vmimage.subr: Support NO_ROOT pkgbase")
Signed-off-by: Siva Mahadevan <me@svmhdvn.name>
Pull-request: https://gi

vmimage.subr: fix typo in checking NO_ROOT var

Reviewed by: imp
Fixes: 08b497dc6c4d ("vmimage.subr: Support NO_ROOT pkgbase")
Signed-off-by: Siva Mahadevan <me@svmhdvn.name>
Pull-request: https://github.com/freebsd/freebsd-src/pull/1867
Sponsored by: The FreeBSD Foundation

show more ...

vmimage.subr: Support NO_ROOT pkgbase

We need to tell pkg to record files in METALOG; otherwise when we
create the VM image it's almost empty.

MFC after: 6 hours
Sponsored by: https://www.patreon.c

vmimage.subr: Support NO_ROOT pkgbase

We need to tell pkg to record files in METALOG; otherwise when we
create the VM image it's almost empty.

MFC after: 6 hours
Sponsored by: https://www.patreon.com/cperciva

show more ...

vmimage.subr: Fix NO_ROOT build in ..._rmcache

When doing a NO_ROOT build we can't run pkg inside a chroot, for
multiple reasons including not having /dev mounted.

MFC after: 6 hours
Sponsored by:

vmimage.subr: Fix NO_ROOT build in ..._rmcache

When doing a NO_ROOT build we can't run pkg inside a chroot, for
multiple reasons including not having /dev mounted.

MFC after: 6 hours
Sponsored by: https://www.patreon.com/cperciva

show more ...

release: Fix NO_ROOT cross-building VM images

Pass PKG_ABI to virtual machine image builds.

Reviewed by: cperciva
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd

release: Fix NO_ROOT cross-building VM images

Pass PKG_ABI to virtual machine image builds.

Reviewed by: cperciva
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D52874

show more ...

release: create pkgbase VM and cloud images

VM and cloud images are now built as packaged base systems by default,
matching the default for installation media.

Setting -DNOPKGBASE allows building a

release: create pkgbase VM and cloud images

VM and cloud images are now built as packaged base systems by default,
matching the default for installation media.

Setting -DNOPKGBASE allows building as non-pkgbase systems.

Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D51483

show more ...

release: Enable installing packages as a non-root user

Provide variables, PKG_REPOS_DIR and PKG_REPO_NAME, to allow the user to
configure the source package repository. Configure pkg to emit added

release: Enable installing packages as a non-root user

Provide variables, PKG_REPOS_DIR and PKG_REPO_NAME, to allow the user to
configure the source package repository. Configure pkg to emit added
files to a metalog which gets concatenated with the main metalog
immediately before we build the root filesystem.

Reviewed by: cperciva, emaste
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D52453

show more ...