makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Dependin

makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Depending on how the image is built, the fsnode structures
are initialized in the walk_dir or read_mtree functions.
However, read_mtree fails to take timestamps passed by -T into account,
leading to nonreproducible images in backends that do not check for -T.
Fix this and make -T backend-agnostic by adding an appropriate check
in read_mtree_keywords while making sure that mtree entries can
still override -T timestamps.

PR: 285630
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Reviewed by: markj, emaste
Differential Revision: https://reviews.freebsd.org/D49531

show more ...

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order of
the sibling nodes is determined by the order they're returned by
readdir(), which isn't very reproducible (e.g., depends on filesystem,
build parallelism).

Add a routine which sorts entries within a directory after its contents
have been read. This is a bit more expensive, but I wasn't able to
measure a significant runtime cost (and I don't think makefs has been
optimized very much to begin with), and we avoid this cost in mtree mode
anyway. This fixes some sources of reproducibility problems.

In mtree mode, for now we let the ordering of METALOG entries determine
the ordering in the fsnode tree. It might be worth sorting these too,
since with parallel installworld they won't have a consistent ordering,
and single-threaded installworld is pretty slow.

Reviewed by: emaste
MFC after: 1 month
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D50197

show more ...

makefs: Fix a couple of HAVE_STRUCT_STAT_BIRTHTIME checks

These macros are meant to be 0/1 not undefined/1. Currently we don't
actually have nbtool_config.h included for these files so these are
alw

makefs: Fix a couple of HAVE_STRUCT_STAT_BIRTHTIME checks

These macros are meant to be 0/1 not undefined/1. Currently we don't
actually have nbtool_config.h included for these files so these are
always false, but that will change in a future commit and break building
on Linux where they are defined to 0.

Reviewed by: emaste, markj
Differential Revision: https://reviews.freebsd.org/D50077

show more ...

makefs: Make it possible to silence warnings about duplicate paths

When generating a VM image from an installworld mtree manifest, makefs
spits out several thousand warnings about duplicate paths in

makefs: Make it possible to silence warnings about duplicate paths

When generating a VM image from an installworld mtree manifest, makefs
spits out several thousand warnings about duplicate paths in the
manifest. These are harmless and have been around for a long time (see
the phabricator revision for some more details), so let's at least have
a way to make makefs quieter.

Reviewed by: brooks, imp, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43513

show more ...

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

makefs: remove vestigial '?' cases from top-level getopt(3) loop

getopt(3) returns '?' when it encounters a flag not present in the in
the optstring or if a flag is missing its option argument. We

makefs: remove vestigial '?' cases from top-level getopt(3) loop

getopt(3) returns '?' when it encounters a flag not present in the in
the optstring or if a flag is missing its option argument. We can
handle this case with the "default" failure case with no loss of
legibility.

Obtained from: OpenBSD makefs.c 1.22

show more ...

makefs: whitespace cleanup (remove space before tab)

MFC after: 1 week

makefs: Add ZFS support

This allows one to take a staged directory tree and create a file
consisting of a ZFS pool with one or more datasets that contain the
contents of the directory tree. This is

makefs: Add ZFS support

This allows one to take a staged directory tree and create a file
consisting of a ZFS pool with one or more datasets that contain the
contents of the directory tree. This is useful for creating virtual
machine images without using the kernel to create a pool; "zpool create"
requires root privileges and currently is not permitted in jails.
makefs -t zfs also provides reproducible images by using a fixed seed
for pseudo-random number generation, used for generating GUIDs and hash
salts. makefs -t zfs requires relatively little by way of machine
resources.

The "zpool_reguid" rc.conf setting can be used to ask a FreeBSD guest to
generate a unique pool GUID upon first boot.

A small number of pool and dataset properties are supported. The pool
is backed by a single disk vdev. Data is always checksummed using
Fletcher-4, no redundant copies are made, and no compression is used.
The manual page documents supported pool and filesystem properties.

The implementation uses a few pieces of ZFS support from with the boot
loader, especially definitions for various on-disk structures, but is
otherwise standalone and in particular doesn't depend on OpenZFS.

This feature should be treated as experimental for now, i.e., important
data shouldn't be trusted to a makefs-created pool, and the command-line
interface is subject to change.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35248

show more ...

makefs: add msdosfs (FAT) support

Add FAT support to makefs by copying some files from sys/fs/msdosfs/ and
updating others with changes from NetBSD.

The six files copied from sys/fs/msdosfs at r348

makefs: add msdosfs (FAT) support

Add FAT support to makefs by copying some files from sys/fs/msdosfs/ and
updating others with changes from NetBSD.

The six files copied from sys/fs/msdosfs at r348251 and modified are:
denode.h direntry.h fat.h msdosfs_fat.c msdosfs_lookup.c msdosfsmount.h

I would prefer to avoid the duplication, but reluctance to doing so was
expressed in a previous review (D11197); for now copy the files and
revisit in the future.

Submitted by: Siva Mahadevan
Discussed with: cem, imp
MFC after: 1 month
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D16438

show more ...

makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Dependin

makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Depending on how the image is built, the fsnode structures
are initialized in the walk_dir or read_mtree functions.
However, read_mtree fails to take timestamps passed by -T into account,
leading to nonreproducible images in backends that do not check for -T.
Fix this and make -T backend-agnostic by adding an appropriate check
in read_mtree_keywords while making sure that mtree entries can
still override -T timestamps.

PR: 285630
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Reviewed by: markj, emaste
Differential Revision: https://reviews.freebsd.org/D49531

show more ...

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order of
the sibling nodes is determined by the order they're returned by
readdir(), which isn't very reproducible (e.g., depends on filesystem,
build parallelism).

Add a routine which sorts entries within a directory after its contents
have been read. This is a bit more expensive, but I wasn't able to
measure a significant runtime cost (and I don't think makefs has been
optimized very much to begin with), and we avoid this cost in mtree mode
anyway. This fixes some sources of reproducibility problems.

In mtree mode, for now we let the ordering of METALOG entries determine
the ordering in the fsnode tree. It might be worth sorting these too,
since with parallel installworld they won't have a consistent ordering,
and single-threaded installworld is pretty slow.

Reviewed by: emaste
MFC after: 1 month
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D50197

show more ...

makefs: Fix a couple of HAVE_STRUCT_STAT_BIRTHTIME checks

These macros are meant to be 0/1 not undefined/1. Currently we don't
actually have nbtool_config.h included for these files so these are
alw

makefs: Fix a couple of HAVE_STRUCT_STAT_BIRTHTIME checks

These macros are meant to be 0/1 not undefined/1. Currently we don't
actually have nbtool_config.h included for these files so these are
always false, but that will change in a future commit and break building
on Linux where they are defined to 0.

Reviewed by: emaste, markj
Differential Revision: https://reviews.freebsd.org/D50077

show more ...

makefs: Make it possible to silence warnings about duplicate paths

When generating a VM image from an installworld mtree manifest, makefs
spits out several thousand warnings about duplicate paths in

makefs: Make it possible to silence warnings about duplicate paths

When generating a VM image from an installworld mtree manifest, makefs
spits out several thousand warnings about duplicate paths in the
manifest. These are harmless and have been around for a long time (see
the phabricator revision for some more details), so let's at least have
a way to make makefs quieter.

Reviewed by: brooks, imp, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43513

show more ...

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

makefs: remove vestigial '?' cases from top-level getopt(3) loop

getopt(3) returns '?' when it encounters a flag not present in the in
the optstring or if a flag is missing its option argument. We

makefs: remove vestigial '?' cases from top-level getopt(3) loop

getopt(3) returns '?' when it encounters a flag not present in the in
the optstring or if a flag is missing its option argument. We can
handle this case with the "default" failure case with no loss of
legibility.

Obtained from: OpenBSD makefs.c 1.22

show more ...

makefs: whitespace cleanup (remove space before tab)

MFC after: 1 week

makefs: Add ZFS support

This allows one to take a staged directory tree and create a file
consisting of a ZFS pool with one or more datasets that contain the
contents of the directory tree. This is

makefs: Add ZFS support

This allows one to take a staged directory tree and create a file
consisting of a ZFS pool with one or more datasets that contain the
contents of the directory tree. This is useful for creating virtual
machine images without using the kernel to create a pool; "zpool create"
requires root privileges and currently is not permitted in jails.
makefs -t zfs also provides reproducible images by using a fixed seed
for pseudo-random number generation, used for generating GUIDs and hash
salts. makefs -t zfs requires relatively little by way of machine
resources.

The "zpool_reguid" rc.conf setting can be used to ask a FreeBSD guest to
generate a unique pool GUID upon first boot.

A small number of pool and dataset properties are supported. The pool
is backed by a single disk vdev. Data is always checksummed using
Fletcher-4, no redundant copies are made, and no compression is used.
The manual page documents supported pool and filesystem properties.

The implementation uses a few pieces of ZFS support from with the boot
loader, especially definitions for various on-disk structures, but is
otherwise standalone and in particular doesn't depend on OpenZFS.

This feature should be treated as experimental for now, i.e., important
data shouldn't be trusted to a makefs-created pool, and the command-line
interface is subject to change.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35248

show more ...

makefs: add msdosfs (FAT) support

Add FAT support to makefs by copying some files from sys/fs/msdosfs/ and
updating others with changes from NetBSD.

The six files copied from sys/fs/msdosfs at r348

makefs: add msdosfs (FAT) support

Add FAT support to makefs by copying some files from sys/fs/msdosfs/ and
updating others with changes from NetBSD.

The six files copied from sys/fs/msdosfs at r348251 and modified are:
denode.h direntry.h fat.h msdosfs_fat.c msdosfs_lookup.c msdosfsmount.h

I would prefer to avoid the duplication, but reluctance to doing so was
expressed in a previous review (D11197); for now copy the files and
revisit in the future.

Submitted by: Siva Mahadevan
Discussed with: cem, imp
MFC after: 1 month
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D16438

show more ...

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

show more ...

Merge ^/head r318658 through r318963.

makefs: add -O (offset) option

NetBSD revs:
ffs.c 1.60
makefs.8 1.44
makefs.c 1.48
makefs.h 1.33
ffs/buf.c 1.20
ffs/mkfs.c 1.27

Obtained from: NetBSD
Relnotes: Yes
Sponsored by: The FreeBSD Founda

makefs: add -O (offset) option

NetBSD revs:
ffs.c 1.60
makefs.8 1.44
makefs.c 1.48
makefs.h 1.33
ffs/buf.c 1.20
ffs/mkfs.c 1.27

Obtained from: NetBSD
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10780

show more ...

Merge ^/head r317503 through r317807.

makefs: clean up warnings

- make functions and variables static where appropriate
- use const char * where appropriate
- remove unused variables

Sponsored by: The FreeBSD Foundation