makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Dependin

makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Depending on how the image is built, the fsnode structures
are initialized in the walk_dir or read_mtree functions.
However, read_mtree fails to take timestamps passed by -T into account,
leading to nonreproducible images in backends that do not check for -T.
Fix this and make -T backend-agnostic by adding an appropriate check
in read_mtree_keywords while making sure that mtree entries can
still override -T timestamps.

PR: 285630
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Reviewed by: markj, emaste
Differential Revision: https://reviews.freebsd.org/D49531

show more ...

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order of
the sibling nodes is determined by the order they're returned by
readdir(), which isn't very reproducible (e.g., depends on filesystem,
build parallelism).

Add a routine which sorts entries within a directory after its contents
have been read. This is a bit more expensive, but I wasn't able to
measure a significant runtime cost (and I don't think makefs has been
optimized very much to begin with), and we avoid this cost in mtree mode
anyway. This fixes some sources of reproducibility problems.

In mtree mode, for now we let the ordering of METALOG entries determine
the ordering in the fsnode tree. It might be worth sorting these too,
since with parallel installworld they won't have a consistent ordering,
and single-threaded installworld is pretty slow.

Reviewed by: emaste
MFC after: 1 month
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D50197

show more ...

makefs: Support st_flags even on Linux

Linux's struct stat does not have an st_flags member, but we still want
to be able to create FreeBSD disk images with non-zero st_flags. Add a
replacement memb

makefs: Support st_flags even on Linux

Linux's struct stat does not have an st_flags member, but we still want
to be able to create FreeBSD disk images with non-zero st_flags. Add a
replacement member to struct fsinode to store it on Linux.

Note that we add a fallback definition of HAVE_STRUCT_STAT_ST_FLAGS for
makefs.h since etdump reaches into makefs and won't have it defined in
its CFLAGS like makefs itself does, and on FreeBSD we don't provide an
nbtool_config.h.

Note also that strtofflags is currently a stub on Linux so this doesn't
yet properly mirror the mtree, but that will change in a future commit.

Reviewed by: emaste, markj
Differential Revision: https://reviews.freebsd.org/D50078

show more ...

makefs: Remove dead code in inode_type()

No functional change intended.

MFC after: 1 week

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Dependin

makefs: Honor -T timestamps when creating images from mtree manifests

makefs backends rely on the fsnode structure to derive most
of the information about the underlying filesystem objects.
Depending on how the image is built, the fsnode structures
are initialized in the walk_dir or read_mtree functions.
However, read_mtree fails to take timestamps passed by -T into account,
leading to nonreproducible images in backends that do not check for -T.
Fix this and make -T backend-agnostic by adding an appropriate check
in read_mtree_keywords while making sure that mtree entries can
still override -T timestamps.

PR: 285630
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Reviewed by: markj, emaste
Differential Revision: https://reviews.freebsd.org/D49531

show more ...

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order

makefs: Make sure that directory entry order is consistent

When walking a directory hierarchy (as opposed to reading an mtree),
makefs builds up a tree of file nodes. Within a directory, the order of
the sibling nodes is determined by the order they're returned by
readdir(), which isn't very reproducible (e.g., depends on filesystem,
build parallelism).

Add a routine which sorts entries within a directory after its contents
have been read. This is a bit more expensive, but I wasn't able to
measure a significant runtime cost (and I don't think makefs has been
optimized very much to begin with), and we avoid this cost in mtree mode
anyway. This fixes some sources of reproducibility problems.

In mtree mode, for now we let the ordering of METALOG entries determine
the ordering in the fsnode tree. It might be worth sorting these too,
since with parallel installworld they won't have a consistent ordering,
and single-threaded installworld is pretty slow.

Reviewed by: emaste
MFC after: 1 month
Sponsored by: Klara, Inc.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D50197

show more ...

makefs: Support st_flags even on Linux

Linux's struct stat does not have an st_flags member, but we still want
to be able to create FreeBSD disk images with non-zero st_flags. Add a
replacement memb

makefs: Support st_flags even on Linux

Linux's struct stat does not have an st_flags member, but we still want
to be able to create FreeBSD disk images with non-zero st_flags. Add a
replacement member to struct fsinode to store it on Linux.

Note that we add a fallback definition of HAVE_STRUCT_STAT_ST_FLAGS for
makefs.h since etdump reaches into makefs and won't have it defined in
its CFLAGS like makefs itself does, and on FreeBSD we don't provide an
nbtool_config.h.

Note also that strtofflags is currently a stub on Linux so this doesn't
yet properly mirror the mtree, but that will change in a future commit.

Reviewed by: emaste, markj
Differential Revision: https://reviews.freebsd.org/D50078

show more ...

makefs: Remove dead code in inode_type()

No functional change intended.

MFC after: 1 week

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

show more ...

Merge ^/head r317808 through r317970.

makefs: further size_t warning cleanup (missing from r317944)

Sponsored by: The FreeBSD Foundation

makefs: cast snprintf return value to size_t to clear warning

Sponsored by: The FreeBSD Foundation

makefs: use emalloc and friends

The emalloc set of error-checking memory allocation routines were added
to libnetbsd in r316572. Use them in makefs to reduce differences with
NetBSD.

NetBSD revs:
c

makefs: use emalloc and friends

The emalloc set of error-checking memory allocation routines were added
to libnetbsd in r316572. Use them in makefs to reduce differences with
NetBSD.

NetBSD revs:
cd9660.c 1.39
ffs.c 1.56
makefs.c 1.42
walk.c 1.27
cd9660/cd9660_archimedes.c 1.2
cd9660/cd9660_eltorito.c 1.20
cd9660/cd9660_write.c 1.16
cd9660/iso9660_rrip.c 1.12
ffs/buf.c 1.17
ffs/mkfs.c 1.26

Obtained from: NetBSD

show more ...

makefs: sync with NetBSD

This is a collection of minor changes as diff reduction against NetBSD.

NetBSD revs:
cd9660.c 1.39
cd9660.h 1.19
makefs.c 1.34

Obtained from: NetBSD
Sponsored by: The F

makefs: sync with NetBSD

This is a collection of minor changes as diff reduction against NetBSD.

NetBSD revs:
cd9660.c 1.39
cd9660.h 1.19
makefs.c 1.34

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation

show more ...

makefs: free specfile structure once it has been applied

NetBSD revs:
walk.c 1.21

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation

Merge ^/head r307736 through r308146.

Be more precise when including headers so that we're less likely to
depend on namespace pollution and as such become more portable. This
means including headers like <sys/types.h> or <stdlib.h>, but

Be more precise when including headers so that we're less likely to
depend on namespace pollution and as such become more portable. This
means including headers like <sys/types.h> or <stdlib.h>, but also
making sure we include system/host headers before local headers.

While here: define ENOATTR as ENOMSG in mtree.c. There is no ENOATTR
on Linux.

With this, makefs is ready for compilation on macOS and Linux.

show more ...

makefs: Provide a -T option to set timestamps to a consistent value

This is taken from the NetBSD versions listed below and adapted to the
makefs version in FreeBSD, along with a bug fix from cem@ t

makefs: Provide a -T option to set timestamps to a consistent value

This is taken from the NetBSD versions listed below and adapted to the
makefs version in FreeBSD, along with a bug fix from cem@ that will be
sent to NetBSD.

Reviewed by: pfg
Approved by: re (gjb)
Obtained from: NetBSD
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D6835

show more ...

Merge from head

IFC @ r254014

IFC @253398