cross-build make.py: Permit specifying cross toolchains by a package name

Add a --cross-toolchain option whose value is a package name. This is
currently only supported with brew on macOS. A sampl

cross-build make.py: Permit specifying cross toolchains by a package name

Add a --cross-toolchain option whose value is a package name. This is
currently only supported with brew on macOS. A sample invocation is
`--cross-toolchain=llvm@19`.

Differential Revision: https://reviews.freebsd.org/D51741

show more ...

cross-build make.py: Handle lld brew packages on macOS

Homebrew uses separate packages for llvm and lld starting with llvm19.
To handle this case automatically, pass the binary_name to
default_cross

cross-build make.py: Handle lld brew packages on macOS

Homebrew uses separate packages for llvm and lld starting with llvm19.
To handle this case automatically, pass the binary_name to
default_cross_toolchain() and only use the automatic path for a given
binary if the binary is found in the path (not just if the directory
exists). Add a second clause for ld.lld that looks for a lld package
if lld wasn't found in the llvm directory.

Note that manually specifying the path cannot use a single
--cross-bindir option. Instead, XLD must be manually set in addition
to --cross-bindir when requesting a specific version.

Reviewed by: arichardson
Differential Revision: https://reviews.freebsd.org/D51653

show more ...

tools/build/make.py: Minor typo fix

If the object directory prefix does not exist, make.py previously exited
with a message indicating that the chosen prefix does not exit.

Reviewed by: emaste
Pull

tools/build/make.py: Minor typo fix

If the object directory prefix does not exist, make.py previously exited
with a message indicating that the chosen prefix does not exit.

Reviewed by: emaste
Pull request: https://github.com/freebsd/freebsd-src/pull/1348

show more ...

tools/build/make.py: Avoid Python 3.7+ subprocess.run capture_output

This is just a convenient alias for setting stdout and stderr to PIPE,
so substitute it for that to be compatible with Python 3.6

tools/build/make.py: Avoid Python 3.7+ subprocess.run capture_output

This is just a convenient alias for setting stdout and stderr to PIPE,
so substitute it for that to be compatible with Python 3.6.

Fixes: 69cfdc81ea7b ("tools/build/make.py: Keep bootstrapped bmake binary up-to-date")

show more ...

tools/build/make.py: Add missing comma to fix tinderbox and worlds

The missing comma meant this was interpreted as a single target called
"tinderboxworlds", and so neither tinderbox nor worlds were

tools/build/make.py: Add missing comma to fix tinderbox and worlds

The missing comma meant this was interpreted as a single target called
"tinderboxworlds", and so neither tinderbox nor worlds were recognised
as being MI targets (i.e. still required TARGET(_ARCH) to be given).

Fixes: 5157b451c654 ("tools/build/make.py: Grow the list of MI targets")

show more ...

tools/build/make.py: Make --with-default-sys-path mirror usr.bin/bmake

The top-level Makefile passes -m to its sub-makes in order to ensure
they use the in-tree mk files in share/mk, but the top-lev

tools/build/make.py: Make --with-default-sys-path mirror usr.bin/bmake

The top-level Makefile passes -m to its sub-makes in order to ensure
they use the in-tree mk files in share/mk, but the top-level make itself
has to rely on whatever environment the bmake used has. For FreeBSD, we
configure the system bmake with .../share/mk:/usr/share/mk, which means
it will pick up src's share/mk whenever run from within the src tree,
but currently for non-FreeBSD we configure our bootstrap bmake only with
bmake's own mk files. This is mostly compatible, with two exceptions:

1. "targets" runs at the top level, but needs TARGET_MACHINE_LIST and
the corresponding MACHINE_ARCH_LIST_${target}, otherwise it will just
print an empty list.

2. "universe" and "universe-toolchain", when run at the top level (i.e.
not via the various wrappers around universe like tinderbox), end up
failing in universe-toolchain itself with:

bmake[1]: "/path/to/freebsd/share/mk/src.sys.obj.mk" line 112: Cannot use MAKEOBJDIR=
Unset MAKEOBJDIR to get default: MAKEOBJDIR='${.CURDIR:S,^${SRCTOP},${OBJTOP},}'

By including .../share/mk in the default sys path like FreeBSD's system
bmake we ensure that we get the in-tree mk files for the top-level make,
not just sub-makes, and avoid such issues.

Note that we cannot (yet) stop using the installed mk files, since the
MAKEOBJDIRPREFIX check in Makefile runs in the object directory and uses
env -i, thereby losing the MAKESYSPATH exported by src.sys.env.mk. Other
such issues may also exist, though are likely rare if so.

Reviewed by: sjg
Differential Revision: https://reviews.freebsd.org/D41544

show more ...

tools/build/make.py: Keep bootstrapped bmake binary up-to-date

We currently assume that any existing bootstrapped bmake binary will
work, but this means it never gets updated as contrib/bmake is, an

tools/build/make.py: Keep bootstrapped bmake binary up-to-date

We currently assume that any existing bootstrapped bmake binary will
work, but this means it never gets updated as contrib/bmake is, and
similarly we won't rebuild it as and when the configure arguments given
to boot-strap change. Whilst the former isn't necessarily a huge problem
given WANT_MAKE_VERSION rarely gets bumped in Makefile, having fewer
variables is a good thing, and so it's easiest if we just always keep it
up-to-date rather than trying to do something similar to what's already
in Makefile (which may or may not be accurate, given updating FreeBSD
gives you an updated bmake, but nothing does so for our bootstrapped
bmake on non-FreeBSD). The latter is more problematic, though, and the
next commit will be changing this configuration.

We thus now add in two checks. The first is to compare MAKE_VERSION
against _MAKE_VERSION from contrib/bmake/VERSION. The second is to
record at bootstrap time the exact configuration used, and compare that
against what we would bootstrap with.

Reviewed by: arichardson, sjg
Differential Revision: https://reviews.freebsd.org/D41556

show more ...

tools/build/make.py: Grow the list of MI targets

tools/build/make.py: Make flake8 clean

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

tools/build/make.py: Drop problematic Linux hack for bmake bootstrap

Since e2eeea75eb8b ("Merge bmake-20201117") missing/sys/cdefs.h has
been present in bmake, and _GNU_SOURCE seems to have been def

tools/build/make.py: Drop problematic Linux hack for bmake bootstrap

Since e2eeea75eb8b ("Merge bmake-20201117") missing/sys/cdefs.h has
been present in bmake, and _GNU_SOURCE seems to have been defined by
config.h for much longer than that, possibly for the entire time OS
cross-build support has been in-tree, so these are obsolete. Moreover,
since 79e02149fcb4 ("Fix dtrace tools bootstrap on non-FreeBSD after
OpenZFS import"), HAVE_STRLCAT and HAVE_STRLCPY have been defined by our
cross-build headers in order to placate DTrace tools (which is not the
right way to solve that problem, but motivates fixing this one). Commit
4fde40d9b540 ("Merge/update to bmake-20230126") changed the strlcpy.c in
bmake from including config.h directly to including make.h, which means
it includes string.h and thus sees these bogus definitions, causing it
to not define the strlcpy compat function on Linux even though it needs
to and thus failing to link. Thus, fix this whole mess by removing the
hack we no longer need.

show more ...

tools/build/make.py: fix cross build on Fedora Linux

Fedora defines shell functions for some commands used by FreeBSD build
scripts. Unortunatelly it makes them behave incorrectly for our purposes.

tools/build/make.py: fix cross build on Fedora Linux

Fedora defines shell functions for some commands used by FreeBSD build
scripts. Unortunatelly it makes them behave incorrectly for our purposes.

For instance 'which which' returns something like:

which ()
{
( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias ...
}

instead of

/usr/bin/which

This patch unsets those functions to restore original/expected behavior

Reviewed by: emaste, imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D36900

show more ...

Allow bootstrapping llvm-tblgen on macOS and Linux

This is needed in order to build various LLVM binutils (e.g. addr2line)
as well as clang/lld/lldb.

Co-authored-by: Jessica Clarke <jrtc27@FreeBSD.

Allow bootstrapping llvm-tblgen on macOS and Linux

This is needed in order to build various LLVM binutils (e.g. addr2line)
as well as clang/lld/lldb.

Co-authored-by: Jessica Clarke <jrtc27@FreeBSD.org>
Test Plan: Compiles on ubuntu 18.04 and macOS 11.4
Reviewed By: dim
Differential Revision: https://reviews.freebsd.org/D31057

show more ...

tools/build/make.py: Avoid calling brew --prefix on macOS unnecessarily

If all the require variables (XCC/XCXX/XCPP/XLD) are already set in the
environment, we don't have to infer a default value fo

tools/build/make.py: Avoid calling brew --prefix on macOS unnecessarily

If all the require variables (XCC/XCXX/XCPP/XLD) are already set in the
environment, we don't have to infer a default value for the cross toolchain
path. This avoids an additional `brew --prefix` call when building with
cheribuild (since it already sets all these variables).

show more ...

tools/build/make.py: Don't call brew --prefix if --cross-bindir is set

Also updated the logic to use subprocess.run() instead of the old
subprocess.getoutput() which also includes stderr and therefo

tools/build/make.py: Don't call brew --prefix if --cross-bindir is set

Also updated the logic to use subprocess.run() instead of the old
subprocess.getoutput() which also includes stderr and therefore
can trigger an exception inside Path().exists().

Reported by: gnn

show more ...

tools/build/make.py: drop workaround for cc --version not being parsed

Previously bsd.compiler.mk was not able to detect the compiler type for
Ubuntu's /usr/bin/cc unless we were invoking the /usr/b

tools/build/make.py: drop workaround for cc --version not being parsed

Previously bsd.compiler.mk was not able to detect the compiler type for
Ubuntu's /usr/bin/cc unless we were invoking the /usr/bin/gcc symlink.
This problem has been fixed by 9c6954329a9285547881ddd60e393b7c55ed30c4
so we can drop the workaround from make.py.

Reviewed By: jrtc27
Differential Revision: https://reviews.freebsd.org/D28323

show more ...

tools/build/make.py: -DNO_CLEAN -> -DWITHOUT_CLEAN

tools/build/make.py: Fix macOS build after a920b9817

If we set STRIPBIN, we also have to set XSTRIPBIN since we otherwise
use the host /usr/bin/strip during buildworld. However, this does not
work o

tools/build/make.py: Fix macOS build after a920b9817

If we set STRIPBIN, we also have to set XSTRIPBIN since we otherwise
use the host /usr/bin/strip during buildworld. However, this does not
work on macOS since /usr/bin/strip doesn't handle ELF binaries.

show more ...

make.py: Also pass STRIPBIN

This is required for cross-building to allow stripping the installed binaries.

Submitted By: Henry Vogt <hv@tue.mpg.de>

GH Actions: Use pre-installed clang packages

Also fix the run by setting up the environment in non-deprecated way.

Always run with --debug to understand better what sort of stuff is happening in
th

GH Actions: Use pre-installed clang packages

Also fix the run by setting up the environment in non-deprecated way.

Always run with --debug to understand better what sort of stuff is happening in
the background. Also split out the bmake bootstrap stage (takes about 31s on
ubuntu, but 1m14 on macOS?)

Drops the dependency on coreutils (realpath, nproc) and thus (?) fixes macOS to
be just as fast (4 logical cores vs 2 physical cores before, go figure.)

Reviewed by: arichardson

show more ...

Revert "When building on Ubuntu bootstrap bmake with bash as the default shell"

This reverts r365950 since the latest bmake update includes fixes for the test
failures that prompted the change.

When building on Ubuntu bootstrap bmake with bash as the default shell

The Ubuntu /bin/sh (dash) removes all environment variables that contain
characters outside the [a-zA-Z0-9_] range and this bre

When building on Ubuntu bootstrap bmake with bash as the default shell

The Ubuntu /bin/sh (dash) removes all environment variables that contain
characters outside the [a-zA-Z0-9_] range and this breaks the bmake tests that
run as part of bootstrapping bmake.
This can be reverted when the bmake tests have been updated.

show more ...

Add a tools/build/make.py script that bootstraps bmake and then runs the build

This makes it possible to compile on non-FreeBSD systems since make will
usually be GNU make there. Even if they includ

Add a tools/build/make.py script that bootstraps bmake and then runs the build

This makes it possible to compile on non-FreeBSD systems since make will
usually be GNU make there. Even if they include bmake, it will often
either be a broken version or too old to build FreeBSD.

This should be the last commit needed to compile FreeBSD on Linux+macOS.
After over two years, I've finally managed to upstream all our local CheriBSD
changes to allow building on Linux (and as a result of being reviewed by more
people they are slightly less ugly than they were before).

It should now be possible to run the following to build on Linux+macOS if you
have LLVM/Clang 10 or newer installed:
MAKEOBJDIRPREFIX=/somewhere ./tools/build/make.py TARGET=amd64 TARGET_ARCH=amd64 buildworld

I have only tested macOS 15, Ubuntu 18.04 and openSUSE Leap, but other Linux
distributions might also work (as long as they ship a recent GLibc and compiler).

Reviewed By: emaste (should be fine to commit to tools/)
Differential Revision: https://reviews.freebsd.org/D16767

show more ...

cross-build make.py: Permit specifying cross toolchains by a package name

Add a --cross-toolchain option whose value is a package name. This is
currently only supported with brew on macOS. A sampl

cross-build make.py: Permit specifying cross toolchains by a package name

Add a --cross-toolchain option whose value is a package name. This is
currently only supported with brew on macOS. A sample invocation is
`--cross-toolchain=llvm@19`.

Differential Revision: https://reviews.freebsd.org/D51741

show more ...