STACKALIGN: Reimplement in terms of __align_down

This changes STACKALIGN to be type-preserving when operating on
pointers.

Reviewed by: brooks, kib
Effort: CHERI upstreaming
Sponsored by: AFRL, DA

STACKALIGN: Reimplement in terms of __align_down

This changes STACKALIGN to be type-preserving when operating on
pointers.

Reviewed by: brooks, kib
Effort: CHERI upstreaming
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D54920

show more ...

Add sys/_align.h replacing machine/_align.h

Define _ALIGNBYTES using sizeof(void *) (no functional change on any
existing architecture) which will allow it to work with CHERI were we
must align thin

Add sys/_align.h replacing machine/_align.h

Define _ALIGNBYTES using sizeof(void *) (no functional change on any
existing architecture) which will allow it to work with CHERI were we
must align things up to capability alignment.

In _ALIGN, replace integer manipulation which does not preserve pointer
provenance with a type and provenance preserving builtin. This requires
modest changes in code which assumes _ALIGN returns an integer, but
those are relatively rare.

Reviewed by: kib, markj
Effort: CHERI upstreaming
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D53947

show more ...

pci: cleanup __PCI_REROUTE_INTERRUPTS

This flag was used as a transition for differing pcib implementations.
Today it is defined for all supported architectures, and can be removed.

Reviewed by: im

pci: cleanup __PCI_REROUTE_INTERRUPTS

This flag was used as a transition for differing pcib implementations.
Today it is defined for all supported architectures, and can be removed.

Reviewed by: imp, jhb
Differential Revision: https://reviews.freebsd.org/D48485

show more ...

Centralize and simpify implemention of some VM macros

These macros have substantially identical implementations on each
platform. Use roundup2/rounddown2 for round_page/trunc_page.

This version st

Centralize and simpify implemention of some VM macros

These macros have substantially identical implementations on each
platform. Use roundup2/rounddown2 for round_page/trunc_page.

This version standardizes on not using explicit casts and instead
preserving the original type. A couple of tweaks were required to
make this work.

Reviewed by: brooks, kib, markj
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D48450

show more ...

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

arm64: increase MAXCPU to 1024, following amd64

As in commit 9051987e40c5 for amd64, support up to 1024 CPU cores.
arm64 hardware with more than 256 CPU cores is currently available and
will become

arm64: increase MAXCPU to 1024, following amd64

As in commit 9051987e40c5 for amd64, support up to 1024 CPU cores.
arm64 hardware with more than 256 CPU cores is currently available and
will become increasingly common over FreeBSD 14's lifetime.

PR: 269572
Reviewed by: andrew
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41319

show more ...

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 wh

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

show more ...

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow m

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

show more ...

arm64: Disable per-thread stack-smashing protection in data_abort()

With PERTHREAD_SSP configured, the compiler's stack-smashing protection
uses a per-thread canary value instead of a global value.

arm64: Disable per-thread stack-smashing protection in data_abort()

With PERTHREAD_SSP configured, the compiler's stack-smashing protection
uses a per-thread canary value instead of a global value. The value is
stored in td->td_md.md_canary; the sp_el0 register always contains a
pointer to that value, and certain functions selected by the compiler
will store the canary value on the stack as a part of the function
prologue (and will verify the copy as part of the epilogue). In
particular, the thread structure may be accessed.

This happens to occur in data_abort(), which leads to the same problem
addressed by commit 2c10be9e06d4 ("arm64: Handle translation faults for
thread structures"). This commit fixes that directly, by disabling SSP
in data_abort() and a couple of related functions by using a function
attribute. It also moves the update of sp_el0 out of C code in case
the compiler decides to start checking the canary in pmap_switch()
someday.

A different solution might be to move the canary value to the PCB, which
currently lives on the kernel stack and isn't subject to the same
problem as thread structures (if only because guard pages inhibit
superpage promotion). However, there isn't any particular reason the
PCB has to live on the stack today; on amd64 it is embedded in struct
thread, reintroducing the same problem. Keeping the reference canary
value at the top of the stack is also rather dubious since it could be
clobbered by a sufficiently large stack overflow.

A third solution could be to go back to the approach of commit
5aa5420ff2e8, and modify UMA to use the direct map for thread structures
even if KASAN is enabled. But, transient promotions and demotions in
the direct map are possible too.

Reviewed by: alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37255

show more ...

Simplify setting a non-4k PAGE_SIZE on arm64

Define PAGE_SIZE and PAGE_MASK based on PAGE_SHIFT. With this we only
need to set one value to change one value to change the page size.

While here remo

Simplify setting a non-4k PAGE_SIZE on arm64

Define PAGE_SIZE and PAGE_MASK based on PAGE_SHIFT. With this we only
need to set one value to change one value to change the page size.

While here remove the unused PAGE_MASK_* macros.

Sponsored by: The FreeBSD Foundation

show more ...

arm64: Stop setting VM_BCACHE_SIZE_MAX

This setting places a (small) limit on the size of the buffer cache,
constraining UFS performance on large servers. The setting comes from
the initial arm64 i

arm64: Stop setting VM_BCACHE_SIZE_MAX

This setting places a (small) limit on the size of the buffer cache,
constraining UFS performance on large servers. The setting comes from
the initial arm64 implementation and appears to be vestigal. Remove it.

Reviewed by: kib
Submitted by: Klara, Inc.
Sponsored by: Ampere Computing
Differential Revision: https://reviews.freebsd.org/D28162

show more ...

Rename the arm64 4k PAGE_* macros

These now have a _4K suffix to allow us to be explicit when we mean
to use a 4k page rather than assuming PAGE_SIZE is 4k.

Sponsored by: Innovate UK

[arm64] Bump MAXMEMDOM value to 8 to match amd64

On some of the server-grade ARM64 machines the number of NUMA domains is higher
than 2. When booting GENERIC kernel on such machines the SRAT parser

[arm64] Bump MAXMEMDOM value to 8 to match amd64

On some of the server-grade ARM64 machines the number of NUMA domains is higher
than 2. When booting GENERIC kernel on such machines the SRAT parser fails
leaving the system with a single domain. To make GENERIC kernel usable on those
server, match the parameter value with the one for amd64 arch.

Reviewed by: allanjude
Differential Revision: https://reviews.freebsd.org/D27368
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.

show more ...

Add largepage support to the arm64 pmap.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc., Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26466

Merge ^/head r357408 through r357661.

Define MAXCPU consistently between the kernel and KLDs.

This reverts r177661. The change is no longer very useful since
out-of-tree KLDs will be built to target SMP kernels anyway. Moveover
it bre

Define MAXCPU consistently between the kernel and KLDs.

This reverts r177661. The change is no longer very useful since
out-of-tree KLDs will be built to target SMP kernels anyway. Moveover
it breaks the KBI in !SMP builds since cpuset_t's layout depends on the
value of MAXCPU, and several kernel interfaces, notably
smp_rendezvous_cpus(), take a cpuset_t as a parameter.

PR: 243711
Reviewed by: jhb, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23512

show more ...

Add sys/_align.h replacing machine/_align.h

Define _ALIGNBYTES using sizeof(void *) (no functional change on any
existing architecture) which will allow it to work with CHERI were we
must align thin

Add sys/_align.h replacing machine/_align.h

Define _ALIGNBYTES using sizeof(void *) (no functional change on any
existing architecture) which will allow it to work with CHERI were we
must align things up to capability alignment.

In _ALIGN, replace integer manipulation which does not preserve pointer
provenance with a type and provenance preserving builtin. This requires
modest changes in code which assumes _ALIGN returns an integer, but
those are relatively rare.

Reviewed by: kib, markj
Effort: CHERI upstreaming
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D53947

show more ...

pci: cleanup __PCI_REROUTE_INTERRUPTS

This flag was used as a transition for differing pcib implementations.
Today it is defined for all supported architectures, and can be removed.

Reviewed by: im

pci: cleanup __PCI_REROUTE_INTERRUPTS

This flag was used as a transition for differing pcib implementations.
Today it is defined for all supported architectures, and can be removed.

Reviewed by: imp, jhb
Differential Revision: https://reviews.freebsd.org/D48485

show more ...

Centralize and simpify implemention of some VM macros

These macros have substantially identical implementations on each
platform. Use roundup2/rounddown2 for round_page/trunc_page.

This version st

Centralize and simpify implemention of some VM macros

These macros have substantially identical implementations on each
platform. Use roundup2/rounddown2 for round_page/trunc_page.

This version standardizes on not using explicit casts and instead
preserving the original type. A couple of tweaks were required to
make this work.

Reviewed by: brooks, kib, markj
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D48450

show more ...

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

arm64: increase MAXCPU to 1024, following amd64

As in commit 9051987e40c5 for amd64, support up to 1024 CPU cores.
arm64 hardware with more than 256 CPU cores is currently available and
will become

arm64: increase MAXCPU to 1024, following amd64

As in commit 9051987e40c5 for amd64, support up to 1024 CPU cores.
arm64 hardware with more than 256 CPU cores is currently available and
will become increasingly common over FreeBSD 14's lifetime.

PR: 269572
Reviewed by: andrew
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41319

show more ...