arm64: bus: Add 36-bit address mask for use in bus space allocations

Obtained from: Juniper Networks, Inc.
MFC after: 1 week
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D44764

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 wh

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

show more ...

arm64: bus: provide bus_space_set_{multi,region}_stream definitions

Reviewed by: andrew
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.

arm64: bus: provide bus_space_set_{multi,region}_stream definitions

Reviewed by: andrew
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36719

show more ...

arm64: bus: unhide bus_space definition with sanitizers included

We'll only be redefining the various bus_* macros, not the definition of
struct bus_space.

Reviewed by: andrew
Sponsored by: Juniper

arm64: bus: unhide bus_space definition with sanitizers included

We'll only be redefining the various bus_* macros, not the definition of
struct bus_space.

Reviewed by: andrew
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36718

show more ...

Fix the sanitizer header checks on arm64

We need to not include the MI _san.h files when builing some parts of
the kernel. Fix these checks in the arm64 header files.

Sponsored by: The FreeBSD Foun

Fix the sanitizer header checks on arm64

We need to not include the MI _san.h files when builing some parts of
the kernel. Fix these checks in the arm64 header files.

Sponsored by: The FreeBSD Foundation

show more ...

Simplify kernel sanitizer interceptors

KASAN and KCSAN implement interceptors for various primitive operations
that are not instrumented by the compiler. KMSAN requires them as well.
Rather than ad

Simplify kernel sanitizer interceptors

KASAN and KCSAN implement interceptors for various primitive operations
that are not instrumented by the compiler. KMSAN requires them as well.
Rather than adding new cases for each sanitizer which requires
interceptors, implement the following protocol:
- When interceptor definitions are required, define
SAN_NEEDS_INTERCEPTORS and SANITIZER_INTERCEPTOR_PREFIX.
- In headers that declare functions which need to be intercepted by a
sanitizer runtime, use SANITIZER_INTERCEPTOR_PREFIX to provide
declarations.
- When SAN_RUNTIME is defined, do not redefine the names of intercepted
functions. This is typically the case in files which implement
sanitizer runtimes but is also needed in, for example, files which
define ifunc selectors for intercepted operations.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

show more ...

Add a way to map arm64 non-posted device memory

On arm64 we currently use a non-posted write for device memory, however
we should move to use posted writes. This is expected to work on most
hardware

Add a way to map arm64 non-posted device memory

On arm64 we currently use a non-posted write for device memory, however
we should move to use posted writes. This is expected to work on most
hardware, however we will need to support a non-posted option for some
broken hardware.

Reviewed by: imp, manu, bcr (manpage)
Differential Revision: https://reviews.freebsd.org/D29722

show more ...

Clean up the style in the arm64 bus.h

MFC after: 2 weeks
Sponsored by: Innovate UK

Generalize bus_space(9) and atomic(9) sanitizer interceptors

Make it easy to define interceptors for new sanitizer runtimes, rather
than assuming KCSAN. Lay a bit of groundwork for KASAN and KMSAN.

Generalize bus_space(9) and atomic(9) sanitizer interceptors

Make it easy to define interceptors for new sanitizer runtimes, rather
than assuming KCSAN. Lay a bit of groundwork for KASAN and KMSAN.

When a sanitizer is compiled in, atomic(9) and bus_space(9) definitions
in atomic_san.h are used by default instead of the inline
implementations in the platform's atomic.h. These definitions are
implemented in the sanitizer runtime, which includes
machine/{atomic,bus}.h with SAN_RUNTIME defined to pull in the actual
implementations.

No functional change intended.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

show more ...

Rename _cscan_atomic.h and _cscan_bus.h to atomic_san.h and bus_san.h

Other kernel sanitizers (KMSAN, KASAN) require interceptors as well, so
put these in a more generic place as a step towards impo

Rename _cscan_atomic.h and _cscan_bus.h to atomic_san.h and bus_san.h

Other kernel sanitizers (KMSAN, KASAN) require interceptors as well, so
put these in a more generic place as a step towards importing the other
sanitizers.

No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29103

show more ...

Add NetBSD compatible bus_space_peek_N() and bus_space_poke_N() functions.
One problem with the bus_space_read_N() and bus_space_write_N() family of
functions is that they provide no protection again

Add NetBSD compatible bus_space_peek_N() and bus_space_poke_N() functions.
One problem with the bus_space_read_N() and bus_space_write_N() family of
functions is that they provide no protection against exceptions which can
occur when no physical hardware or device responds to the read or write
cycles. In such a situation, the system typically would panic due to a
kernel-mode bus error. The bus_space_peek_N() and bus_space_poke_N() family
of functions provide a mechanism to handle these exceptions gracefully
without the risk of crashing the system.

Typical example is access to PCI(e) configuration space in bus enumeration
function on badly implemented PCI(e) root complexes (RK3399 or Neoverse
N1 N1SDP and/or access to PCI(e) register when device is in deep sleep state.

This commit adds a real implementation for arm64 only. The remaining
architectures have bus_space_peek()/bus_space_poke() emulated by using
bus_space_read()/bus_space_write() (without exception handling).

MFC after: 1 month
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D25371

show more ...

arm64: clean up empty lines in .c and .h files

Add genet driver for Raspberry Pi 4B Ethernet

Add driver for Broadcom "GENET" version 5, as found in BCM-2711 on
Raspberry Pi 4B. The driver is derived in part from the bcmgenet.c
driver in NetBSD,

Add genet driver for Raspberry Pi 4B Ethernet

Add driver for Broadcom "GENET" version 5, as found in BCM-2711 on
Raspberry Pi 4B. The driver is derived in part from the bcmgenet.c
driver in NetBSD, along with bcmgenetreg.h.

Reviewed by: manu
Obtained from: in part from NetBSD
Relnotes: yes, note addition
Differential Revision: https://reviews.freebsd.org/D24436

show more ...

Port the NetBSD KCSAN runtime to FreeBSD.

Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in
the FreeBSD kernel. It is a useful tool for finding data races between
threads exe

Port the NetBSD KCSAN runtime to FreeBSD.

Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in
the FreeBSD kernel. It is a useful tool for finding data races between
threads executing on different CPUs.

This can be enabled by enabling KCSAN in the kernel config, or by using the
GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later
needs a compiler change to allow -fsanitize=thread that KCSAN uses.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22315

show more ...

Fix the definition of bus_space_read_stream_8 on arm64.

This is currently unused, however will be when the Kernel Concurrency
Sanitizer (KCSAN) is imported from NetBSD.

Sponsored by: DARPA, AFRL

arm64: bus: Add 36-bit address mask for use in bus space allocations

Obtained from: Juniper Networks, Inc.
MFC after: 1 week
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D44764

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 wh

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

show more ...

arm64: bus: provide bus_space_set_{multi,region}_stream definitions

Reviewed by: andrew
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.

arm64: bus: provide bus_space_set_{multi,region}_stream definitions

Reviewed by: andrew
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36719

show more ...

arm64: bus: unhide bus_space definition with sanitizers included

We'll only be redefining the various bus_* macros, not the definition of
struct bus_space.

Reviewed by: andrew
Sponsored by: Juniper

arm64: bus: unhide bus_space definition with sanitizers included

We'll only be redefining the various bus_* macros, not the definition of
struct bus_space.

Reviewed by: andrew
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36718

show more ...

Fix the sanitizer header checks on arm64

We need to not include the MI _san.h files when builing some parts of
the kernel. Fix these checks in the arm64 header files.

Sponsored by: The FreeBSD Foun

Fix the sanitizer header checks on arm64

We need to not include the MI _san.h files when builing some parts of
the kernel. Fix these checks in the arm64 header files.

Sponsored by: The FreeBSD Foundation

show more ...

Simplify kernel sanitizer interceptors

KASAN and KCSAN implement interceptors for various primitive operations
that are not instrumented by the compiler. KMSAN requires them as well.
Rather than ad

Simplify kernel sanitizer interceptors

KASAN and KCSAN implement interceptors for various primitive operations
that are not instrumented by the compiler. KMSAN requires them as well.
Rather than adding new cases for each sanitizer which requires
interceptors, implement the following protocol:
- When interceptor definitions are required, define
SAN_NEEDS_INTERCEPTORS and SANITIZER_INTERCEPTOR_PREFIX.
- In headers that declare functions which need to be intercepted by a
sanitizer runtime, use SANITIZER_INTERCEPTOR_PREFIX to provide
declarations.
- When SAN_RUNTIME is defined, do not redefine the names of intercepted
functions. This is typically the case in files which implement
sanitizer runtimes but is also needed in, for example, files which
define ifunc selectors for intercepted operations.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

show more ...

Add a way to map arm64 non-posted device memory

On arm64 we currently use a non-posted write for device memory, however
we should move to use posted writes. This is expected to work on most
hardware

Add a way to map arm64 non-posted device memory

On arm64 we currently use a non-posted write for device memory, however
we should move to use posted writes. This is expected to work on most
hardware, however we will need to support a non-posted option for some
broken hardware.

Reviewed by: imp, manu, bcr (manpage)
Differential Revision: https://reviews.freebsd.org/D29722

show more ...

Clean up the style in the arm64 bus.h

MFC after: 2 weeks
Sponsored by: Innovate UK