arm64: Teach CHECK_CPU_FEAT to handle more values

CHECK_CPU_FEAT only supported checking for features that were enabled
when the field is non-zero. There are some features we might need to
check whe

arm64: Teach CHECK_CPU_FEAT to handle more values

CHECK_CPU_FEAT only supported checking for features that were enabled
when the field is non-zero. There are some features we might need to
check where we move between two non-zero values.

Support this by passing in the field value name to compare rather than
assuming a non-zero value.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D51375

show more ...

arm64: add CHECK_CPU_FEAT() for checking feature support in assembly

Add a new macro under asm.h to check whether a given CPU feature is
supported. There are a number of existing places where an ID

arm64: add CHECK_CPU_FEAT() for checking feature support in assembly

Add a new macro under asm.h to check whether a given CPU feature is
supported. There are a number of existing places where an ID register is
checked, and these have been updated in this change. These are for GIC
special registers, HAFDBS and HCX.

When calling, pass a temporary registers who's value is not important,
the name of the ID register (minus the exception level), the name of the
feature, and a label to jump to should the feature not be present. The
feature name should match with the macros defined in armreg.h or
hypervisor.h. Any feature-specific instructions can then be placed
between the macro and the label.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D48813
Signed-off-by: Harry Moulton <harry.moulton@arm.com>

show more ...

arm64: Use the PAN msr mnemonic rather than .inst

Switch from creating the msr instructions to manage PAN to use the
"msr pan, #1" instruction directly. When this was added clang didn't
have support

arm64: Use the PAN msr mnemonic rather than .inst

Switch from creating the msr instructions to manage PAN to use the
"msr pan, #1" instruction directly. When this was added clang didn't
have support to assemble the instructions. This appears to have been
added to clang 13 which is sufficiently old enough.

Binutils releases from around the same time appear to have added this
instruction so any modern gcc should also support this instruction.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47817

show more ...

arm64: Add PAC instruction macros

To support Pointer Authentication (PAC) in assembly files add a pair of
macros that sign the link register. When used before storing to the
stack it will allow hard

arm64: Add PAC instruction macros

To support Pointer Authentication (PAC) in assembly files add a pair of
macros that sign the link register. When used before storing to the
stack it will allow hardware to detect if it has changed before using
it in the return instruction.

Reviewed by: markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42226

show more ...

arm64: Support creating a BTI & PAC note

To detect when an object file is build with the Branch Target
Identification (BTI) and Pointer Authentication Code (PAC) extensions
there is an elf note the

arm64: Support creating a BTI & PAC note

To detect when an object file is build with the Branch Target
Identification (BTI) and Pointer Authentication Code (PAC) extensions
there is an elf note the compiler will insert. It will only do so from
a high level language, e.g. C or C++.

To get the not in assembly add the GNU_PROPERTY_AARCH64_FEATURE_1_NOTE
macro that can be used to create it, and the
GNU_PROPERTY_AARCH64_FEATURE_1_VAL macro to insert the correct value
based on which combination of BTI and PAC are enabled.

Reviewed by: markj (earlier version), emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42225

show more ...

arm64: Add BTI landing pads to assembly functions

When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pa

arm64: Add BTI landing pads to assembly functions

When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pad instruction.

There are three options for these instructions:
1. A breakpoint instruction
2. A pointer authentication PACIASP/PACIBSP
3. A BTI instruction

Option 1 will raise a breakpoint exception so isn't useable in either
cases. Option 2 could be used in some function entry cases, but needs
to be paired with an authentication instruction, and is normally only
used in non-leaf functions we can't use it in this case. This leaves
option 3.

There are four variants of the instruction, the C variant is used on
function entry and the J variant is for jumping within a function.
There is also a JC that works with both and one with no target that
works with neither.

Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42078

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 wh

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

show more ...

Mark EENTRY as .text

To allow it to be used before ENTRY we need to ensure the symbol is
in the .text section. It also needs to be aligned correctly.

While here mark the symbol type as a function a

Mark EENTRY as .text

To allow it to be used before ENTRY we need to ensure the symbol is
in the .text section. It also needs to be aligned correctly.

While here mark the symbol type as a function as in the ENTRY macro.

Reported by: jrtc27
Sponsored by: Arm Ltd

show more ...

Handle functions that use a nop in the arm64 fbt

To trace leaf asm functions we can insert a single nop instruction as
the first instruction in a function and trigger off this.

Reviewed by: gnn
Spo

Handle functions that use a nop in the arm64 fbt

To trace leaf asm functions we can insert a single nop instruction as
the first instruction in a function and trigger off this.

Reviewed by: gnn
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D28132

show more ...

Move the literal pool to the end of asm functions

This keeps the data at the end of a function, near to where it's used.
Sponsored by: Innovate UK

Add LENTRY and LEND to arm64

These allow us to mark local asm functions as a function

Sponsored by: Innovate UK

Add CFI start/end proc directives to arm64, i386, and ppc

Follow-up to r353959 and r368070: do the same for other architectures.

arm32 already seems to use its own .fnstart/.fnend directives, which

Add CFI start/end proc directives to arm64, i386, and ppc

Follow-up to r353959 and r368070: do the same for other architectures.

arm32 already seems to use its own .fnstart/.fnend directives, which
appear to be ARM-specific variants of the same thing. Likewise, MIPS
uses .frame directives.

Reviewed by: arichardson
Differential Revision: https://reviews.freebsd.org/D27387

show more ...

Stop speculation past an eret instruction

On arm64 the eret instruction is used to return from an exception handler.
Some implementations may speculate past this instruction into the next
function.

Stop speculation past an eret instruction

On arm64 the eret instruction is used to return from an exception handler.
Some implementations may speculate past this instruction into the next
function. As the user may control many registers in these functions add
a synchronisation barrier sequence after the eret instruction to stop these
CPUs from speculating out of the exception handler.

PR: 242676
Submitted by: Anthony Steinhauser <asteinhauser@google.com> (previous version)
MFC after: 1 week

show more ...

arm64: Teach CHECK_CPU_FEAT to handle more values

CHECK_CPU_FEAT only supported checking for features that were enabled
when the field is non-zero. There are some features we might need to
check whe

arm64: Teach CHECK_CPU_FEAT to handle more values

CHECK_CPU_FEAT only supported checking for features that were enabled
when the field is non-zero. There are some features we might need to
check where we move between two non-zero values.

Support this by passing in the field value name to compare rather than
assuming a non-zero value.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D51375

show more ...

arm64: add CHECK_CPU_FEAT() for checking feature support in assembly

Add a new macro under asm.h to check whether a given CPU feature is
supported. There are a number of existing places where an ID

arm64: add CHECK_CPU_FEAT() for checking feature support in assembly

Add a new macro under asm.h to check whether a given CPU feature is
supported. There are a number of existing places where an ID register is
checked, and these have been updated in this change. These are for GIC
special registers, HAFDBS and HCX.

When calling, pass a temporary registers who's value is not important,
the name of the ID register (minus the exception level), the name of the
feature, and a label to jump to should the feature not be present. The
feature name should match with the macros defined in armreg.h or
hypervisor.h. Any feature-specific instructions can then be placed
between the macro and the label.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D48813
Signed-off-by: Harry Moulton <harry.moulton@arm.com>

show more ...

arm64: Use the PAN msr mnemonic rather than .inst

Switch from creating the msr instructions to manage PAN to use the
"msr pan, #1" instruction directly. When this was added clang didn't
have support

arm64: Use the PAN msr mnemonic rather than .inst

Switch from creating the msr instructions to manage PAN to use the
"msr pan, #1" instruction directly. When this was added clang didn't
have support to assemble the instructions. This appears to have been
added to clang 13 which is sufficiently old enough.

Binutils releases from around the same time appear to have added this
instruction so any modern gcc should also support this instruction.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47817

show more ...

arm64: Add PAC instruction macros

To support Pointer Authentication (PAC) in assembly files add a pair of
macros that sign the link register. When used before storing to the
stack it will allow hard

arm64: Add PAC instruction macros

To support Pointer Authentication (PAC) in assembly files add a pair of
macros that sign the link register. When used before storing to the
stack it will allow hardware to detect if it has changed before using
it in the return instruction.

Reviewed by: markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42226

show more ...

arm64: Support creating a BTI & PAC note

To detect when an object file is build with the Branch Target
Identification (BTI) and Pointer Authentication Code (PAC) extensions
there is an elf note the

arm64: Support creating a BTI & PAC note

To detect when an object file is build with the Branch Target
Identification (BTI) and Pointer Authentication Code (PAC) extensions
there is an elf note the compiler will insert. It will only do so from
a high level language, e.g. C or C++.

To get the not in assembly add the GNU_PROPERTY_AARCH64_FEATURE_1_NOTE
macro that can be used to create it, and the
GNU_PROPERTY_AARCH64_FEATURE_1_VAL macro to insert the correct value
based on which combination of BTI and PAC are enabled.

Reviewed by: markj (earlier version), emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42225

show more ...

arm64: Add BTI landing pads to assembly functions

When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pa

arm64: Add BTI landing pads to assembly functions

When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pad instruction.

There are three options for these instructions:
1. A breakpoint instruction
2. A pointer authentication PACIASP/PACIBSP
3. A BTI instruction

Option 1 will raise a breakpoint exception so isn't useable in either
cases. Option 2 could be used in some function entry cases, but needs
to be paired with an authentication instruction, and is normally only
used in non-leaf functions we can't use it in this case. This leaves
option 3.

There are four variants of the instruction, the C variant is used on
function entry and the J variant is for jumping within a function.
There is also a JC that works with both and one with no target that
works with neither.

Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42078

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 wh

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

show more ...

Mark EENTRY as .text

To allow it to be used before ENTRY we need to ensure the symbol is
in the .text section. It also needs to be aligned correctly.

While here mark the symbol type as a function a

Mark EENTRY as .text

To allow it to be used before ENTRY we need to ensure the symbol is
in the .text section. It also needs to be aligned correctly.

While here mark the symbol type as a function as in the ENTRY macro.

Reported by: jrtc27
Sponsored by: Arm Ltd

show more ...

Handle functions that use a nop in the arm64 fbt

To trace leaf asm functions we can insert a single nop instruction as
the first instruction in a function and trigger off this.

Reviewed by: gnn
Spo

Handle functions that use a nop in the arm64 fbt

To trace leaf asm functions we can insert a single nop instruction as
the first instruction in a function and trigger off this.

Reviewed by: gnn
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D28132

show more ...

Move the literal pool to the end of asm functions

This keeps the data at the end of a function, near to where it's used.
Sponsored by: Innovate UK