arm/atomic.h: remove misleading comment

Noted and reviewed by: jrtc27
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D52626

arm, powerpc, riscv: implement atomic_set/clear_16

Reviewed by: jrtc27, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D52626

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm
statement's constraint for '*p' (the value to load). CMPXCHG8B always
writes back some value, even when the value exchange does not happen in
which case what was read is written back. atomic_load_acq_64_i586()
further takes care of the operation atomically writing back the same
value that was read in any case. All in all, this makes the inline
asm's write back undetectable by any other code, whether executing on
other CPUs or code on the same CPU before and after the call to
atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will
trigger a #GP(0) if the memory address is part of a read-only mapping.
This unfortunate property is however out of scope of the C abstract
machine, and in particular independent of whether the 'uint64_t' pointed
to is declared 'const' or not.

Approved by: markj (mentor)
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46887

show more ...

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from:

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631

show more ...

atomic(9): Remove fcmpset-based fallback for atomic_testand(clear|set)

All architectures implement a MD version

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.fre

atomic(9): Remove fcmpset-based fallback for atomic_testand(clear|set)

All architectures implement a MD version

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47629

show more ...

arm: Implement atomic_testandset_acq_long as a simple wrapper

Use a memory barrier after calling the existing atomic_testandset_long
rather than using the fcmpset-based fallback version from
<sys/_a

arm: Implement atomic_testandset_acq_long as a simple wrapper

Use a memory barrier after calling the existing atomic_testandset_long
rather than using the fcmpset-based fallback version from
<sys/_atomic_subword.h>.

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47628

show more ...

arm: Assume __ARM_ARCH == 7

The only supported 32-bit Arm architecture is Armv7. Remove old checks
for earlier architecture revisions.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.f

arm: Assume __ARM_ARCH == 7

The only supported 32-bit Arm architecture is Armv7. Remove old checks
for earlier architecture revisions.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45957

show more ...

arm: add missing atomic-pointer functions

The pointer function types were missing for these functions, so add
them. Valuable for places where type sizes vary by architecture.

Differential Revision

arm: add missing atomic-pointer functions

The pointer function types were missing for these functions, so add
them. Valuable for places where type sizes vary by architecture.

Differential Revision: https://reviews.freebsd.org/D37778
Reviewed by: imp, andrew
Pull Request: https://github.com/freebsd/freebsd-src/pull/1126

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm: Move contents of atomic-v6.h into atomic.h

Previously we had an armv4/v5 and armv6/v7 implementation of the atomic
operations. As we have removed armv4/v5 support we can now merge the
armv6/v7

arm: Move contents of atomic-v6.h into atomic.h

Previously we had an armv4/v5 and armv6/v7 implementation of the atomic
operations. As we have removed armv4/v5 support we can now merge the
armv6/v7 code into atomic.h

Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41135

show more ...

arm: Explain why _atomic_subword.h is needed

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41134

Remove left over includes from armv4/v5

These includes were added for armv4 and armv5 support. Remove them as
this has been removed from the tree.

Sponsored by: Arm Ltd
Differential Revision: https

Remove left over includes from armv4/v5

These includes were added for armv4 and armv5 support. Remove them as
this has been removed from the tree.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41133

show more ...

Remove now unused armv4 and not-INTRNG files.

Expand generic subword atomic primitives

The goal of this change is to make the atomic_load_acq_{8,16},
atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives
available in MI-namespa

Expand generic subword atomic primitives

The goal of this change is to make the atomic_load_acq_{8,16},
atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives
available in MI-namespace.

The second goal is to get this draft out of my local tree, as anything that
requires a full tinderbox is a big burden out of tree. MD specifics can be
refined individually afterwards.

The generic implementations may not be ideal for your architecture; feel
free to implement better versions. If no subword_atomic definitions are
needed, the include can be removed from your arch's machine/atomic.h.
Generic definitions are guarded by defined macros of the same name. To
avoid picking up conflicting generic definitions, some macro defines are
added to various MD machine/atomic.h to register an existing implementation.

Include _atomic_subword.h in arm and arm64 machine/atomic.h.

For some odd reason, KCSAN only generates some versions of primitives.
Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and
atomic_testandset.*_long. There are other questionably disabled primitives,
but I didn't run into them, so I left them alone. KCSAN is only built for
amd64 in tinderbox for now.

Add atomic_subword implementations of atomic_load_acq_{8,16} implemented
using masking and atomic_load_acq_32.

Add generic atomic_subword implementations of atomic_testandset_long(),
atomic_testandclear_long(), and atomic_testandset_acq_long(), using
atomic_fcmpset_long() and atomic_fcmpset_acq_long().

On x86, add atomic_testandset_acq_long as an alias for
atomic_testandset_long.

Reviewed by: kevans, rlibby (previous versions both)
Differential Revision: https://reviews.freebsd.org/D22963

show more ...

Merge ^/head r341764 through r341812.

Implement atomic_swap_xxx() for all platforms.

Differential Revision: https://reviews.freebsd.org/D18450
Reviewed by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

arm/atomic.h: remove misleading comment

Noted and reviewed by: jrtc27
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D52626

arm, powerpc, riscv: implement atomic_set/clear_16

Reviewed by: jrtc27, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D52626

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm
statement's constraint for '*p' (the value to load). CMPXCHG8B always
writes back some value, even when the value exchange does not happen in
which case what was read is written back. atomic_load_acq_64_i586()
further takes care of the operation atomically writing back the same
value that was read in any case. All in all, this makes the inline
asm's write back undetectable by any other code, whether executing on
other CPUs or code on the same CPU before and after the call to
atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will
trigger a #GP(0) if the memory address is part of a read-only mapping.
This unfortunate property is however out of scope of the C abstract
machine, and in particular independent of whether the 'uint64_t' pointed
to is declared 'const' or not.

Approved by: markj (mentor)
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46887

show more ...

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from:

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631

show more ...

atomic(9): Remove fcmpset-based fallback for atomic_testand(clear|set)

All architectures implement a MD version

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.fre

atomic(9): Remove fcmpset-based fallback for atomic_testand(clear|set)

All architectures implement a MD version

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47629

show more ...

arm: Implement atomic_testandset_acq_long as a simple wrapper

Use a memory barrier after calling the existing atomic_testandset_long
rather than using the fcmpset-based fallback version from
<sys/_a

arm: Implement atomic_testandset_acq_long as a simple wrapper

Use a memory barrier after calling the existing atomic_testandset_long
rather than using the fcmpset-based fallback version from
<sys/_atomic_subword.h>.

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47628

show more ...

arm: Assume __ARM_ARCH == 7

The only supported 32-bit Arm architecture is Armv7. Remove old checks
for earlier architecture revisions.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.f

arm: Assume __ARM_ARCH == 7

The only supported 32-bit Arm architecture is Armv7. Remove old checks
for earlier architecture revisions.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45957

show more ...

arm: add missing atomic-pointer functions

The pointer function types were missing for these functions, so add
them. Valuable for places where type sizes vary by architecture.

Differential Revision

arm: add missing atomic-pointer functions

The pointer function types were missing for these functions, so add
them. Valuable for places where type sizes vary by architecture.

Differential Revision: https://reviews.freebsd.org/D37778
Reviewed by: imp, andrew
Pull Request: https://github.com/freebsd/freebsd-src/pull/1126

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/