arm, powerpc, riscv: implement atomic_set/clear_16

Reviewed by: jrtc27, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D52626

atomics: Constify loads: riscv: Convert missed atomic_load_acq_64()

Reviewed by: kib
Fixes: 5e9a82e898d5 ("atomics: Constify loads")
MFC after: 4 days
Sponsored by: The FreeBSD Fo

atomics: Constify loads: riscv: Convert missed atomic_load_acq_64()

Reviewed by: kib
Fixes: 5e9a82e898d5 ("atomics: Constify loads")
MFC after: 4 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48497

show more ...

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm
statement's constraint for '*p' (the value to load). CMPXCHG8B always
writes back some value, even when the value exchange does not happen in
which case what was read is written back. atomic_load_acq_64_i586()
further takes care of the operation atomically writing back the same
value that was read in any case. All in all, this makes the inline
asm's write back undetectable by any other code, whether executing on
other CPUs or code on the same CPU before and after the call to
atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will
trigger a #GP(0) if the memory address is part of a read-only mapping.
This unfortunate property is however out of scope of the C abstract
machine, and in particular independent of whether the 'uint64_t' pointed
to is declared 'const' or not.

Approved by: markj (mentor)
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46887

show more ...

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from:

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631

show more ...

riscv: Add implementations of atomic_testand(set|clear)_(32|64|long)

These use amoor and amoand rather than a loop.

Also define atomic_testandset_acq_(64|long) using amoor.aq.

Reviewed by: mhorne,

riscv: Add implementations of atomic_testand(set|clear)_(32|64|long)

These use amoor and amoand rather than a loop.

Also define atomic_testandset_acq_(64|long) using amoor.aq.

Reviewed by: mhorne, kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47627

show more ...

riscv: Implement atomic operations

Implement atomic_load_acq_16, atomic_store_rel_16.

These are needed by bhyve(8) PCIe bus emulation code.

Group 16-bit atomic functions similarly to 32 and 64-bit

riscv: Implement atomic operations

Implement atomic_load_acq_16, atomic_store_rel_16.

These are needed by bhyve(8) PCIe bus emulation code.

Group 16-bit atomic functions similarly to 32 and 64-bit.

Reviewed by: mhorne
Differential Revision: https://reviews.freebsd.org/D45228

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

riscv: fix errors in some atomic type aliases

This appears to be a copy-and-paste error that has simply been
overlooked. The tree contains only two calls to any of the affected
variants, but recent

riscv: fix errors in some atomic type aliases

This appears to be a copy-and-paste error that has simply been
overlooked. The tree contains only two calls to any of the affected
variants, but recent additions to the test suite started exercising the
call to atomic_clear_rel_int() in ng_leave_write(), reliably causing
panics.

Apparently, the issue was inherited from the arm64 atomic header. That
instance was addressed in c90baf6817a0, but the fix did not make its way
to RISC-V.

Note that the particular test case ng_macfilter_test:main still appears
to fail on this platform, but this change reduces the panic to a
timeout.

PR: 253237
Reported by: Jenkins, arichardson
Reviewed by: kp, arichardson
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D29064

show more ...

riscv: clean up empty lines in .c and .h files

Fix atomic_*cmpset32 on riscv64 with clang.

The lr.w instruction used to read the value from memory sign-extends
the value read from memory. GCC sign-extends the 32-bit comparison
value passed in w

Fix atomic_*cmpset32 on riscv64 with clang.

The lr.w instruction used to read the value from memory sign-extends
the value read from memory. GCC sign-extends the 32-bit comparison
value passed in whereas clang currently does not. As a result, if the
value being compared has the MSB set, the comparison fails for
matching 32-bit values when compiled with clang.

Use a cast to explicitly sign-extend the unsigned comparison value.
This works with both GCC and clang.

There is commentary in the RISC-V spec that suggests that GCC's
approach is more correct, but it is not clear if the commentary in the
RISC-V spec is binding.

Reviewed by: mhorne
Obtained from: Axiado
MFC after: 2 weeks
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D22084

show more ...

Merge ^/head r352764 through r353315.

riscv: use the common sub-word {,f}cmpset implementation

Reviewed by: mhorne
Differential Revision: https://reviews.freebsd.org/D21888

Merge ^/head r337286 through r337585.

arm, powerpc, riscv: implement atomic_set/clear_16

Reviewed by: jrtc27, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D52626

atomics: Constify loads: riscv: Convert missed atomic_load_acq_64()

Reviewed by: kib
Fixes: 5e9a82e898d5 ("atomics: Constify loads")
MFC after: 4 days
Sponsored by: The FreeBSD Fo

atomics: Constify loads: riscv: Convert missed atomic_load_acq_64()

Reviewed by: kib
Fixes: 5e9a82e898d5 ("atomics: Constify loads")
MFC after: 4 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D48497

show more ...

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm
statement's constraint for '*p' (the value to load). CMPXCHG8B always
writes back some value, even when the value exchange does not happen in
which case what was read is written back. atomic_load_acq_64_i586()
further takes care of the operation atomically writing back the same
value that was read in any case. All in all, this makes the inline
asm's write back undetectable by any other code, whether executing on
other CPUs or code on the same CPU before and after the call to
atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will
trigger a #GP(0) if the memory address is part of a read-only mapping.
This unfortunate property is however out of scope of the C abstract
machine, and in particular independent of whether the 'uint64_t' pointed
to is declared 'const' or not.

Approved by: markj (mentor)
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46887

show more ...

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from:

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631

show more ...

riscv: Add implementations of atomic_testand(set|clear)_(32|64|long)

These use amoor and amoand rather than a loop.

Also define atomic_testandset_acq_(64|long) using amoor.aq.

Reviewed by: mhorne,

riscv: Add implementations of atomic_testand(set|clear)_(32|64|long)

These use amoor and amoand rather than a loop.

Also define atomic_testandset_acq_(64|long) using amoor.aq.

Reviewed by: mhorne, kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47627

show more ...

riscv: Implement atomic operations

Implement atomic_load_acq_16, atomic_store_rel_16.

These are needed by bhyve(8) PCIe bus emulation code.

Group 16-bit atomic functions similarly to 32 and 64-bit

riscv: Implement atomic operations

Implement atomic_load_acq_16, atomic_store_rel_16.

These are needed by bhyve(8) PCIe bus emulation code.

Group 16-bit atomic functions similarly to 32 and 64-bit.

Reviewed by: mhorne
Differential Revision: https://reviews.freebsd.org/D45228

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

riscv: fix errors in some atomic type aliases

This appears to be a copy-and-paste error that has simply been
overlooked. The tree contains only two calls to any of the affected
variants, but recent

riscv: fix errors in some atomic type aliases

This appears to be a copy-and-paste error that has simply been
overlooked. The tree contains only two calls to any of the affected
variants, but recent additions to the test suite started exercising the
call to atomic_clear_rel_int() in ng_leave_write(), reliably causing
panics.

Apparently, the issue was inherited from the arm64 atomic header. That
instance was addressed in c90baf6817a0, but the fix did not make its way
to RISC-V.

Note that the particular test case ng_macfilter_test:main still appears
to fail on this platform, but this change reduces the panic to a
timeout.

PR: 253237
Reported by: Jenkins, arichardson
Reviewed by: kp, arichardson
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D29064

show more ...

riscv: clean up empty lines in .c and .h files

Fix atomic_*cmpset32 on riscv64 with clang.

The lr.w instruction used to read the value from memory sign-extends
the value read from memory. GCC sign-extends the 32-bit comparison
value passed in w

Fix atomic_*cmpset32 on riscv64 with clang.

The lr.w instruction used to read the value from memory sign-extends
the value read from memory. GCC sign-extends the 32-bit comparison
value passed in whereas clang currently does not. As a result, if the
value being compared has the MSB set, the comparison fails for
matching 32-bit values when compiled with clang.

Use a cast to explicitly sign-extend the unsigned comparison value.
This works with both GCC and clang.

There is commentary in the RISC-V spec that suggests that GCC's
approach is more correct, but it is not clear if the commentary in the
RISC-V spec is binding.

Reviewed by: mhorne
Obtained from: Axiado
MFC after: 2 weeks
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D22084

show more ...

Merge ^/head r352764 through r353315.

riscv: use the common sub-word {,f}cmpset implementation

Reviewed by: mhorne
Differential Revision: https://reviews.freebsd.org/D21888