rtld: Use a helper variable to simplify a few lines

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D54507

rtld: Use uintptr_t instead of Elf_Addr for init/fini function pointers

This is a no-op on non-CHERI architectures, but is required for CHERI
where Elf_Addr is only an address and not a complete poi

rtld: Use uintptr_t instead of Elf_Addr for init/fini function pointers

This is a no-op on non-CHERI architectures, but is required for CHERI
where Elf_Addr is only an address and not a complete pointer.

While here, consistently use `uintptr_t *` for arrays of init/fini
function pointers.

Reviewed by: imp, kib
Effort: CHERI upstreaming
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D54711

show more ...

rtld: Simplify walking program headers

Store phnum in Obj_Entry instead of phsize and use that to simplify
the terminate expressions when iterating over program headers.

Reviewed by: kib
Obtained f

rtld: Simplify walking program headers

Store phnum in Obj_Entry instead of phsize and use that to simplify
the terminate expressions when iterating over program headers.

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D54710

show more ...

rtld: silence clang's unterminated-string-initialization warning

Since the initializer is used in other places where we can't just
replace it with a char-by-char initializer, this adds a macro for t

rtld: silence clang's unterminated-string-initialization warning

Since the initializer is used in other places where we can't just
replace it with a char-by-char initializer, this adds a macro for the
nonstring attribute (match the linuxkpi definition).

Reviewed by: emaste, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D52535

show more ...

rtld-elf: Mark LD_SHOW_AUXV insecure

This prevents dumping the memory layout of setugid processes.

MFC after: 3 days
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D54033

rtld-elf: move powerpc-specific auxv compat code into arch hook

Tested by: Timothy Pearson (tpearson_raptorengineering.com)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revisi

rtld-elf: move powerpc-specific auxv compat code into arch hook

Tested by: Timothy Pearson (tpearson_raptorengineering.com)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D53801

show more ...

rtld: Add UTRACE_LOAD_OBJECT traces for rtld and the main binary

These two objects are loaded by the kernel not rtld, but adding these
two traces means that UTRACE_LOAD_OBJECT traces now describe th

rtld: Add UTRACE_LOAD_OBJECT traces for rtld and the main binary

These two objects are loaded by the kernel not rtld, but adding these
two traces means that UTRACE_LOAD_OBJECT traces now describe the
mappings for all executables and DSOs in a process' address space.

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D52034

show more ...

rtld: Compute obj->maxsize for obj_rtld

Compute this while parsing the program headers in parse_rtld_phdr().

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision

rtld: Compute obj->maxsize for obj_rtld

Compute this while parsing the program headers in parse_rtld_phdr().

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D52033

show more ...

rtld-elf: Fix VDSO Obj_Entry mapsize and report in utrace(2)

Subtracing addr is only appropriate for position-dependent objects,
where vaddrbase would also be the same value. For position-independen

rtld-elf: Fix VDSO Obj_Entry mapsize and report in utrace(2)

Subtracing addr is only appropriate for position-dependent objects,
where vaddrbase would also be the same value. For position-independent
objects, like the VDSO (which we already assume due to setting vaddrbase
to 0), the segments start at 0, not addr.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D51924

show more ...

rtld-elf: Track allocated TCBs internally and use for distribute_static_tls

Currently rtld delegates to libc or libthr to initialise the TCBs for
all existing threads when dlopen is called for a lib

rtld-elf: Track allocated TCBs internally and use for distribute_static_tls

Currently rtld delegates to libc or libthr to initialise the TCBs for
all existing threads when dlopen is called for a library that is using
static TLS. This creates an odd split where rtld manages all of TLS for
dynamically-linked executables except for this specific case, and is
unnecessarily complex, including having to reason about the locking due
to dropping the bind lock so libthr can take the thread list lock
without deadlocking if any of the code run whilst that lock is held ends
up calling back into rtld (such as for lazy PLT resolution).

The only real reason we call out into libc / libthr is that we don't
have a list of threads in rtld and that's how we find the currently used
TCBs to initialise (and at the same time do the copy in the callee
rather than adding overhead with some kind of callback that provides the
TCB to rtld. If we instead keep a list of allocated TCBs in rtld itself
then we no longer need to do this, and can just copy the data in rtld.
How these TCBs are mapped to threads is irrelevant, rtld can just treat
all TCBs equally and ensure that each TCB's static TLS data block
remains in sync with the current set of loaded modules, just as how
_rtld_allocate_tls creates a fresh TCB and associated data without any
embedded threading model assumptions.

As an implementation detail, to avoid a separate allocation for the list
entry and having to find that allocation from the TCB to remove and free
it on deallocation, we allocate a fake TLS offset for it and embed the
list entry there in each TLS block.

This will also make it easier to add a new TLS ABI downstream in
CheriBSD, especially in the presence of library compartmentalisation.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50920

show more ...

rtld-elf: Extract part of allocate_tls_offset into allocate_tls_offset_common

This will be used to allocate additional space for a TAILQ_ENTRY by rtld
at a known offset from the TCB, as if it were T

rtld-elf: Extract part of allocate_tls_offset into allocate_tls_offset_common

This will be used to allocate additional space for a TAILQ_ENTRY by rtld
at a known offset from the TCB, as if it were TLS data.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D51068

show more ...

rtld-elf: Add Add AT_HWCAP3 and AT_HWCAP4

Add the AT_HWCAP3 and AT_HWCAP4 format strings to auxfmt.

Reviewed by: brooks, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org

rtld-elf: Add Add AT_HWCAP3 and AT_HWCAP4

Add the AT_HWCAP3 and AT_HWCAP4 format strings to auxfmt.

Reviewed by: brooks, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D51007

show more ...

rtld-elf: Pass TCB to allocate_module_tls to avoid re-getting

The only caller already has the current TCB to hand, so just pass it
down rather than get it again. This also makes it clear in the call

rtld-elf: Pass TCB to allocate_module_tls to avoid re-getting

The only caller already has the current TCB to hand, so just pass it
down rather than get it again. This also makes it clear in the caller
that it depends on the (current) TCB, rather than being storage that
could be assigned to any thread (concurrency issues aside).

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50594

show more ...

rtld-elf: Pass struct tcb * around rather than struct dtv **

When this code was first written we didn't have even a struct tcb, so to
make it MI a pointer to the DTV pointer in the TCB was passed ar

rtld-elf: Pass struct tcb * around rather than struct dtv **

When this code was first written we didn't have even a struct tcb, so to
make it MI a pointer to the DTV pointer in the TCB was passed around.
Now that we have a struct tcb we can simplify the code by instead
passing around a pointer to that, and the MI code can access the tcb_dtv
member wherever it happens to be in the layout. This reduces boilerplate
in all the various callers of tls_get_addr_common/slow and makes it
clearer that tls_get_addr_common/slow are operating on the TCB, rather
than obfuscating it slightly through the double pointer.

Whilst here, clarify the comments in aarch64's TLSDESC dynamic resolver,
which were using tp without clarifying what this was for (previously a
pointer to the DTV pointer, now a pointer to the TCB, which happen to be
the same thing for Variant I TLS, and in the case of AArch64 are what
TPIDR_EL0 point to directly, with no offset/bias).

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50591

show more ...

Revert "rtld: fix allocate_module_tls() variant I fallback to static allocation"

This was applying a NetBSD fix to FreeBSD. However, the original code
was correct for FreeBSD. NetBSD's obj->tlsoffse

Revert "rtld: fix allocate_module_tls() variant I fallback to static allocation"

This was applying a NetBSD fix to FreeBSD. However, the original code
was correct for FreeBSD. NetBSD's obj->tlsoffset is relative to the end
of the TCB, not the TCB itself, whilst ours is relative to the TCB[1]
itself. For example, our allocate_tls uses (char *)tcb + obj->tlsoffset
for the memcpy and memset calls.

Without this reverted, for dynamically loaded shared objects, Initial
Exec accesses to TLS variables on variant I architectures (non-x86) use
the correct address, whilst General Dynamic and dlsym(3) use the
incorrect address (TLS_TCB_SIZE past the start). Note that, on arm64,
LLVM only supports TLSDESC (including LLD) and TLSDESC will use the
static resolver if the variable ends up allocated to the static TLS
block, even in the presence of dlopen(3), so only dlsym(3) shows the
discrepancy there.

Whilst here, add a comment to explain this difference to try and avoid
the same mistake being made in future.

[1] In the case of variant II, it's the amount to subtract, so still
positive

This reverts commit e9a38ed2fa61fd264a80f24ceb35f39b0ac6463d.

Reviewed by: kib (prior version)
Fixes: e9a38ed2fa61 ("rtld: fix allocate_module_tls() variant I fallback to static allocation")
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D50565

show more ...

rtld: remove stray tabs

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

rtld-elf: Use size_t rather than uintptr_t for an index

This was previously an Elf_Addr and was turned into a uintptr_t when
really it should have been made a size_t. Even on CHERI both actually
wor

rtld-elf: Use size_t rather than uintptr_t for an index

This was previously an Elf_Addr and was turned into a uintptr_t when
really it should have been made a size_t. Even on CHERI both actually
work in this case, but it's better style (and more consistent with
elsewhere in the file) to use size_t instead.

Fixes: 4642b638a522 ("rtld-elf: Consistently use uintptr_t for TLS implementation")

show more ...

tls: Introduce struct dtv and struct dtv_slot

Rather than treating the DTV as a raw array of uintptr_t, use proper
struct types and gain the benefit of having different types for
different members.

tls: Introduce struct dtv and struct dtv_slot

Rather than treating the DTV as a raw array of uintptr_t, use proper
struct types and gain the benefit of having different types for
different members. In particular, the module slots now have real pointer
types so less casting is generally needed.

Note that, whilst struct dtv_slot may seem a little unnecessary, this
will help downstream in CheriBSD where we wish to be able to easily
alter the layout of a module's slot, which this helps abstract.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50231

show more ...

rtld-elf: Use clear pointer provenance when updating DTV pointer

On traditional architectures where uintptr_t is just a plain integer,
there is no provenance from the order of operations. However, o

rtld-elf: Use clear pointer provenance when updating DTV pointer

On traditional architectures where uintptr_t is just a plain integer,
there is no provenance from the order of operations. However, on CHERI
there is even for uintptr_t, and in future this code will use actual
pointer types anyway, where the provenance does technically matter even
for non-CHERI. Commute and associate the operands appropriately to
ensure the provenance is for the new allocation, not the old one.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50230

show more ...

rtld-elf: Use struct tcb * rather than uintptr_t ** for the tcb

This lets us access via named struct members rather than magic
hard-coded indices.

Reviewed by: kib
Differential Revision: https://re

rtld-elf: Use struct tcb * rather than uintptr_t ** for the tcb

This lets us access via named struct members rather than magic
hard-coded indices.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50229

show more ...

rtld-elf: Use variables more consistent with Variant I for Variant II TLS

Firstly, the second argument to allocate_tls is the old TCB (versus
oldtls, which has less of a clear meaning), so rename it

rtld-elf: Use variables more consistent with Variant I for Variant II TLS

Firstly, the second argument to allocate_tls is the old TCB (versus
oldtls, which has less of a clear meaning), so rename it to oldtcb like
Variant I.

Secondly, segbase and oldsegbase are oriented towards what ends up in
the segment registers, but that's not the main concern here, and those
don't convey what they actually point to. Instead, rename segbase to tcb
and change it to a uintptr_t **, and remove oldsegbase as it's always
equal to oldtcb, again both matching Variant I.

Thirdly, rename tls to tls_block, again both matching Variant I.

Finally, similarly rename tls to tcb in free_tls, and oldtls to oldtcb
in the common _rtld_allocate_tls.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50232

show more ...

rtld-elf: Consistently use uintptr_t for TLS implementation

Elf_Addr is the format of addresses in the ELF file with the current
ABI's default class. This is normally the same as the format of an
ad

rtld-elf: Consistently use uintptr_t for TLS implementation

Elf_Addr is the format of addresses in the ELF file with the current
ABI's default class. This is normally the same as the format of an
address at run time, though technically exceptions do exist outside of
FreeBSD's currently-supported architectures (for example, IA-64's LP64
supports both ELFCLASS32 and ELFCLASS64 file formats; LP64 vs ILP32 is
an orthogonal EF_IA_64_ABI64 flag). On traditional architectures,
including all currently-supported FreeBSD architectures, addresses and
pointers are synonymous, but on CHERI they are not, as pointers are
capabilities that contain metadata alongside the address. In the cases
here, the quantities are run-time pointers, not addresses (and
definitely not ELF file addresses), so we should use pointer-ish types.

Note that we already use uintptr_t in struct tcb (both Variant I and
Variant II) but still use Elf_Addr in various places here (including
different argument types for tls_get_addr_slow and tls_get_addr_common).

Also use char * for addr rather than even uintptr_t, since most of the
time we want it to be an actual pointer.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D50226

show more ...

rtld-elf: Fix executable's TLS module index for direct exec

For direct exec mode we reuse map_object, but tls_max_index is
initialised to 1. As a result, the executable ends up being assigned
module

rtld-elf: Fix executable's TLS module index for direct exec

For direct exec mode we reuse map_object, but tls_max_index is
initialised to 1. As a result, the executable ends up being assigned
module 2 (and the generation is pointlessly incremented, unlike in
digest_phdr for the normal case). For most architectures this is
harmless, since TLS linker relaxation will optimise General Dynamic
accesses to Initial Exec or Local Exec for executables, but on RISC-V
this relaxation does not exist, yet the linker will initialise the
tls_index in the GOT with module 1, and at run time the call to
__tls_get_addr will fail with:

ld-elf.so.1: Can't find module with TLS index 1

Fix this by making map_object use 1 for obj->tlsindex when it's loading
the main executable, and don't bother to increment tls_dtv_generation
either, matching digest_phdr (though that one is harmless).

(Note this also applies to MIPS on stable/13)

Reviewed by: kib
Fixes: 0fc65b0ab82c ("Make ld-elf.so.1 directly executable.")
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D50186

show more ...

rtld-elf: Fix UB for direct exec with no extra rtld arguments

If no extra rtld arguments are provided, rtld_argc will be 1 (for
argv[0] and so we are shifting the entire memory range down by a singl

rtld-elf: Fix UB for direct exec with no extra rtld arguments

If no extra rtld arguments are provided, rtld_argc will be 1 (for
argv[0] and so we are shifting the entire memory range down by a single
pointer. However, unlike argv and envp, auxp's entries are two pointers
in size, not one, and so in this case the source and destination
overlap, meaning simple assignment is UB (C99 6.5.16.1p3). On many
architectures this ends up being harmless as the compiler will emit
double machine word loads and stores, or if it splits them it may still
schedule them such that it works in this case, but our RISC-V baseline
does not include such instructions and LLVM ends up picking a schedule
that copies the second word before the first word, thereby replacing the
first word with a copy of the second word. This results in direct exec
mode segfaulting on RISC-V when given no arguments.

Fix this by using a temporary in the source and let the compiler safely
elide its use.

Reviewed by: kib
Fixes: 0fc65b0ab82c ("Make ld-elf.so.1 directly executable.")
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D50185

show more ...

rtld-elf: Fix dl_iterate_phdr's dlpi_tls_data for PowerPC and RISC-V

The implementation of dl_iterate_phdr abuses tls_get_addr_slow to get to
the start of the TLS block, inlining the implementation

rtld-elf: Fix dl_iterate_phdr's dlpi_tls_data for PowerPC and RISC-V

The implementation of dl_iterate_phdr abuses tls_get_addr_slow to get to
the start of the TLS block, inlining the implementation of
__tls_get_addr as if the tls_index's ti_offset were 0 (historically it
called __tls_get_addr itself but changed due to locking issues). For
most architectures, tls_index's ti_offset (relocated by DTPOFF/DTPREL
for GOT entries) is just the offset within that module's TLS block.
However, for PowerPC and RISC-V, which have a non-zero TLS_DTV_OFFSET
and thus are designed assuming DTV entries are biased by that value,
ti_offset normally has TLS_DTV_OFFSET pre-subtracted, but it's
__tls_get_addr's responsibility to compensate for that. By using an
offset of zero here, tls_get_addr_slow will return a pointer to the
start of the TLS block itself, so by adding TLS_DTV_OFFSET we will point
TLS_DTV_OFFSET past the module's TLS block.

Fix this by removing the extra bias (the alternative would be to pass
-TLS_DTV_OFFSET and keep the addition, which would more closely follow
what __tls_get_addr does, but this is more direct).

(Note this also applies to MIPS on stable/13)

Reviewed by: kib
Fixes: d36d68161517 ("rtld dl_iterate_phdr(): dlpi_tls_data is wrong")
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D50184

show more ...