style(9): white space after ; and around binary operators

in for() loops. Also, use 'while', where only the
conditional test of 'for' was used.

Reviewed by: sjg

kern: Use proper prototype for SYSINIT functions

MFC after: 1 week

vm_domainset: Refactor iterators, multiple fixes

vm_domainset_iter_first() would not check if the initial domain selected
by the policy was effectively valid (i.e., allowed by the domainset and
not

vm_domainset: Refactor iterators, multiple fixes

vm_domainset_iter_first() would not check if the initial domain selected
by the policy was effectively valid (i.e., allowed by the domainset and
not marked as ignored by vm_domainset_iter_ignore()). It would just try
to skip it if it had less pages than 'free_min', and would not take into
account the possibility of no domains being valid.

Factor out code that logically belongs to the iterator machinery and is
not tied to how allocations (or impossibility thereof) are to be
handled. This allows to remove duplicated code between
vm_domainset_iter_page() and vm_domainset_iter_policy(), and between
vm_domainset_iter_page_init() and _vm_domainset_iter_policy_init().
This also allows to remove the 'pages' parameter from
vm_domainset_iter_page_init().

This also makes the two-phase logic clearer, revealing an inconsistency
between setting 'di_minskip' to true in vm_domainset_iter_init()
(implying that, in the case of waiting allocations, further attempts
after the first sleep should just allocate for the first domain,
regardless of their situation with respect to their 'free_min') and
trying to skip the first domain if it has too few pages in
vm_domainset_iter_page_init() and _vm_domainset_iter_policy_init(). Fix
this inconsistency by resetting 'di_minskip' to 'true' in
vm_domainset_iter_first() instead so that, after each vm_wait_doms()
(waiting allocations that could not be satisfied immediately), we again
start with only the domains that have more than 'free_min' pages.

While here, fix the minor quirk that the round-robin policy would start
with the domain after the one pointed to by the initial value of
'di_iter' (this just affects the case of resetting '*di_iter', and would
not cause domain skips in other circumstances, i.e., for waiting
allocations that actually wait or at each subsequent new iterator
creation with same iteration index storage).

PR: 277476
Tested by: Kenneth Raplee <kenrap_kennethraplee.com>
Fixes: 7b11a4832691 ("Add files for r327895")
Fixes: e5818a53dbd2 ("Implement several enhancements to NUMA policies.")
Fixes: 23984ce5cd24 ("Avoid resource deadlocks when one domain has exhausted its memory."...)
MFC after: 10 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D51251

show more ...

malloc: extend malloc_usable_size() for contigmalloc

Extend malloc_usable_size() for contigmalloc; it seems the only
outside consumer is LinuxKPI ksize() which by itself has little to
no consumer e

malloc: extend malloc_usable_size() for contigmalloc

Extend malloc_usable_size() for contigmalloc; it seems the only
outside consumer is LinuxKPI ksize() which by itself has little to
no consumer either.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Suggested by: jhb (see D46657)
Reviewed by: jhb, markj
Fixes: 9e6544dd6e02
Differential Revision: https://reviews.freebsd.org/D49571

show more ...

malloc: Fix DEBUG_REDZONE for contigmalloc()

When free() was adapted to support allocations originating from
contigmalloc(), redzone(9) support was not included. redzone(9)
involves adjusting the p

malloc: Fix DEBUG_REDZONE for contigmalloc()

When free() was adapted to support allocations originating from
contigmalloc(), redzone(9) support was not included. redzone(9)
involves adjusting the pointer to freed memory before looking up the
slab cookie, so it's not straightforward to make contigmalloc() opt out
of redzone support.

Thus, augment contigmalloc() to support redzone.

Reported by: glebius
Tested by: dhw
MFC after: 2 weeks
Fixes: 9e6544dd6e02 ("malloc(9): extend contigmalloc(9) by a "slab cookie"")

show more ...

malloc(9): assert wait flags.

The check introduced in d3c11994e1e3 was bogus, combining a non-atomic
“once” flag with an equally non-thread-safe ppsratecheck. Rather than
fix it, just assert what i

malloc(9): assert wait flags.

The check introduced in d3c11994e1e3 was bogus, combining a non-atomic
“once” flag with an equally non-thread-safe ppsratecheck. Rather than
fix it, just assert what it attempts to enforce: that the malloc flags
must include exactly one of M_WAITOK and M_NOWAIT.

Fixes: d3c11994e1e3de7445305abd0d41dce2b8d3e6dc
Sponsored by: Klara, Inc.
Reviewed by: olce, kevans
Differential Revision: https://reviews.freebsd.org/D47309

show more ...

malloc(9): Check for M_NEVERFREED

The recently introduced M_NEVERFREED flag is not meant to be used for
regular malloc requests. Enforce this by checking for M_NEVERFREED
in malloc_dbg.

Reviewed by

malloc(9): Check for M_NEVERFREED

The recently introduced M_NEVERFREED flag is not meant to be used for
regular malloc requests. Enforce this by checking for M_NEVERFREED
in malloc_dbg.

Reviewed by: alc, kib, markj
Differential Revision: https://reviews.freebsd.org/D46199

show more ...

kern_malloc: Restore working KASAN runtime after free() and zfree() folding

In the zfree() case, the explicit_bzero() calls zero all the allocation,
including the redzone which malloc() has marked a

kern_malloc: Restore working KASAN runtime after free() and zfree() folding

In the zfree() case, the explicit_bzero() calls zero all the allocation,
including the redzone which malloc() has marked as invalid. So calling
kasan_mark() before those is in fact necessary.

This fixes a crash at boot when 'ldconfig' is run and tries to get
random bytes through getrandom() (relevant part of the stack is
read_random_uio() -> zfree() -> explicit_bzero()) for kernels with KASAN
compiled in.

Approved by: markj (mentor)
Fixes: 4fab5f005482 ("kern_malloc: fold free and zfree together into one __always_inline func")
MFC after: 10 days
MFC with: 4fab5f005482
Sponsored by: The FreeBSD Foundation

show more ...

kern_malloc: fold free and zfree together into one __always_inline func

free() and zfree() are essentially the same copy and pasted code with
the extra explicit_bzero() (and formerly kasan) calls.

kern_malloc: fold free and zfree together into one __always_inline func

free() and zfree() are essentially the same copy and pasted code with
the extra explicit_bzero() (and formerly kasan) calls. Add a bool to add
the extra functionality and make both functions a wrapper around the common
code and let the compiler do the optimization based on the bool input
when inlining.

No functional changes intended.

Suggested by: kib (in D45812)
Sponsored by: The FreeBSD Foundation
MFC after: 10 days
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D46101

show more ...

Deprecate contigfree(9) in favour of free(9)

As of 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b contigfree(9) is no longer
needed and should not be used anymore. We leave a wrapper for 3rd party
code i

Deprecate contigfree(9) in favour of free(9)

As of 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b contigfree(9) is no longer
needed and should not be used anymore. We leave a wrapper for 3rd party
code in at least 15.x but remove (almost) all other cases from the tree.

This leaves one use of contigfree(9) untouched; that was the original
trigger for 9e6544dd6e02 and is handled in D45813 (to be committed
seperately later).

Sponsored by: The FreeBSD Foundation
Reviewed by: markj, kib
Tested by: pho (10h stress test run)
Differential Revision: https://reviews.freebsd.org/D46099

show more ...

malloc: Handle large malloc sizes in malloc_size()

Reviewed by: kib, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D45661

kern_malloc: remove '.' and '\n' from KASSERT/panic

KASSERT and panic calls should not have a '.' at the end of the message
and do not need '\n'. Remove these.

No functional changes.

Reported by:

kern_malloc: remove '.' and '\n' from KASSERT/panic

KASSERT and panic calls should not have a '.' at the end of the message
and do not need '\n'. Remove these.

No functional changes.

Reported by: markj (during review of D45812)
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: markj, kib
Differential Revision: https://reviews.freebsd.org/D46093

show more ...

malloc(9): extend contigmalloc(9) by a "slab cookie"

Extend kern_malloc.c internals to also cover contigmalloc(9) by a
"slab cookie" and not just malloc/malloc_large. This allows us to
call free(9)

malloc(9): extend contigmalloc(9) by a "slab cookie"

Extend kern_malloc.c internals to also cover contigmalloc(9) by a
"slab cookie" and not just malloc/malloc_large. This allows us to
call free(9) even on contigmalloc(9) addresses and deprecate
contigfree(9). Update the contigmalloc(9) man page accordingly.

The way this is done (free(9) working for contigmalloc) will hide the
UMA/VM bits from a consumer which may otherwise need to know whether
the original allocation was by malloc or contigmalloc by looking at
the cookie (likely via an accessor function). This simplifies
the implementation of consumers of mixed environments a lot.

This is preliminary work to allow LinuxKPI to be adjusted to better
play by the rules Linux puts out for various allocations.
Most of this was described/explained to me by jhb.

One may observe that realloc(9) is currently unchanged (and contrary
to [contig]malloc/[contig]free an implementation may need access
the "slab cookie" information given it will likely be implementation
dependent which allocation type to use if size changes beyond the
usable size of the initial allocation).

Described by: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 10 days
Reviewed by: markj, kib
Differential Revision: https://reviews.freebsd.org/D45812

show more ...

Adjust comments referencing vm_mem_init()

I cannot find a time where the function was not named this.

Reviewed by: kib, markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Rev

Adjust comments referencing vm_mem_init()

I cannot find a time where the function was not named this.

Reviewed by: kib, markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45383

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

uma: Improve memory modified after free panic messages

- Pass zone pointer to trash_ctor() and report zone name in the panic
message. It may be difficult to figyre out zone just by the item size.

uma: Improve memory modified after free panic messages

- Pass zone pointer to trash_ctor() and report zone name in the panic
message. It may be difficult to figyre out zone just by the item size.
- Do not pass user arguments to internal trash calls, pass thezone.
- Report malloc type name in the same unified panic message.
- Report corruption offset from the beginning of the items instead of
the full pointer. It makes panic message shorter and more readable.

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

ddb: annotate some commands with DB_CMD_MEMSAFE

This is not completely exhaustive, but covers a large majority of
commands in the tree.

Reviewed by: markj
Sponsored by: Juniper Networks, Inc.
Spons

ddb: annotate some commands with DB_CMD_MEMSAFE

This is not completely exhaustive, but covers a large majority of
commands in the tree.

Reviewed by: markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D35583

show more ...

realloc(9): Move slab and zone under #ifndef DEBUG_REDZONE.

malloc: Unmark KASAN redzones if the full allocation size was requested

Consumers that want the full allocation size will typically access the
full buffer, so mark the entire allocation as valid to

malloc: Unmark KASAN redzones if the full allocation size was requested

Consumers that want the full allocation size will typically access the
full buffer, so mark the entire allocation as valid to avoid useless
KASAN reports.

Sponsored by: The FreeBSD Foundation

show more ...

malloc_aligned(9): allow zero size and alignment

For alignment we do not need to do anything to make it operational.
For size, upgrade zero sized request to one byte so that we do not
request insane

malloc_aligned(9): allow zero size and alignment

For alignment we do not need to do anything to make it operational.
For size, upgrade zero sized request to one byte so that we do not
request insane amount of memory for placeholder.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32127

show more ...

uma: Add KMSAN hooks

For now, just hook the allocation path: upon allocation, items are
marked as initialized (absent M_ZERO). Some zones are exempted from
this when it would otherwise raise false

uma: Add KMSAN hooks

For now, just hook the allocation path: upon allocation, items are
marked as initialized (absent M_ZERO). Some zones are exempted from
this when it would otherwise raise false positives.

Use kmsan_orig() to update the origin map for UMA and malloc(9)
allocations. This allows KMSAN to print the return address when an
uninitialized UMA item is implicated in a report. For example:
panic: MSan: Uninitialized UMA memory from m_getm2+0x7fe

Sponsored by: The FreeBSD Foundation

show more ...

amd64: Populate the KMSAN shadow maps and integrate with the VM

- During boot, allocate PDP pages for the shadow maps. The region above
KERNBASE is currently not shadowed.
- Create a dummy shadow

amd64: Populate the KMSAN shadow maps and integrate with the VM

- During boot, allocate PDP pages for the shadow maps. The region above
KERNBASE is currently not shadowed.
- Create a dummy shadow for the vm page array. For now, this array is
not protected by the shadow map to help reduce kernel memory usage.
- Grow shadows when growing the kernel map.
- Increase the default kernel stack size when KMSAN is enabled. As with
KASAN, sanitizer instrumentation appears to create stack frames large
enough that the default value is not sufficient.
- Disable UMA's use of the direct map when KMSAN is configured. KMSAN
cannot validate the direct map.
- Disable unmapped I/O when KMSAN configured.
- Lower the limit on paging buffers when KMSAN is configured. Each
buffer has a static MAXPHYS-sized allocation of KVA, which in turn
eats 2*MAXPHYS of space in the shadow map.

Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31295

show more ...

malloc(9): provide missing malloc_aligned implementation

Pointy hat: kevans
Fixes: 6162cf885c00 ("malloc(9): Document/complete aligned variants")