vm_map_entry_delete(): fix the calculation of swap release

Reported and tested by: andrew
Fixes: d160447129fe060b28bcd6ba429d17afdf494ff2
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

vm_object: remove the charge member

State that the object charge is zero if object->cred == NULL, or equal
to the ptoa(object->size) otherwise.

Besides being much simpler, the transition to use obj

vm_object: remove the charge member

State that the object charge is zero if object->cred == NULL, or equal
to the ptoa(object->size) otherwise.

Besides being much simpler, the transition to use object->size corrects
the architectural issue with the use of object->charge. The split
operations effectively carve the holes in the charged regions, but
single counter cannot properly express it. As result, coalescing
anonymous mappings cannot calculate correctly if the extended mapping
already backed by the existing object is already accounted or not [1].

To properly solve the issue, either we need to start tracking exact
charged regions in the anonymous objects, which has the significant
overhead and complications. Or give up on the slight over-accounting
and charge the whole object unconditionally, as it is done in the patch.

Reported by: mmel, pho [1]
Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D54572

show more ...

rfork(2): fix swap accounting in vmspace_unshare()

When an attempt to increase the swap charge for the ucred failed, we
must forcibly increase the charge to allow the vmspace_destroy()
operation to

rfork(2): fix swap accounting in vmspace_unshare()

When an attempt to increase the swap charge for the ucred failed, we
must forcibly increase the charge to allow the vmspace_destroy()
operation to correctly un-charge the accumulated objects.

Add a swap_reserve_force_by_cred() helper and use it in
vmspace_unshare(), same as it is done in normal fork operations.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D54572

show more ...

vm_object_coalesce(): return swap reservation back if overcharged

It is possible for both vm_map_insert() and vm_object_coalesce() to charge
both for the same region. The issue is that vm_map_inser

vm_object_coalesce(): return swap reservation back if overcharged

It is possible for both vm_map_insert() and vm_object_coalesce() to charge
both for the same region. The issue is that vm_map_insert() must charge
in advance to ensure that the mapping would not exceed the swap limit,
but then the coalesce might decide to extend the object, and already
(partially) backs the mapped region.

Handle this by passing to vm_object_coalesce() exact information about
the charging mode of the extending range 'not charging', 'charged' using
flags instead of simple boolean. In vm_object_coalesce(), detect
overcharge and undo it if needed.

Note that this relies on vm_object_coalesce() call being the last action
in vm_map_insert() before extending the previous map entry.

Reported and tested by: pho
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D54338

show more ...

pmap_growkernel(): do not panic immediately, optionally return the error

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/

pmap_growkernel(): do not panic immediately, optionally return the error

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D47935

show more ...

vm_map: fix iterator jump size

The index value in the loop in vm_map_pmap_enter jumps by 1, or some
superpage size, in each iteration. Jump by the superpage size only
when the entire superpage is be

vm_map: fix iterator jump size

The index value in the loop in vm_map_pmap_enter jumps by 1, or some
superpage size, in each iteration. Jump by the superpage size only
when the entire superpage is being mapped.

Reported by: pho
Reported-by: syzbot+1cc9ede76727d2ea2e8d@syzkaller.appspotmail.com
Reviewed by: alc, kib, markj
Tested by: pho
Fixes: Fixes: b3d89a0cde94 ("vm_map: use page iterators in pmap_enter")
Differential Revision: https://reviews.freebsd.org/D49987

show more ...

vm_map: use page iterators in pmap_enter

Change vm_map_pmap_enter to use pctrie iterators to iterate over
pages, rather than using TAILQ links.

Reviewed by: kib
Differential Revision: https://revie

vm_map: use page iterators in pmap_enter

Change vm_map_pmap_enter to use pctrie iterators to iterate over
pages, rather than using TAILQ links.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D49896

show more ...

vm_map: convert several bool members into flags

Extend flags to u_int.
Move system_map and needs_wakeup bools into flags.

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision

vm_map: convert several bool members into flags

Extend flags to u_int.
Move system_map and needs_wakeup bools into flags.

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47934

show more ...

vm_map: wrap map->system_map checks into wrapper

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47934

vm_map: overlap system map mutex and user man sx

This saves 616-584 = 32 bytes per struct vmspace on amd64, which allows
to pack 7 vmspaces per page vs. 6 for non-overlapping layout.

I used anonymo

vm_map: overlap system map mutex and user man sx

This saves 616-584 = 32 bytes per struct vmspace on amd64, which allows
to pack 7 vmspaces per page vs. 6 for non-overlapping layout.

I used anonymous union member feature to avoid too much churn.

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D47934

show more ...

vm: rename MAP_STACK_GROWS_DOWN to MAP_STACK_AREA

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47892

vm_map: remove _GN suffix from MAP_ENTRY_STACK_GAP and MAP_CREATE_STACK_GAP symbols`

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebs

vm_map: remove _GN suffix from MAP_ENTRY_STACK_GAP and MAP_CREATE_STACK_GAP symbols`

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47892

show more ...

sys/vm: remove support for growing-up stacks

Reviewed by: alc, dougm, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D47892

vm_map: add vm_map_find_locked(9)

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D46678

vm ASLR: Handle VM_NRESERV_LEVEL == 0

Some flavors of powerpc don't enable superpage reservations.

Fixes: 3e00c11a4f43 ("arm64: Support the L3 ATTR_CONTIGUOUS page ...")

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the bas

arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]

Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.

Add support for L3C pages to shm_create_largepage().

Add support for creating L3C page mappings to pmap_enter(psind=1).

Add support for reporting L3C page mappings to mincore(2) and
procstat(8).

Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.

Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766

show more ...

vm: Reduce address space fragmentation

jemalloc performs two types of virtual memory allocations: (1) large
chunks of virtual memory, where the chunk size is a multiple of a
superpage and explicitly

vm: Reduce address space fragmentation

jemalloc performs two types of virtual memory allocations: (1) large
chunks of virtual memory, where the chunk size is a multiple of a
superpage and explicitly aligned, and (2) small allocations, mostly
128KB, where no alignment is requested. Typically, it starts with a
small allocation, and over time it makes both types of allocation.

With anon_loc being updated on every allocation, we wind up with a
repeating pattern of a small allocation, a large gap, and a large,
aligned allocation. (As an aside, we wind up allocating a reservation
for these small allocations, but it will never fill because the next
large, aligned allocation updates anon_loc, leaving a gap that will
never be filled with other small allocations.)

With this change, anon_loc isn't updated on every allocation. So, the
small allocations will be clustered together, the large allocations will
be clustered together, and there will be fewer gaps between the
anonymous memory allocations. In addition, I see a small reduction in
reservations allocated (e.g., 1.6% during buildworld), fewer partially
populated reservations, and a small increase in 64KB page promotions on
arm64.

Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39845

show more ...

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
a

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
across all architectures, except for powerpc where using the direct
map addresses involved extra steps in some cases.

The MI/MD split was replaced by a default uma_small_alloc
implementation that can be overridden by architecture-specific code by
defining the UMA_MD_SMALL_ALLOC symbol. Furthermore, UMA_USE_DMAP was
introduced to replace most UMA_MD_SMALL_ALLOC uses.

Reviewed by: markj, kib
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D45084

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

vm_map.c: fix syntax

Fixes: c718009884b3d65528deaff24712cbf98e3be656
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

vm_map.c: plug several more places which might modify entry->offset

for the GUARD entries protecting stacks gaps.

syzkaller: https://syzkaller.appspot.com/bug?extid=c325d6a75e4fd0a68714
Reviewed by

vm_map.c: plug several more places which might modify entry->offset

for the GUARD entries protecting stacks gaps.

syzkaller: https://syzkaller.appspot.com/bug?extid=c325d6a75e4fd0a68714
Reviewed by: dougm, markj (previous version)
Tested by: pho (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41475

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vm: Fix address hints of 0 with MAP_32BIT

Also, rename min_addr to default_addr, which better reflects what it
represents. The min_addr is not a minimum address in the same way that
max_addr is act

vm: Fix address hints of 0 with MAP_32BIT

Also, rename min_addr to default_addr, which better reflects what it
represents. The min_addr is not a minimum address in the same way that
max_addr is actually a maximum address that can be allocated. For
example, a non-zero hint can be less than min_addr and be allocated.

Reported by: dchagin
Reviewed by: dchagin, kib, markj
Fixes: d8e6f4946cec0 "vm: Fix anonymous memory clustering under ASLR"
Differential Revision: https://reviews.freebsd.org/D41397

show more ...

vm_map_protect(): add VM_MAP_PROTECT_GROWSDOWN flag

which requests to propagate lowest stack segment protection to the grow gap.
This seems to be required for Linux emulation.

Reported by: dchagin

vm_map_protect(): add VM_MAP_PROTECT_GROWSDOWN flag

which requests to propagate lowest stack segment protection to the grow gap.
This seems to be required for Linux emulation.

Reported by: dchagin
Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41099

show more ...