amd64 pmap: add kva_layout.kva_max

And use it instead of manually providing the same value as sentinel in
code for sysctl vm.pmap.kernel_maps.

Reviewed by: markj
Sponsored by: The FreeBSD Foundatio

amd64 pmap: add kva_layout.kva_max

And use it instead of manually providing the same value as sentinel in
code for sysctl vm.pmap.kernel_maps.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D51453

show more ...

amd64 K*SAN: add kva_layout members for K*SAN maps

The members are added regardless of the kernel config.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://revi

amd64 K*SAN: add kva_layout members for K*SAN maps

The members are added regardless of the kernel config.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D51453

show more ...

amd64 pmap: for LA57 move large map out of kernel pml4 slot

and allow it to consume up to 32 pml5 slots (32 x 2048 TB), same as
DMAP.

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
Di

amd64 pmap: for LA57 move large map out of kernel pml4 slot

and allow it to consume up to 32 pml5 slots (32 x 2048 TB), same as
DMAP.

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D51364

show more ...

amd64: for LA57, connect DMAP directly to PML5, currently reserving 32 slots

This removes the 4T limit on the amd64 usable phys memory.

Other large KVA consumers like large map, should be moved out

amd64: for LA57, connect DMAP directly to PML5, currently reserving 32 slots

This removes the 4T limit on the amd64 usable phys memory.

Other large KVA consumers like large map, should be moved out of the
top PML4 entry as well. After that, kmem on LA57 amd64 can be extended.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D51085

show more ...

amd64: add kva_layout

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D51085

amd64: express KV4ADDR() using KV5ADDR()

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D51053

amd64: add machine/pte.h

Following arm64 and risc-v, move definitions that describe
hardware-enforced layout of PTEs and #PF error bits, into a dedicated
header.

Reviewed by: markj
Sponsored by: Th

amd64: add machine/pte.h

Following arm64 and risc-v, move definitions that describe
hardware-enforced layout of PTEs and #PF error bits, into a dedicated
header.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D47749

show more ...

Remove stray whitespaces from sys/amd64/

Signed-off-by: Joshua Rogers <Joshua@Joshua.Hu>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1418

amd64: add variables indicating INVLPGB works

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D45191

pmap: move the smp_targeted_tlb_shutdown pointer stuff to amd64 pmap.h

Fixes: bec000c9c1ef409989685bb03ff0532907befb4aESC
Sponsored by: The FreeBSD Foundation

pmap: Convert boolean_t to bool.

Reviewed by: kib (older version)
Differential Revision: https://reviews.freebsd.org/D39921

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreove

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreover, the name is a
bit confusing given the existence of pmap_bootstrap_san(). Just remove
it and call kasan_init_early() directly like we do on amd64. It will
not be used by KMSAN in a forthcoming patch series.

No functional change intended.

MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43403

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

amd64/arm64 pmap: Stop requiring the accessed bit for superpage promotion

Stop requiring all of the PTEs to have the accessed bit set for superpage
promotion to occur. Given that change, add suppor

amd64/arm64 pmap: Stop requiring the accessed bit for superpage promotion

Stop requiring all of the PTEs to have the accessed bit set for superpage
promotion to occur. Given that change, add support for promotion to
pmap_enter_quick(), which does not set the accessed bit in the PTE that
it creates.

Since the final mapping within a superpage-aligned and sized region of a
memory-mapped file is typically created by a call to pmap_enter_quick(),
we now achieve promotions in circumstances where they did not occur
before, for example, the X server's read-only mapping of libLLVM-15.so.

See also https://www.usenix.org/system/files/atc20-zhu-weixi_0.pdf

Reviewed by: kib, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40478

show more ...

amd64: pmap.h put a guard around a pcpu.h function

pmap_get_pcid() calls zpcpu_get() which is defined in pcpu.h.
It is unclear why we do not include that header but like right
above the change add a

amd64: pmap.h put a guard around a pcpu.h function

pmap_get_pcid() calls zpcpu_get() which is defined in pcpu.h.
It is unclear why we do not include that header but like right
above the change add another guard around pmap_get_pcid().
This allows some LinuxKPI headers to compile again.

Suggested by: markj
MFC after: 10 days

show more ...

pmap_{un}map_io_transient: Use bool instead of boolean_t.

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D39920

amd64: store pcids pmap data in pcpu zone

This change eliminates the struct pmap_pcid array embedded into struct
pmap and sized by MAXCPU, which would bloat with MAXCPU increase. Also
it removes fa

amd64: store pcids pmap data in pcpu zone

This change eliminates the struct pmap_pcid array embedded into struct
pmap and sized by MAXCPU, which would bloat with MAXCPU increase. Also
it removes false sharing of cache lines, since the array elements are
mostly locally accessed by corresponding CPUs.

Suggested by: mjg
Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D39890

show more ...

amd64: add pmap_get_pcid() helper

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D39890

amd64: move definition of the struct pmap_pcids into _pmap.h

and rename the structure to pmap_pcid.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: htt

amd64: move definition of the struct pmap_pcids into _pmap.h

and rename the structure to pmap_pcid.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D39890

show more ...

amd64: fix PKRU and swapout interaction

When vm_map_remove() is called from vm_swapout_map_deactivate_pages()
due to swapout, PKRU attributes for the removed range must be kept
intact. Provide a va

amd64: fix PKRU and swapout interaction

When vm_map_remove() is called from vm_swapout_map_deactivate_pages()
due to swapout, PKRU attributes for the removed range must be kept
intact. Provide a variant of pmap_remove(), pmap_map_delete(), to
allow pmap to distinguish between real removes of the UVA mappings
and any other internal removes, e.g. swapout.

For non-amd64, pmap_map_delete() is stubbed by define to pmap_remove().

Reported by: andrew
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D39556

show more ...

amd64 pmap.h: make it easier to use the header for other consumers

Guard pmap_invlpg() definition with checks that only provide it when
both sys/pcpu.h and machine/cpufunc.h were already included.

amd64 pmap.h: make it easier to use the header for other consumers

Guard pmap_invlpg() definition with checks that only provide it when
both sys/pcpu.h and machine/cpufunc.h were already included.

Requested by: Elliott Mitchell
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

amd64: for small cores, use (big hammer) INVPCID_CTXGLOB instead of INVLPG

A hypothetical CPU bug makes invalidation of global PTEs using INVLPG
in pcid mode unreliable, it seems. The workaround is

amd64: for small cores, use (big hammer) INVPCID_CTXGLOB instead of INVLPG

A hypothetical CPU bug makes invalidation of global PTEs using INVLPG
in pcid mode unreliable, it seems. The workaround is applied for all
CPUs with small cores, since we do not know the scope of the issue, and
the right fix.

Reviewed by: alc (previous version)
Discussed with: emaste, markj
Tested by: karels
PR: 261169, 266145
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D37770

show more ...

sys: Consolidate common implementation details of PV entries.

Add a <sys/_pv_entry.h> intended for use in <machine/pmap.h> to
define struct pv_entry, pv_chunk, and related macros and inline
function

sys: Consolidate common implementation details of PV entries.

Add a <sys/_pv_entry.h> intended for use in <machine/pmap.h> to
define struct pv_entry, pv_chunk, and related macros and inline
functions.

Note that powerpc does not yet use this as while the mmu_radix pmap
in powerpc uses the new scheme (albeit with fewer PV entries in a
chunk than normal due to an used pv_pmap field in struct pv_entry),
the Book-E pmaps for powerpc use the older style PV entries without
chunks (and thus require the pv_pmap field).

Suggested by: kib
Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36685

show more ...

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.f

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36548

show more ...