riscv: rework page table bootstrap

The overall goal of the change is to reduce the amount of work done in
locore assembly, and defer as much as possible until pmap_bootstrap().
Currently, half the s

riscv: rework page table bootstrap

The overall goal of the change is to reduce the amount of work done in
locore assembly, and defer as much as possible until pmap_bootstrap().
Currently, half the setup is done in assembly, and then we pass the l1pt
address to pmap_bootstrap() where it is amended with other mappings.

Inspiration and understanding has been taken from amd64's
create_pagetables() routine, and I try to present the page table
construction in the same way: a linear procedure with commentary
explaining what we are doing and why. Thus the core of the new
implementation is contained in pmap_create_pagetables().

Once pmap_create_pagetables() has finished, we switch to the new
pagetable root and leave the bootstrap ones created by locore behind,
resulting in a minimal 8kB of wasted space.

Having the whole procedure in one place, in C code, allows it to be more
easily understood, while also making it more amenable to future changes
which depend on CPU feature/errata detection.

Note that with this change the size of the early devmap is bumped up
from one to four L2 pages (8MB).

Reviewed by: markj
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45327

show more ...

riscv: Construct an identity map in locore.S

This is useful for two reasons. Within this change, it allows the
early DTB mapping to be eliminated, as we can now just dereference the
physical address

riscv: Construct an identity map in locore.S

This is useful for two reasons. Within this change, it allows the
early DTB mapping to be eliminated, as we can now just dereference the
physical address provided by FW and copy the DTB contents into KVA.

It will also aid in an upcoming change: the larger reworking of page
table bootstrapping on this platform.

Reviewed by: markj, jhb
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45324

show more ...

vm: Simplify startup page dumping conditional

This commit introduces the MINIDUMP_STARTUP_PAGE_TRACKING symbol and
uses it to simplify several instances of a complex preprocessor conditional
for add

vm: Simplify startup page dumping conditional

This commit introduces the MINIDUMP_STARTUP_PAGE_TRACKING symbol and
uses it to simplify several instances of a complex preprocessor conditional
for adding pages allocated when bootstraping the kernel to minidumps.

Reviewed by: markj, mhorne
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D45085

show more ...

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
a

uma: Deduplicate uma_small_alloc

This commit refactors the UMA small alloc code and
removes most UMA machine-dependent code.
The existing machine-dependent uma_small_alloc code is almost identical
across all architectures, except for powerpc where using the direct
map addresses involved extra steps in some cases.

The MI/MD split was replaced by a default uma_small_alloc
implementation that can be overridden by architecture-specific code by
defining the UMA_MD_SMALL_ALLOC symbol. Furthermore, UMA_USE_DMAP was
introduced to replace most UMA_MD_SMALL_ALLOC uses.

Reviewed by: markj, kib
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D45084

show more ...

arm64, riscv: remove unused declaration

It is inherited from arm, where the global exists and is used. No
functional change.

Reviewed by: markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundatio

arm64, riscv: remove unused declaration

It is inherited from arm, where the global exists and is used. No
functional change.

Reviewed by: markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45323

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

riscv: Use PMAP_MAPDEV_EARLY_SIZE in locore and pmap_bootstrap

Use PMAP_MAPDEV_EARLY_SIZE instead of assuming that its value is always
L2_SIZE. Add compile-time assertions to check that the size mat

riscv: Use PMAP_MAPDEV_EARLY_SIZE in locore and pmap_bootstrap

Use PMAP_MAPDEV_EARLY_SIZE instead of assuming that its value is always
L2_SIZE. Add compile-time assertions to check that the size matches the
expectations in locore.

Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D40110

show more ...

riscv: Avoid passing invalid addresses to pmap_fault()

After the addition of SV48 support, VIRT_IS_VALID() did not exclude
addresses that are in the SV39 address space hole but not in the SV48
addre

riscv: Avoid passing invalid addresses to pmap_fault()

After the addition of SV48 support, VIRT_IS_VALID() did not exclude
addresses that are in the SV39 address space hole but not in the SV48
address space hole. This can result in mishandling of accesses to that
range when in SV39 mode.

Fix the problem by modifying VIRT_IS_VALID() to use the runtime address
space bounds. Then, if the address is invalid, and pcb_onfault is set,
give vm_fault_trap() a chance to veto the access instead of panicking.

PR: 265439
Reviewed by: jhb
Reported and tested by: Robert Morris <rtm@lcs.mit.edu>
Fixes: 31218f3209ac ("riscv: Add support for enabling SV48 mode")
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35952

show more ...

riscv: Add support for enabling SV48 mode

This increases the size of the user map from 256GB to 128TB. The kernel
map is left unchanged for now.

For now SV48 mode is left disabled by default, but

riscv: Add support for enabling SV48 mode

This increases the size of the user map from 256GB to 128TB. The kernel
map is left unchanged for now.

For now SV48 mode is left disabled by default, but can be enabled with a
tunable. Note that extant hardware does not implement SV48, but QEMU
does.

- In pmap_bootstrap(), allocate a L0 page and attempt to enable SV48
mode. If the write to SATP doesn't take, the kernel continues to run
in SV39 mode.
- Define VM_MAX_USER_ADDRESS to refer to the SV48 limit. In SV39 mode,
the region [VM_MAX_USER_ADDRESS_SV39, VM_MAX_USER_ADDRESS_SV48] is not
mappable.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34280

show more ...

riscv: Define a SV48 memory map

No functional change intended.

Reviewed by: kib, jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34273

riscv: Add various pmap definitions needed to support SV48 mode

No functional change intended.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https:/

riscv: Add various pmap definitions needed to support SV48 mode

No functional change intended.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34272

show more ...

riscv: handle page faults in the unmappable region

When handling a kernel page fault, check explicitly that stval resides
in either the user or kernel address spaces, and make the page fault
fatal i

riscv: handle page faults in the unmappable region

When handling a kernel page fault, check explicitly that stval resides
in either the user or kernel address spaces, and make the page fault
fatal if not. Otherwise, a properly crafted address may appear to
pmap_fault() as a valid and present page in the kernel map, causing the
page fault to be retried continuously. This is mainly due to the fact
that the upper bits of virtual addresses are not validated by most of
the pmap code.

Faults of this nature should only occur due to some kind of bug in the
kernel, but it is best to handle them gracefully when they do.

Handle user page faults in the same way, sending a SIGSEGV immediately
when a malformed address is encountered.

Add an assertion to pmap_l1(), which should help catch other bugs of
this kind that make it this far.

Reviewed by: jrtc27, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31208

show more ...

arm64, riscv: Set VM_KMEM_SIZE_SCALE to 1

This setting limits the amount of memory that can be allocated to UMA.
On systems with a direct map and ample KVA, however, there is no reason
for VM_KMEM_S

arm64, riscv: Set VM_KMEM_SIZE_SCALE to 1

This setting limits the amount of memory that can be allocated to UMA.
On systems with a direct map and ample KVA, however, there is no reason
for VM_KMEM_SIZE_SCALE to be larger than 1. This appears to have been
inherited from the 32-bit ARM platform definitions.

Also remove VM_KMEM_SIZE_MIN, which is not needed when
VM_KMEM_SIZE_SCALE is defined to be 1.[*]

Reviewed by: alc, kp, kib
Reported by: alc [*]
Submitted by: Klara, Inc.
Sponsored by: Ampere Computing
Differential Revision: https://reviews.freebsd.org/D28225

show more ...

arm64: fix early devmap assertion

The purpose of this KASSERT is to ensure that we do not run out of space
in the early devmap. However, the devmap grew beyond its initial size of
2MB in r336519, an

arm64: fix early devmap assertion

The purpose of this KASSERT is to ensure that we do not run out of space
in the early devmap. However, the devmap grew beyond its initial size of
2MB in r336519, and this assertion did not grow with it.

A devmap mapping of a 1080p framebuffer requires 1920x1080 bytes, or
1.977 MB, so it is just barely able to fit without triggering the
assertion, provided no other devices are mapped before it. With the
addition of `options GDB` in GENERIC by bbfa199cbc16, the uart is now
mapped for the purposes of a debug port, before mapping the framebuffer.
The presence of both these conditions pushes the selected virtual
address just below the threshold, triggering the assertion.

To fix this, use the correct size of the devmap, defined by
PMAP_MAPDEV_EARLY_SIZE. Since this code is shared with RISC-V, define
it for that platform as well (although it is a different size).

PR: 25241
Reported by: gbe
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

show more ...

Add a vmparam.h constant indicating pmap support for large pages.

Enable SHM_LARGEPAGE support on arm64.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc., Klara, Inc.
Differential Revisio

Add a vmparam.h constant indicating pmap support for large pages.

Enable SHM_LARGEPAGE support on arm64.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc., Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26467

show more ...

Move vm_page_dump bitset array definition to MI code

These definitions were repeated by all architectures, with small
variations. Consolidate the common definitons in machine
independent code and us

Move vm_page_dump bitset array definition to MI code

These definitions were repeated by all architectures, with small
variations. Consolidate the common definitons in machine
independent code and use bitset(9) macros for manipulation. Many
opportunities for deduplication remain in the machine dependent
minidump logic. The only intended functional change is increasing
the bit index type to vm_pindex_t, allowing the indexing of pages
with address of 8 TiB and greater.

Reviewed by: kib, markj
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D26129

show more ...

Add macros simplifying the fake preload setup

This is in preparation for booting via loader(8). Lift these macros from arm64
so we don't need to worry about the size when inserting new elements. Thi

Add macros simplifying the fake preload setup

This is in preparation for booting via loader(8). Lift these macros from arm64
so we don't need to worry about the size when inserting new elements. This
could have been done in r359673, but I didn't think I would be returning to
this function so soon.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D24910

show more ...

RISC-V: copy the DTB to early KVA

The location of the device-tree blob is passed to the kernel by the
previous booting stage (i.e. BBL or OpenSBI). Currently, we leave it
untouched and mark the 1MB

RISC-V: copy the DTB to early KVA

The location of the device-tree blob is passed to the kernel by the
previous booting stage (i.e. BBL or OpenSBI). Currently, we leave it
untouched and mark the 1MB of memory holding it as unavailable.

Instead, do what is done by other fake_preload_metadata() routines and
copy to the DTB to KVA space. This is more in line with what loader(8)
will provide us in the future, and it allows us to reclaim the hole in
physical memory.

Reviewed by: markj, kp (earlier version)
Differential Revision: https://reviews.freebsd.org/D24152

show more ...

Merge ^/head r352764 through r353315.

Eliminate an unused declaration. The variable in question is only defined
and used on sparc64.

MFC after: 1 week

MFhead@r344270

Merge ^/head r343956 through r344177.

Implement transparent 2MB superpage promotion for RISC-V.

This includes support for pmap_enter(..., psind=1) as described in the
commit log message for r321378.

The changes are largely modelled aft

Implement transparent 2MB superpage promotion for RISC-V.

This includes support for pmap_enter(..., psind=1) as described in the
commit log message for r321378.

The changes are largely modelled after amd64. arm64 has more stringent
requirements around superpage creation to avoid the possibility of TLB
conflict aborts, and these requirements do not apply to RISC-V, which
like amd64 permits simultaneous caching of 4KB and 2MB translations for
a given page. RISC-V's PTE format includes only two software bits, and
as these are already consumed we do not have an analogue for amd64's
PG_PROMOTED. Instead, pmap_remove_l2() always invalidates the entire
2MB address range.

pmap_ts_referenced() is modified to clear PTE_A, now that we support
both hardware- and software-managed reference and dirty bits. Also
fix pmap_fault_fixup() so that it does not set PTE_A or PTE_D on kernel
mappings.

Reviewed by: kib (earlier version)
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18863
Differential Revision: https://reviews.freebsd.org/D18864
Differential Revision: https://reviews.freebsd.org/D18865
Differential Revision: https://reviews.freebsd.org/D18866
Differential Revision: https://reviews.freebsd.org/D18867
Differential Revision: https://reviews.freebsd.org/D18868

show more ...

Merge ^/head r340918 through r341763.