amd64: add LASS support

In short, LASS enforces all kernel memory accesses to have bit 63 set to
1, and all userspace accesses have bit 63 set to 0. Violations of these
rules cause #GP. There are n

amd64: add LASS support

In short, LASS enforces all kernel memory accesses to have bit 63 set to
1, and all userspace accesses have bit 63 set to 0. Violations of these
rules cause #GP. There are natural loopholes, like SMAP with rflags.AC=1
allows kernel to access userspace.

Enablement is simple, we need to set CR4.LASS bit on all CPUs. There
are complications when kernel has to execute code at low addresses, e.g.
for la57 trampoline, or calling into EFI RT. The patch turns CR4.LASS
off around these regions.

LASS is officially documented in SDM, since at least rev. 085, October
2024. Tested in simics.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D55218

show more ...

amd64: Fix sys/pcpu.h usage in vmm_host.h and md_var.h

Include sys/pcpu in vmm_host.h as its structs and functions are used
there, and add a forward declaration of struct pcpu to md_var.h as it
is u

amd64: Fix sys/pcpu.h usage in vmm_host.h and md_var.h

Include sys/pcpu in vmm_host.h as its structs and functions are used
there, and add a forward declaration of struct pcpu to md_var.h as it
is used in some function prototypes.

Reviewed by: corvink, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D51550

show more ...

amd64: add wrmsr_early_safe(9)

The variant of wrmsr_safe(9) that might work before IDT and curpcb are
initialized. Assumes BSP, and that all APs are parked.

Before calling wrmsr_early_safe(), the

amd64: add wrmsr_early_safe(9)

The variant of wrmsr_safe(9) that might work before IDT and curpcb are
initialized. Assumes BSP, and that all APs are parked.

Before calling wrmsr_early_safe(), the wrmsr_early_safe_start() should
be called, afterward wrmsr_early_safe_end() restores the bootenv IDT.

Reviewed by: markj
Tested by: glebius
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D52607

show more ...

amd64: make LA57 mode for a process opt-in by default

The vm.pmap.prefer_la48 manages the mode.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: http

amd64: make LA57 mode for a process opt-in by default

The vm.pmap.prefer_la48 manages the mode.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D49913

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

amd64: be more precise when enabling the AlderLake small core PCID workaround

In particular, do not enable the workaround if INVPCID is not supported
by the core.

Reported by: "Chen, Alvin W" <Weik

amd64: be more precise when enabling the AlderLake small core PCID workaround

In particular, do not enable the workaround if INVPCID is not supported
by the core.

Reported by: "Chen, Alvin W" <Weike.Chen@Dell.com>
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D37940

show more ...

kasan: Create a shadow for the bootstack prior to hammer_time()

When the kernel is compiled with -asan-stack=true, the address sanitizer
will emit inline accesses to the shadow map. In other words,

kasan: Create a shadow for the bootstack prior to hammer_time()

When the kernel is compiled with -asan-stack=true, the address sanitizer
will emit inline accesses to the shadow map. In other words, some
shadow map accesses are not intercepted by the KASAN runtime, so they
cannot be disabled even if the runtime is not yet initialized by
kasan_init() at the end of hammer_time().

This went unnoticed because the loader will initialize all PML4 entries
of the bootstrap page table to point to the same PDP page, so early
shadow map accesses do not raise a page fault, though they are silently
corrupting memory. In fact, when the loader does not copy the staging
area, we do get a page fault since in that case only the first and last
PML4Es are populated by the loader. But due to another bug, the loader
always treated KASAN kernels as non-relocatable and thus always copied
the staging area.

It is not really practical to annotate hammer_time() and all callees
with __nosanitizeaddress, so instead add some early initialization which
creates a shadow for the boot stack used by hammer_time(). This is only
needed by KASAN, not by KMSAN, but the shared pmap code handles both.

Reported by: mhorne
Reviewed by: kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35449

show more ...

amd64: -m32 support for machine/md_var.h

Install the i386 md_var.h under /usr/include/i386 on amd64 and include
when targeting i386.

This is a mostly kernel-only header required by procstat's ZFS s

amd64: -m32 support for machine/md_var.h

Install the i386 md_var.h under /usr/include/i386 on amd64 and include
when targeting i386.

This is a mostly kernel-only header required by procstat's ZFS support.
It is pulled in by the i386 machine/counter.h.

Reviewed by: jhb, imp

show more ...

Unstaticize {get,set}_fpcontext() on amd64

This will be used to fix Linux signal delivery.

Discussed With: kib
Sponsored By: EPSRC

amd64: do not assume that kernel is loaded at 2M physical

Allow any 2M aligned contiguous location below 4G for the staging
area location. It should still be mapped by loader at KERNBASE.

The assu

amd64: do not assume that kernel is loaded at 2M physical

Allow any 2M aligned contiguous location below 4G for the staging
area location. It should still be mapped by loader at KERNBASE.

The assumption kernel makes about loader->kernel handoff with regard to
the MMU programming are explicitly listed at the beginning of hammer_time(),
where kernphys is calculated. Now kernphys is the variable instead of
symbol designating the physical address.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31121

show more ...

amd64: make efi_boot global

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31121

amd64: retire sse2_pagezero

All page zeroing is using temporal stores with rep movs*, the routine is
unused for several years.

Should a need arise for zeroing using non-temporal stores, a more
opti

amd64: retire sse2_pagezero

All page zeroing is using temporal stores with rep movs*, the routine is
unused for several years.

Should a need arise for zeroing using non-temporal stores, a more
optimized variant can be implemented with a more descriptive name.

show more ...

amd64: Store full 64bit of FIP/FDP for 64bit processes when using XSAVE.

If current process is 64bit, use rex-prefixed version of XSAVE
(XSAVE64). If current process is 32bit and CPU supports savin

amd64: Store full 64bit of FIP/FDP for 64bit processes when using XSAVE.

If current process is 64bit, use rex-prefixed version of XSAVE
(XSAVE64). If current process is 32bit and CPU supports saving
segment registers cs/ds in the FPU save area, use non-prefixed variant
of XSAVE.

Reported and tested by: Michał Górny <mgorny@mgorny@moritz.systems>
PR: 250043
Reviewed by: emaste, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D26643

show more ...

Move ctx_switch_xsave declaration to amd64 md_var.h.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days

Move vm_page_dump bitset array definition to MI code

These definitions were repeated by all architectures, with small
variations. Consolidate the common definitons in machine
independent code and us

Move vm_page_dump bitset array definition to MI code

These definitions were repeated by all architectures, with small
variations. Consolidate the common definitons in machine
independent code and use bitset(9) macros for manipulation. Many
opportunities for deduplication remain in the machine dependent
minidump logic. The only intended functional change is increasing
the bit index type to vm_pindex_t, allowing the indexing of pages
with address of 8 TiB and greater.

Reviewed by: kib, markj
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D26129

show more ...

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

amd64 pmap: LA57 AKA 5-level paging

Since LA57 was moved to the main SDM document with revision 072, it
seems that we should have a support for it, and silicons are coming.

This patch makes pmap su

amd64 pmap: LA57 AKA 5-level paging

Since LA57 was moved to the main SDM document with revision 072, it
seems that we should have a support for it, and silicons are coming.

This patch makes pmap support both LA48 and LA57 hardware. The
selection of page table level is done at startup, kernel always
receives control from loader with 4-level paging. It is not clear how
UEFI spec would adapt LA57, for instance it could hand out control in
LA57 mode sometimes.

To switch from LA48 to LA57 requires turning off long mode, requesting
LA57 in CR4, then re-entering long mode. This is somewhat delicate
and done in pmap_bootstrap_la57(). AP startup in LA57 mode is much
easier, we only need to toggle a bit in CR4 and load right value in CR3.

I decided to not change kernel map for now. Single PML5 entry is
created that points to the existing kernel_pml4 (KML4Phys) page, and a
pml5 entry to create our recursive mapping for vtopte()/vtopde().
This decision is motivated by the fact that we cannot overcommit for
KVA, so large space there is unusable until machines start providing
wider physical memory addressing. Another reason is that I do not
want to break our fragile autotuning, so the KVA expansion is not
included into this first step. Nice side effect is that minidumps are
compatible.

On the other hand, (very) large address space is definitely
immediately useful for some userspace applications.

For userspace, numbering of pte entries (or page table pages) is
always done for 5-level structures even if we operate in 4-level mode.
The pmap_is_la57() function is added to report the mode of the
specified pmap, this is done not to allow simultaneous 4-/5-levels
(which is not allowed by hw), but to accomodate for EPT which has
separate level control and in principle might not allow 5-leve EPT
despite x86 paging supports it. Anyway, it does not seems critical to
have 5-level EPT support now.

Tested by: pho (LA48 hardware)
Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D25273

show more ...

amd64: move pcb out of kstack to struct thread.

This saves 320 bytes of the precious stack space.

The only negative aspect of the change I can think of is that the
struct thread increased by 320 by

amd64: move pcb out of kstack to struct thread.

This saves 320 bytes of the precious stack space.

The only negative aspect of the change I can think of is that the
struct thread increased by 320 bytes obviously, and that 320 bytes are
not swapped out anymore. I believe the freed stack space is much more
important than that. Also, current struct thread size is 1392 bytes
on amd64, so UMA will allocate two thread structures per (4KB) slab,
which leaves a space for pcb without increasing zone memory use.

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D22138

show more ...

Merge ^/head r351317 through r351731.

amd64: rework PCPU allocation

Move pcpu KVA out of .bss into dynamically allocated VA at
pmap_bootstrap(). This avoids demoting superpage mapping .data/.bss.
Also it makes possible to use pmap_qent

amd64: rework PCPU allocation

Move pcpu KVA out of .bss into dynamically allocated VA at
pmap_bootstrap(). This avoids demoting superpage mapping .data/.bss.
Also it makes possible to use pmap_qenter() for installation of
domain-local pcpu page on NUMA configs.

Refactor pcpu and IST initialization by moving it to helper functions.

Reviewed by: markj
Tested by: pho
Discussed with: jeff
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D21320

show more ...

Merge ^/head r339813 through r340125.

Add pci_early function to detect Intel stolen memory.

On some Intel devices BIOS does not properly reserve memory (called
"stolen memory") for the GPU. If the stolen memory is claimed by the
OS, fu

Add pci_early function to detect Intel stolen memory.

On some Intel devices BIOS does not properly reserve memory (called
"stolen memory") for the GPU. If the stolen memory is claimed by the
OS, functions that depend on stolen memory (like frame buffer
compression) can't be used.

A function called pci_early_quirks that is called before the virtual
memory system is started was added. In Linux, this PCI early quirks
function iterates through all PCI slots to check for any device that
require quirks. While this more generic solution is preferable I only
ported the Intel graphics specific parts because I think my
implementation would be too similar to Linux GPL'd solution after
looking at the Linux code too much.

The code regarding Intel graphics stolen memory was ported from
Linux. In the case of Intel graphics stolen memory this
pci_early_quirks will read the stolen memory base and size from north
bridge registers. The values are stored in global variables that is
later read by linuxkpi_gplv2. Linuxkpi stores these values in a
Linux-specific structure that is read by the drm driver.

Relevant linuxkpi code is here:
https://github.com/FreeBSDDesktop/kms-drm/blob/drm-v4.16/linuxkpi/gplv2/src/linux_compat.c#L37

For now, only amd64 arch is suppor ted since that is the only arch
supported by the new drm drivers. I was told that Intel GPUs are
always located on 0:2:0 so these values are hard coded for now.

Note that the structure and early execution of the detection code is
not required in its current form, but we expect that the code will be
added shortly which fixes the potential BIOS bugs by reserving the
stolen range in phys_avail[]. This must be done as early as possible
to avoid conflicts with the potential usage of the memory in kernel.

Submitted by: Johannes Lundberg <johalun0@gmail.com>
Reviewed by: bwidawsk, imp
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D16719
Differential revision: https://reviews.freebsd.org/D17775

show more ...

Merge ^/head r339015 through r339669.

amd64: flush L1 data cache on syscall return with an error.

The knob allows to select the flushing mode or turn it off/on. The
idea, as well as the list of the ignored syscall errors, were taken
fr

amd64: flush L1 data cache on syscall return with an error.

The knob allows to select the flushing mode or turn it off/on. The
idea, as well as the list of the ignored syscall errors, were taken
from https://www.openwall.com/lists/kernel-hardening/2018/10/11/10 .

I was not able to measure statistically significant difference between
flush enabled vs disabled using syscall_timing getuid.

Reviewed by: bwidawsk
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D17536

show more ...

Merge ^/head r338026 through r338297, and resolve conflicts.