amd64: do not handle fs/gs bases conditionally on the selector

For machines with RDFSBASE support, we only saved and restored fs and gs
base registers when corresponding segment register contained t

amd64: do not handle fs/gs bases conditionally on the selector

For machines with RDFSBASE support, we only saved and restored fs and gs
base registers when corresponding segment register contained the
predefined value. This breaks some valid uses of the LDT together with
WR{F,G}SBASE.

Unconditionally save bases, and restore them on return to usermode. The
PCB_FULL_IRET optimization is still in place.

Reviewed by: markj, olce
Reported and tested: Alex S <iwtcex@gmail.com>
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D50414

show more ...

Remove stray whitespaces from sys/amd64/

Signed-off-by: Joshua Rogers <Joshua@Joshua.Hu>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1418

exception: Fix typos

Signed-off-by: Elyes Haouas <ehaouas@noos.fr>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/885

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

AST: rework

Make most AST handlers dynamically registered. This allows to have
subsystem-specific handler source located in the subsystem files,
instead of making subr_trap.c aware of it. For inst

AST: rework

Make most AST handlers dynamically registered. This allows to have
subsystem-specific handler source located in the subsystem files,
instead of making subr_trap.c aware of it. For instance, signal
delivery code on return to userspace is now moved to kern_sig.c.

Also, it allows to have some handlers designated as the cleanup (kclear)
type, which are called both at AST and on thread/process exit. For
instance, ast(), exit1(), and NFS server no longer need to be aware
about UFS softdep processing.

The dynamic registration also allows third-party modules to register AST
handlers if needed. There is one caveat with loadable modules: the
code does not make any effort to ensure that the module is not unloaded
before all threads processed through AST handler in it. In fact, this
is already present behavior for hwpmc.ko and ufs.ko. I do not think it
is worth the efforts and the runtime overhead to try to fix it.

Reviewed by: markj
Tested by: emaste (arm64), pho
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D35888

show more ...

amd64: remove lfence after swapgs on syscall entry

According to the description of SBSS issue at
https://software.intel.com/content/www/us/en/develop/articles/software-security-guidance/technical-do

amd64: remove lfence after swapgs on syscall entry

According to the description of SBSS issue at
https://software.intel.com/content/www/us/en/develop/articles/software-security-guidance/technical-documentation/speculative-behavior-swapgs-and-segment-registers.html
lfence after swapgs is needed only for the case when swapgs could be
speculatively executed. Since syscall entry, unlike exception and
interrupt entries, executes swapgs unconditionally, there is no
opportunity for speculation.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31682

show more ...

amd64: Add MD bits for KMSAN

Interrupt and exception handlers must call kmsan_intr_enter() prior to
calling any C code. This is because the KMSAN runtime maintains some
TLS in order to track initia

amd64: Add MD bits for KMSAN

Interrupt and exception handlers must call kmsan_intr_enter() prior to
calling any C code. This is because the KMSAN runtime maintains some
TLS in order to track initialization state of function parameters and
return values across function calls. Then, to ensure that this state is
kept consistent in the face of asynchronous kernel-mode excpeptions, the
runtime uses a stack of TLS blocks, and kmsan_intr_enter() and
kmsan_intr_leave() push and pop that stack, respectively.

Use these functions in amd64 interrupt and exception handlers. Note
that handlers for user->kernel transitions need not be annotated.

Also ensure that trap frames pushed by the CPU and by handlers are
marked as initialized before they are used.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31467

show more ...

x86: remove gcov kernel support

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D29529

amd64 nmi handler: fix comment about %ebx

Reported by: markj
MFC after: 3 days

amd64: preserve %cr2 in NMI/MCE/DBG handlers.

These handlers could interrupt code which has interrupts disabled,
and if a spurious page fault occurs during exception handler run,
we get clobbered %c

amd64: preserve %cr2 in NMI/MCE/DBG handlers.

These handlers could interrupt code which has interrupts disabled,
and if a spurious page fault occurs during exception handler run,
we get clobbered %cr2 in higher level stack.

This is mostly a speculation, but it is based on hints from good sources.

MFC after: 1 week
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27772

show more ...

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

Simplify non-pti syscall entry on amd64.

Limit manipulations to use %rax as scratch to the pti portion of the
syscall entry code.

Submitted by: alc
Reviewed by: markj
MFC after: 1 week
Differential

Simplify non-pti syscall entry on amd64.

Limit manipulations to use %rax as scratch to the pti portion of the
syscall entry code.

Submitted by: alc
Reviewed by: markj
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D25722

show more ...

amd64 pmap: microoptimize local shootdowns for PCID PTI configurations

When pmap operates in PTI mode, we must reload %cr3 on return to
userspace. In non-PCID mode the reload always flushes all non

amd64 pmap: microoptimize local shootdowns for PCID PTI configurations

When pmap operates in PTI mode, we must reload %cr3 on return to
userspace. In non-PCID mode the reload always flushes all non-global
TLB entries and we take advantage of it by only invalidating the KPT
TLB entries (there is no cached UPT entries at all).

In PCID mode, we flush both KPT and UPT TLB explicitly, but we can
take advantage of the fact that PCID mode command to reload %cr3
includes a flag to flush/not flush target TLB. In particular, we can
avoid the flush for UPT, instead record that load of pc_ucr3 into %cr3
on return to usermode should be flushing. This is done by providing
either all-1s or ~CR3_PCID_MASK in pc_ucr3_load_mask. The mask is
automatically reset to all-1s on return to usermode.

Similarly, we can avoid flushing UPT TLB on context switch, replacing
it by setting pc_ucr3_load_mask. This unifies INVPCID and non-INVPCID
PTI ifunc, leaving only 4 cases instead of 6. This trick is also
applicable both to the TLB shootdown IPI handlers, since handlers
interrupt the target thread.

But then we need to check pc_curpmap in handlers, and this would
reopen the same race for INVPCID machines as was fixed in r306350 for
non-INVPCID. To not introduce the same bug, unconditionally do
spinlock_enter() in pmap_activate().

Reviewed by: alc, markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
Differential revision: https://reviews.freebsd.org/D25483

show more ...

amd64: in double fault handler, do not rely on sane gsbase value.

Typical reasons for doublefault faults are either kernel stack
overflow or bugs in the code that manipulates protection CPU state.
T

amd64: in double fault handler, do not rely on sane gsbase value.

Typical reasons for doublefault faults are either kernel stack
overflow or bugs in the code that manipulates protection CPU state.
The later code is the code which often has to set up gsbase for
kernel. Switching to explicit load of GSBASE MSR in the fault handler
makes it more probable to output a useful information.

Now all IST handlers have nmi_pcpu structure on top of their stacks.

It would be even more useful to save gsbase value at the moment of the
fault. I did not this because I do not want to modify PCB layout now.

Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

amd64: only set PCB_FULL_IRET pcb flag when #gp or similar exception comes
from usermode.

If CPU supports RDFSBASE, the flag also means that userspace fsbase
and gsbase are already written into pcb,

amd64: only set PCB_FULL_IRET pcb flag when #gp or similar exception comes
from usermode.

If CPU supports RDFSBASE, the flag also means that userspace fsbase
and gsbase are already written into pcb, which might be not true when
we handle #gp from kernel.

The offender is rdmsr_safe(), and the visible result is corrupted
userspace TLS base.

Reported by: pstef
Sponsored by: The FreeBSD Foundation
MFC after: 3 days

show more ...

amd64: Store %cr3 into pcpu saved_ucr3 on double fault.

Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

amd64: prevents speculations over swapgs reload of %gs base.

Such speculations could use user-controlled %gs base, esp. since
FreeBSD supports WRGSBASE instructions.

Place LFENCEs on entry for each

amd64: prevents speculations over swapgs reload of %gs base.

Such speculations could use user-controlled %gs base, esp. since
FreeBSD supports WRGSBASE instructions.

Place LFENCEs on entry for each basic block after the test for
previous kernel/user mode on the kernel entry, which prevents the
speculation. Code accesses %gs-based PCPU before any serialization
instructions are executed, like %cr3 reload for KPTI.

With pti disabled, on haswell i7-4770S machine, "syscall_timings getppid"
shows when no lfence is added to syscall path:
test loop time iterations periteration
getppid 0 1.040918865 4643611 0.000000224
getppid 1 1.004985962 4481816 0.000000224
getppid 2 1.005196483 4482363 0.000000224
with lfence:
getppid 0 1.043701091 4554779 0.000000229
getppid 1 1.016930328 4438094 0.000000229
getppid 2 1.023223117 4466640 0.000000229
and ministat reports 'No difference proven at 95.0% confidence.'

Security: CVE-2019-1125
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

show more ...

amd64: Streamline exceptions and interrupts handlers.

PTI-mode entry points were coded to set up the environment identical
to non-PTI entry and then fall-through to non-PTI handlers, mostly.
This ha

amd64: Streamline exceptions and interrupts handlers.

PTI-mode entry points were coded to set up the environment identical
to non-PTI entry and then fall-through to non-PTI handlers, mostly.
This has the drawback of requiring two more SWAPGS, first to access
PCPU, and then to return to the state expected by the non-PTI entry
point.

Eliminate the duplication by doing more in entry stubs both for PTI
and non-PTI, and adjusting the common code to expect that SWAPGS and
some minimal registers saving is done by entries.

Some less often used entries, in particular, #GP, #NP, and #SS, which
can fault on doreti, are left as is because there are basically four
variants of entrance, and they are not performance-critical,
esp. comparing with e.g. #PF or interrupts.

Reviewed by: markj (previous version)
Tested by: pho (previous version)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

show more ...

MFHead @348740

Sponsored by: The FreeBSD Foundation

Mitigations for Microarchitectural Data Sampling.

Microarchitectural buffers on some Intel processors utilizing
speculative execution may allow a local process to obtain a memory
disclosure. An att

Mitigations for Microarchitectural Data Sampling.

Microarchitectural buffers on some Intel processors utilizing
speculative execution may allow a local process to obtain a memory
disclosure. An attacker may be able to read secret data from the
kernel or from a process when executing untrusted code (for example,
in a web browser).

Reference: https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00233.html
Security: CVE-2018-12126, CVE-2018-12127, CVE-2018-12130, CVE-2019-11091
Security: FreeBSD-SA-19:07.mds
Reviewed by: jhb
Tested by: emaste, lwhsu
Approved by: so (gtetlow)

show more ...

Merge ^/head r343712 through r343806.

amd64: clear callee-preserved registers on syscall exit.

%r8, %r10, and on non-KPTI configuration %r9 were not restored on fast
return from a syscall.

Reviewed by: markj
Approved by: so
Security: C

amd64: clear callee-preserved registers on syscall exit.

%r8, %r10, and on non-KPTI configuration %r9 were not restored on fast
return from a syscall.

Reviewed by: markj
Approved by: so
Security: CVE-2019-5595
Sponsored by: The FreeBSD Foundation
MFC after: 0 minutes

show more ...

Merge ^/head r338026 through r338297, and resolve conflicts.

Merge ^/head r336870 through r337285, and resolve conflicts.

amd64: do not handle fs/gs bases conditionally on the selector

For machines with RDFSBASE support, we only saved and restored fs and gs
base registers when corresponding segment register contained t

amd64: do not handle fs/gs bases conditionally on the selector

For machines with RDFSBASE support, we only saved and restored fs and gs
base registers when corresponding segment register contained the
predefined value. This breaks some valid uses of the LDT together with
WR{F,G}SBASE.

Unconditionally save bases, and restore them on return to usermode. The
PCB_FULL_IRET optimization is still in place.

Reviewed by: markj, olce
Reported and tested: Alex S <iwtcex@gmail.com>
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D50414

show more ...