vmm: Add ability to destroy VMs on close

This change adds the ability to tie a virtual machine's lifecycle to
a /dev/vmmctl file descriptor. A user can request `vmmctl` to destroy a
virtual machine

vmm: Add ability to destroy VMs on close

This change adds the ability to tie a virtual machine's lifecycle to
a /dev/vmmctl file descriptor. A user can request `vmmctl` to destroy a
virtual machine on close using the `VMMCTL_CREATE_DESTROY_ON_CLOSE` flag
when creating the virtual machine. `vmmctl` tracks such virtual machines
in per-descriptor lists.

Differential Revision: https://reviews.freebsd.org/D53729
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
MFC after: 3 months

show more ...

libvmmapi: Add support for setting up and configuring guest NUMA domains

This patch reworks libvmmapi to provide support for emulating NUMA
domains in guests.

More specifically, it reworks 'vm_setu

libvmmapi: Add support for setting up and configuring guest NUMA domains

This patch reworks libvmmapi to provide support for emulating NUMA
domains in guests.

More specifically, it reworks 'vm_setup_memory' to setup system memory
segments for each guest NUMA domain.

An emulated NUMA domain is described by a 'struct vmdom' in vmmapi.h.
Aside from its size in bytes, each domain can be configured to use a
specific domainset(9) policy and domain mask.
'vm_setup_memory' now takes two additional arguments - an array of
struct vmdoms and the array's size. It then proceeds to set up a memory
segment for each specified domain using the existing memory mapping
scheme. If no domain info is passed, the memory setup falls back to
the original, non-NUMA behaviour.

Differential Revision: https://reviews.freebsd.org/D44566
Reviewed by: markj

show more ...

libvmmapi: Fix auto-loading of vmm.ko

- We should autoload vmm.ko when creating a VM with vm_openf(), to
preserve behaviour prior to commit 99127fd10362.
- kldload(2) returns a non-zero value upon

libvmmapi: Fix auto-loading of vmm.ko

- We should autoload vmm.ko when creating a VM with vm_openf(), to
preserve behaviour prior to commit 99127fd10362.
- kldload(2) returns a non-zero value upon success, so the existing code
was wrong.

Reviewed by: jhb
Reported by: olivier
Fixes: 99127fd10362 ("libvmmapi: Use the vmmctl device file to create and destroy VMs")
Differential Revision: https://reviews.freebsd.org/D48797

show more ...

libvmmapi: Use the vmmctl device file to create and destroy VMs

This deprecates the vm_create() and vm_open() interfaces and introduces
vm_openf(), which takes flags controlling its behaviour. In p

libvmmapi: Use the vmmctl device file to create and destroy VMs

This deprecates the vm_create() and vm_open() interfaces and introduces
vm_openf(), which takes flags controlling its behaviour. In particular,
it will optionally create a VM first, and it can optionally reinitialize
an existing VM. This enables some simplification of existing consumers.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D47030

show more ...

libvmmapi: Conditionalize compilation of some functions

Hide definitions of several functions that currently don't have
implementatations in the arm64 vmm port. In particular, add a
WITH_VMMAPI_SNA

libvmmapi: Conditionalize compilation of some functions

Hide definitions of several functions that currently don't have
implementatations in the arm64 vmm port. In particular, add a
WITH_VMMAPI_SNAPSHOT preprocessor variable that can be used to enable
compilation of save/restore functions, and conditionalize compilation of
some functions only used by amd64 bhyve. If in the long term they
remain amd64-only, they can move to vmmapi_machdep.c, but for now it's
not clear to me that that's the right thing to do.

MFC after: 2 weeks
Sponsored by: Innovate UK

show more ...

libvmmapi: Zero out the structure passed to VM_GET_MEMSEG

Avoid assuming that the kernel zeros the name buffer, it does not do
this for zero-length segments.

MFC after: 2 weeks
Sponsored by: Innova

libvmmapi: Zero out the structure passed to VM_GET_MEMSEG

Avoid assuming that the kernel zeros the name buffer, it does not do
this for zero-length segments.

MFC after: 2 weeks
Sponsored by: Innovate UK

show more ...

libvmmapi: Add arm64 support

- Define wrappers for some MD ioctls.
- Provide a list of vmm device ioctls for cap_ioctl_limit().
- Disable use of the lowmem region.

Reviewed by: corvink
MFC after: 2

libvmmapi: Add arm64 support

- Define wrappers for some MD ioctls.
- Provide a list of vmm device ioctls for cap_ioctl_limit().
- Disable use of the lowmem region.

Reviewed by: corvink
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41005

show more ...

libvmmapi: Make memory segment handling a bit more abstract

libvmmapi leaves a hole at [3GB, 4GB) in the guest physical address
space. This hole is not used in the arm64 port, which maps everything

libvmmapi: Make memory segment handling a bit more abstract

libvmmapi leaves a hole at [3GB, 4GB) in the guest physical address
space. This hole is not used in the arm64 port, which maps everything
above 4GB. This change makes the code a bit more general to accomodate
arm64 more naturally. In particular:

- Remove vm_set_lowmem_limit(): it is unused and doesn't have
well-defined constraints, e.g., nothing prevents a consumer from
setting a lowmem limit above the highmem base.
- Define a constant for the highmem base and use that everywhere that
the base is currently hard-coded.
- Make the lowmem limit a compile-time constant instead of a vmctx field.
- Store segment info in an array.
- Add vm_get_highmem_base(), for use in bhyve since the current value is
hard-coded in some places.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41004

show more ...

libvmmapi: Move PCI passthrough ioctl wrappers into a separate file

The arm64 port doesn't implement PCI passthrough and in particular
doesn't define the ioctls used by these wrappers. It might be

libvmmapi: Move PCI passthrough ioctl wrappers into a separate file

The arm64 port doesn't implement PCI passthrough and in particular
doesn't define the ioctls used by these wrappers. It might be that the
ppt ioctl interface will require modification to support arm64. Until
that's sorted out one way or another, put this code in a separate file
so that it's easy to conditionally compile.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41003

show more ...

libvmmapi: Move more amd64-specific ioctl wrappers to vmmapi_machdep.c

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: h

libvmmapi: Move more amd64-specific ioctl wrappers to vmmapi_machdep.c

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41002

show more ...

libvmmapi: Split the ioctl list into MI and MD lists

To enable use in capability mode, libvmmapi needs a list of all the
ioctls that might be invoked on the vmm device handle. Some of these
ioctls

libvmmapi: Split the ioctl list into MI and MD lists

To enable use in capability mode, libvmmapi needs a list of all the
ioctls that might be invoked on the vmm device handle. Some of these
ioctls are amd64-specific. Move the ioctl list to vmmapi_machdep.c and
define a list of MI ioctls so that the arm64 port can build its own list
without duplicating common ioctls. No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41001

show more ...

libvmmapi: Move VM capability names to vmmapi_machdep.c

Add some missing entries while here.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://re

libvmmapi: Move VM capability names to vmmapi_machdep.c

Add some missing entries while here.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41000

show more ...

libvmmapi: Move some ioctl wrappers to vmmapi_machdep.c

ioctls relating to segments and various x86-specific interrupt
controllers are easy candidates to move to vmmapi_machdep.c.

In vmmapi.h I'm j

libvmmapi: Move some ioctl wrappers to vmmapi_machdep.c

ioctls relating to segments and various x86-specific interrupt
controllers are easy candidates to move to vmmapi_machdep.c.

In vmmapi.h I'm just ifdefing MD prototypes for now. We could instead
split vmmapi.h into multiple headers, e.g., vmmapi.h and
vmmapi_machdep.h, but it's not obvious to me yet that that's the right
approach.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40999

show more ...

libvmmapi: Add a subdirectory for amd64-specific code

Move vmmapi_freebsd.c there. It contains x86-specific code used only by
bhyveload(8).

Move vcpu_reset() into vmmapi_machdep.c. It is also x86

libvmmapi: Add a subdirectory for amd64-specific code

Move vmmapi_freebsd.c there. It contains x86-specific code used only by
bhyveload(8).

Move vcpu_reset() into vmmapi_machdep.c. It is also x86-specific.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40998

show more ...

libvmmapi: add missing capability strings

Signed-off-by: Rob Norris <robn@despairlabs.com>

Reviewed by: markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D44642

lib: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

lib: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

libvmmapi: Remove some unneeded includes

These are amd64-specific and so can't be used when targetting arm64, but
they don't appear to be needed.

No functional change intended.

MFC after: 1 week
S

libvmmapi: Remove some unneeded includes

These are amd64-specific and so can't be used when targetting arm64, but
they don't appear to be needed.

No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

show more ...

vmm: Avoid embedding cpuset_t ioctl ABIs

Commit 0bda8d3e9f7a ("vmm: permit some IPIs to be handled by userspace")
embedded cpuset_t into the vmm(4) ioctl ABI. This was a mistake since
we otherwise

vmm: Avoid embedding cpuset_t ioctl ABIs

Commit 0bda8d3e9f7a ("vmm: permit some IPIs to be handled by userspace")
embedded cpuset_t into the vmm(4) ioctl ABI. This was a mistake since
we otherwise have some leeway to change the cpuset_t for the whole
system, but we want to keep the vmm ioctl ABI stable.

Rework IPI reporting to avoid this problem. Along the way, make VM_RUN
a bit more efficient:
- Split vmexit metadata out of the main VM_RUN structure. This data is
only written by the kernel.
- Have userspace pass a cpuset_t pointer and cpusetsize in the VM_RUN
structure, as is done for cpuset syscalls.
- Have the destination CPU mask for VM_EXITCODE_IPIs live outside the
vmexit info structure, and make VM_RUN copy it out separately. Zero
out any extra bytes in the CPU mask, like cpuset syscalls do.
- Modify the vmexit handler prototype to take a full VM_RUN structure.

PR: 271330
Reviewed by: corvink, jhb (previous versions)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40113

show more ...

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

bhyve: Remove vmctx member from struct vm_snapshot_meta.

This is a userland-only pointer that isn't relevant to the kernel and
doesn't belong in the ioctl structure shared between userland and the
k

bhyve: Remove vmctx member from struct vm_snapshot_meta.

This is a userland-only pointer that isn't relevant to the kernel and
doesn't belong in the ioctl structure shared between userland and the
kernel. For the kernel, the old structure for the ioctl is still
supported under COMPAT_FREEBSD13.

This changes vm_snapshot_req() in libvmmapi to accept an explicit
vmctx argument.

It also changes vm_snapshot_guest2host_addr to take an explicit vmctx
argument. As part of this change, move the declaration for this
function and its wrapper macro from vmm_snapshot.h to snapshot.h as it
is a userland-only API.

Reviewed by: corvink, markj
Differential Revision: https://reviews.freebsd.org/D38125

show more ...

libvmmapi: Add a struct vcpu and use it in most APIs.

This replaces the 'struct vm, int vcpuid' tuple passed to most API
calls and is similar to the changes recently made in vmm(4) in the
kernel.

s

libvmmapi: Add a struct vcpu and use it in most APIs.

This replaces the 'struct vm, int vcpuid' tuple passed to most API
calls and is similar to the changes recently made in vmm(4) in the
kernel.

struct vcpu is an opaque type managed by libvmmapi. For now it stores
a pointer to the VM context and an integer id.

As an immediate effect this removes the divergence between the kernel
and userland for the instruction emulation code introduced by the
recent vmm(4) changes.

Since this is a major change to the vmmapi API, bump VMMAPI_VERSION to
0x200 (2.0) and the shared library major version.

While here (and since the major version is bumped), remove unused
vcpu argument from vm_setup_pptdev_msi*().

Add new functions vm_suspend_all_cpus() and vm_resume_all_cpus() for
use by the debug server. The underyling ioctl (which uses a vcpuid of
-1) remains unchanged, but the userlevel API now uses separate
functions for global CPU suspend/resume.

Reviewed by: corvink, markj
Differential Revision: https://reviews.freebsd.org/D38124

show more ...

libvmm: add missing ioctl's to vm_ioctl_cmds

Reviewed by: corvink, markj
MFC after: 1 week
Sponsored by: vStack
Differential Revision: https://reviews.freebsd.org/D38866

vmm: Use struct vcpu in the instruction emulation code.

This passes struct vcpu down in place of struct vm and and integer
vcpu index through the in-kernel instruction emulation code. To
minimize u

vmm: Use struct vcpu in the instruction emulation code.

This passes struct vcpu down in place of struct vm and and integer
vcpu index through the in-kernel instruction emulation code. To
minimize userland disruption, helper macros are used for the vCPU
arguments passed into and through the shared instruction emulation
code.

A few other APIs used by the instruction emulation code have also been
updated to accept struct vcpu in the kernel including
vm_get/set_register and vm_inject_fault.

Reviewed by: corvink, markj
Differential Revision: https://reviews.freebsd.org/D37161

show more ...