ipsec offload: ipsec_accel_fill_xh() should indirect through fn pointer

The config with IPSEC_SUPPORT + IPSEC_OFFLOAD is the valid one.

Fixes: 5be5a0bde5f990dbc680272eee74132bcde815f2
Sponsored by:

ipsec offload: ipsec_accel_fill_xh() should indirect through fn pointer

The config with IPSEC_SUPPORT + IPSEC_OFFLOAD is the valid one.

Fixes: 5be5a0bde5f990dbc680272eee74132bcde815f2
Sponsored by: Nvidia networking

show more ...

machine/stdarg.h -> sys/stdarg.h

Switch to using sys/stdarg.h for va_list type and va_* builtins.

Make an attempt to insert the include in a sensible place. Where
style(9) was followed this is eas

machine/stdarg.h -> sys/stdarg.h

Switch to using sys/stdarg.h for va_list type and va_* builtins.

Make an attempt to insert the include in a sensible place. Where
style(9) was followed this is easy, where it was ignored, aim for the
first block of sys/*.h headers and don't get too fussy or try to fix
other style bugs.

Reviewed by: imp
Exp-run by: antoine (PR 286274)
Pull Request: https://github.com/freebsd/freebsd-src/pull/1595

show more ...

pfkey2: use correct value for a key length

The length of key data is specified via sadb_key_bits field.
Use specified size for buffer allocation and key copying.
Also add a check that the value of s

pfkey2: use correct value for a key length

The length of key data is specified via sadb_key_bits field.
Use specified size for buffer allocation and key copying.
Also add a check that the value of sadb_key_bits is not zero,
as explicitly required in RFC 2367.

PR: 241010
Submitted by: jean-francois.hren at stormshield eu
MFC after: 1 week

show more ...

netipsec: Pass the right mbuf up

Note that key_spdacquire() is dead code, as the SADB_X_SPDACQUIRE
message handler is not set.

PR: 243057
MFC after: 2 weeks

ipsec: Drain async ipsec_offload work when destroying a vnet

Re-apply commit e196b12f4d4d. This was reverted by commit 28294dc92476
because it could trigger a deadlock, but the underlying problem t

ipsec: Drain async ipsec_offload work when destroying a vnet

Re-apply commit e196b12f4d4d. This was reverted by commit 28294dc92476
because it could trigger a deadlock, but the underlying problem there
was fixed in commit f76826b892de.

Reported by: KASAN
Reviewed by: kib
Fixes: ef2a572bf6bd ("ipsec_offload: kernel infrastructure")
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D46483

show more ...

Revert "ipsec: Drain async ipsec_offload work when destroying a vnet"

This change can cause a deadlock in some cases, since it's possible for
VNET teardown to happen in the context of taskqueue_thre

Revert "ipsec: Drain async ipsec_offload work when destroying a vnet"

This change can cause a deadlock in some cases, since it's possible for
VNET teardown to happen in the context of taskqueue_thread, and
ipsec_accel_sync() drains taskqueue_thread's work queue.

This reverts commit e196b12f4d4d13982d42cf7841b77dca1405effa.

show more ...

ipsec: Drain async ipsec_offload work when destroying a vnet

The ipsec_offload code in some cases releases object references in an
asynchronous context where it needs to set the current VNET. Make

ipsec: Drain async ipsec_offload work when destroying a vnet

The ipsec_offload code in some cases releases object references in an
asynchronous context where it needs to set the current VNET. Make sure
that all such work completes before the VNET is actually destroyed,
otherwise a use-after-free is possible.

Reported by: KASAN
Reviewed by: kib
Fixes: ef2a572bf6bd ("ipsec_offload: kernel infrastructure")
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D46483

show more ...

ipsec_offload: add ipsec_accel_drv_sa_lifetime_fetch()

A function to fetch hardware counters for offloaded SA on specific
interface.

Sponsored by: NVidia networking

ipsec_offload: indirect two more functions on the ipsec.ko module load

Specifically, ipsec_accel_on_ifdown() and ipsec_accel_drv_sa_lifetime_update()
should be present in kernel for future mlx5en dr

ipsec_offload: indirect two more functions on the ipsec.ko module load

Specifically, ipsec_accel_on_ifdown() and ipsec_accel_drv_sa_lifetime_update()
should be present in kernel for future mlx5en driver to be statically
linkable into the kernel built with IPSEC_HOOKS + IPSEC_OFFLOAD.

Sponsored by: NVIDIA networking

show more ...

ipsec_offload: kernel infrastructure

Inline IPSEC offload moves almost whole IPSEC processing from the
CPU/MCU and possibly crypto accelerator, to the network card.

The transmitted packet content i

ipsec_offload: kernel infrastructure

Inline IPSEC offload moves almost whole IPSEC processing from the
CPU/MCU and possibly crypto accelerator, to the network card.

The transmitted packet content is not touched by CPU during TX
operations, kernel only does the required policy and security
association lookups to find out that given flow is offloaded, and then
packet is transmitted as plain text to the card. For driver convenience,
a metadata is attached to the packet identifying SA which must process
the packet. Card does encryption of the payload, padding, calculates
authentication, and does the reformat according to the policy.

Similarly, on receive, card does the decapsulation, decryption, and
authentification. Kernel receives the identifier of SA that was
used to process the packet, together with the plain-text packet.

Overall, payload octets are only read or written by card DMA engine,
removing a lot of memory subsystem overhead, and saving CPU time because
IPSEC algos calculations are avoided.

If driver declares support for inline IPSEC offload (with the
IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct
if_ipsec_accel_methods), kernel offers the SPD and SAD to driver.
Driver decides which policies and SAs can be offloaded based on
hardware capacity, and acks/nacks each SA for given interface to
kernel. Kernel needs to keep this information to make a decision to
skip software processing on TX, and to assume processing already done
on RX. This shadow SPD/SAD database of offloads is rooted from
policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs
(struct secasvar accel_ipfs, struct ifp_handle_sav).

Some extensions to the PF_KEY socket allow to limit interfaces for
which given SP/SA could be offloaded (proposed for offload). Also,
additional statistics extensions allow to observe allocation/octet/use
counters for specific SA.

Since SPs and SAs are typically instantiated in non-sleepable context,
while offloading them into card is expected to require costly async
manipulations of the card state, calls to the driver for offload and
termination are executed in the threaded taskqueue. It also solves
the issue of allocating resources needed for the offload database.
Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the
owning SP/SA, the offload must be terminated before last reference is
dropped. ipsec_accel only adds transient references to ensure safe
pointer ownership by taskqueue.

Maintaining the SA counters for hardware-accelerated packets is the
duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update()
is provided to hide accel infrastructure from drivers which would use
expected callout to query hardware periodically for updates.

Reviewed by: rscheff (transport, stack integration), np
Sponsored by: NVIDIA networking
Differential revision: https://reviews.freebsd.org/D44219

show more ...

ipsec: make key_do_allocsp() global

Sponsored by: NVIDIA networking

Add UDP encapsulation of ESP in IPv6

This patch provides UDP encapsulation of ESP packets over IPv6.
Ports the IPv4 code to IPv6 and adds support for IPv6 in udpencap.c
As required by the RFC and un

Add UDP encapsulation of ESP in IPv6

This patch provides UDP encapsulation of ESP packets over IPv6.
Ports the IPv4 code to IPv6 and adds support for IPv6 in udpencap.c
As required by the RFC and unlike in IPv4 encapsulation,
UDP checksums are calculated.

Co-authored-by: Aurelien Cazuc <aurelien.cazuc.external@stormshield.eu>
Sponsored-by: Stormshield
Sponsored-by: Wiktel
Sponsored-by: Klara, Inc.

show more ...

sys: Remove $FreeBSD$: one-line .c comment pattern

Remove /^/[*/]\s*\$FreeBSD\$.*\n/

ipsec: Make algorithm tables read-only

No functional change intended.

MFC after: 1 week

netipsec/key.c: use designated initializers for arrays

Also de-expand nitems() use in related asserts, and fix maxsize array
name in the assert message.

Sponsored by: NVidia networking

netipsec: some style

Sponsored by: NVidia networking

ipsec: only update lastused when it changes

to limit cache-line bouncing.

Note that as there is no atomic_store we are hoping the compiler wont
speculatively do the store. It is not employed becaus

ipsec: only update lastused when it changes

to limit cache-line bouncing.

Note that as there is no atomic_store we are hoping the compiler wont
speculatively do the store. It is not employed because the size depends
on target arch.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D38433

show more ...

ipsec: Clear pad bytes in PF_KEY messages

Various handlers for SADB messages will allocate a new mbuf and populate
some structures in it. Some of these structures, such as struct
sadb_supported, co

ipsec: Clear pad bytes in PF_KEY messages

Various handlers for SADB messages will allocate a new mbuf and populate
some structures in it. Some of these structures, such as struct
sadb_supported, contain small reserved fields that are not initialized
and are thus leaked to userspace.

Fix the problem by adding a helper to allocate zeroed mbufs. This
reduces code duplication and the overhead of zeroing these messages
isn't harmful.

Reviewed by: zlei, melifaro
Reported by: KMSAN
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D38068

show more ...

ipsec: add key_havesp_any

Saves on work in a common case of checking both directions.

Note further work in the area is impending to elide these in the common
case to begin with.

Sponsored by: Rubi

ipsec: add key_havesp_any

Saves on work in a common case of checking both directions.

Note further work in the area is impending to elide these in the common
case to begin with.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D36485

show more ...

ipsec: prohibit unknown directions in key_havesp

Eliminates a branch checking for its validity.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.

ipsec: prohibit unknown directions in key_havesp

Eliminates a branch checking for its validity.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D36485

show more ...

ipsec: add support for CHACHA20POLY1305

Based on a patch by ae@.

Reviewed by: gbe (man page), pauamma (man page)
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https:/

ipsec: add support for CHACHA20POLY1305

Based on a patch by ae@.

Reviewed by: gbe (man page), pauamma (man page)
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D37180

show more ...

keysock: do not use raw socket code

This makes key socket implementation self contained and removes one
of the last dependencies on the raw socket code and pr_output method.

There are very subtle A

keysock: do not use raw socket code

This makes key socket implementation self contained and removes one
of the last dependencies on the raw socket code and pr_output method.

There are very subtle API visible changes:
- now key socket would return EOPNOTSUPP instead of EINVAL on
syscalls that are not supposed to be called on a key socket.
- key socket buffer sizes are now controlled by net.key sysctls instead
of net.raw. The latter were not documented anywhere, and even Internet
search doesn't find any references or discussions related to them.

Reviewed by: melifaro
Differential revision: https://reviews.freebsd.org/D36123

show more ...

Adjust function definitions in netipsec's key.c to avoid clang 15 warnings

With clang 15, the following -Werror warnings are produced:

sys/netipsec/key.c:6432:15: error: a function declaration

Adjust function definitions in netipsec's key.c to avoid clang 15 warnings

With clang 15, the following -Werror warnings are produced:

sys/netipsec/key.c:6432:15: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
key_getcomb_ah()
^
void
sys/netipsec/key.c:6489:19: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
key_getcomb_ipcomp()
^
void

This is because key_getcomb_ah() and key_getcomb_ipcomp() are declared
with (void) argument lists, but defined with empty argument lists. Make
the definitions match the declarations.

MFC after: 3 days

show more ...

ipsec: replace SECASVAR mtx by rmlock

This mutex is a significant point of contention in the ipsec code, and
can be relatively trivially replaced by a read-mostly lock.
It does require a separate l

ipsec: replace SECASVAR mtx by rmlock

This mutex is a significant point of contention in the ipsec code, and
can be relatively trivially replaced by a read-mostly lock.
It does require a separate lock for the replay protection, which we do
here by adding a separate mutex.

This improves throughput (without replay protection) by 10-15%.

MFC after: 3 weeks
Sponsored by: Orange Business Services
Differential Revision: https://reviews.freebsd.org/D35763

show more ...

ipsec: sprinkle CURVNET_ASSERT_SET

Reviewed by: ae
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D34314