cxgbe(4): Delete leftovers from $FreeBSD$ removal

No functional change.

MFC after: 1 week
Sponsored by: Chelsio Communications

Export t4_tom's symbols for other offload modules

This allows iw_cxgbe.ko, cxgbei.ko, nvmf_che.ko, etc. to be loaded when
debug.link_elf_leak_locals and debug.link_elf_obj_leak_locals are
disabled.

Export t4_tom's symbols for other offload modules

This allows iw_cxgbe.ko, cxgbei.ko, nvmf_che.ko, etc. to be loaded when
debug.link_elf_leak_locals and debug.link_elf_obj_leak_locals are
disabled.

PR: 291250
MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

sys: Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Add opt_kern_tls.h to the sources from t4_tom.ko.

Missed in r353328.

Sponsored by: Chelsio Communications

cxgbe(4): Delete leftovers from $FreeBSD$ removal

No functional change.

MFC after: 1 week
Sponsored by: Chelsio Communications

Export t4_tom's symbols for other offload modules

This allows iw_cxgbe.ko, cxgbei.ko, nvmf_che.ko, etc. to be loaded when
debug.link_elf_leak_locals and debug.link_elf_obj_leak_locals are
disabled.

Export t4_tom's symbols for other offload modules

This allows iw_cxgbe.ko, cxgbei.ko, nvmf_che.ko, etc. to be loaded when
debug.link_elf_leak_locals and debug.link_elf_obj_leak_locals are
disabled.

PR: 291250
MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

sys: Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Add opt_kern_tls.h to the sources from t4_tom.ko.

Missed in r353328.

Sponsored by: Chelsio Communications

Support for TLS offload of TOE connections on T6 adapters.

The TOE engine in Chelsio T6 adapters supports offloading of TLS
encryption and TCP segmentation for offloaded connections. Sockets
using

Support for TLS offload of TOE connections on T6 adapters.

The TOE engine in Chelsio T6 adapters supports offloading of TLS
encryption and TCP segmentation for offloaded connections. Sockets
using TLS are required to use a set of custom socket options to upload
RX and TX keys to the NIC and to enable RX processing. Currently
these socket options are implemented as TCP options in the vendor
specific range. A patched OpenSSL library will be made available in a
port / package for use with the TLS TOE support.

TOE sockets can either offload both transmit and reception of TLS
records or just transmit. TLS offload (both RX and TX) is enabled by
setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
enabled on the relevant interface. Transmit offload can be used on
any "normal" or TLS TOE socket by using the custom socket option to
program a transmit key. This permits most TOE sockets to
transparently offload TLS when applications use a patched SSL library
(e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
library). Receive offload can only be used with TOE sockets using the
TLS mode. The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
list of TCP port numbers. Any connection with either a local or
remote port number in that list will be created as a TLS socket rather
than a plain TOE socket. Note that although this sysctl accepts an
arbitrary list of port numbers, the sysctl(8) tool is only able to set
sysctl nodes to a single value. A TLS socket will hang without
receiving data if used by an application that is not using a patched
SSL library. Thus, the tls_rx_ports node should be used with care.
For a server mostly concerned with offloading TLS transmit, this node
is not needed as plain TOE sockets will fall back to software crypto
when using an unpatched SSL library.

New per-interface statistics nodes are added giving counts of TLS
packets and payload bytes (payload bytes do not include TLS headers or
authentication tags/MACs) offloaded via the TOE engine, e.g.:

dev.cc.0.stats.rx_tls_octets: 149
dev.cc.0.stats.rx_tls_records: 13
dev.cc.0.stats.tx_tls_octets: 26501823
dev.cc.0.stats.tx_tls_records: 1620

TLS transmit work requests are constructed by a new variant of
t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.

TLS transmit work requests require a buffer containing IVs. If the
IVs are too large to fit into the work request, a separate buffer is
allocated when constructing a work request. This buffer is associated
with the transmit descriptor and freed when the descriptor is ACKed by
the adapter.

Received TLS frames use two new CPL messages. The first message is a
CPL_TLS_DATA containing the decryped payload of a single TLS record.
The handler places the mbuf containing the received payload on an
mbufq in the TOE pcb. The second message is a CPL_RX_TLS_CMP message
which includes a copy of the TLS header and indicates if there were
any errors. The handler for this message places the TLS header into
the socket buffer followed by the saved mbuf with the payload data.
Both of these handlers are contained in tom/t4_tls.c.

A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
including send_rx_credits(), a new send_rx_modulate(), and
t4_close_conn().

TLS keys for both transmit and receive are stored in onboard memory
in the NIC in the "TLS keys" memory region.

In some cases a TLS socket can hang with pending data available in the
NIC that is not delivered to the host. As a workaround, TLS sockets
are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
any data is read from a TLS socket. In addition, a fallback timer will
periodically send CPL_RX_DATA_ACK messages to the NIC for connections
that are still in the handshake phase. Once the connection has
finished the handshake and programmed RX keys via the socket option,
the timer is stopped.

A new function select_ulp_mode() is used to determine what sub-mode a
given TOE socket should use (plain TOE, DDP, or TLS). The existing
set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
handles initialization of TLS-specific state when necessary in
addition to DDP-specific state.

Since TLS sockets do not receive individual TCP segments but always
receive full TLS records, they can receive more data than is available
in the current window (e.g. if a 16k TLS record is received but the
socket buffer is itself 16k). To cope with this, just drop the window
to 0 when this happens, but track the overage and "eat" the overage as
it is read from the socket buffer not opening the window (or adding
rx_credits) for the overage bytes.

Reviewed by: np (earlier version)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D14529

show more ...

Merge ^/head r317808 through r317970.

cxgbe/t4_tom: Per-connection rate limiting for TCP sockets handled by
the TOE. For now this capability is always enabled in kernels with
options RATELIMIT. t4_tom will check if_capenable once the b

cxgbe/t4_tom: Per-connection rate limiting for TCP sockets handled by
the TOE. For now this capability is always enabled in kernels with
options RATELIMIT. t4_tom will check if_capenable once the base driver
gets code to support rate limiting for any socket (TOE or not).

This was tested with iperf3 and netperf ToT as they already support
SO_MAX_PACING_RATE sockopt. There is a bug in firmwares prior to
1.16.45.0 that affects the BSD driver only and results in rate-limiting
at an incorrect rate. This will resolve by itself as soon as 1.16.45.0
or later firmware shows up in the driver.

Relnotes: Yes
Sponsored by: Chelsio Communications

show more ...

sys/modules: normalize .CURDIR-relative paths to SRCTOP

This simplifies make output/logic

Tested with: `cd sys/modules; make ALL_MODULES=` on amd64
MFC after: 1 month
Sponsored by: Dell EMC Isilon

Merge from head@274682

IFC @r269962

Submitted by: Anish Gupta (akgupt3@gmail.com)

Sync to HEAD@r269943.

Move most of the 15 variations on generating opt_inet.h and
opt_inet6.h into kmod.mk by forcing almost everybody to eat the same
dogfood. While at it, consolidate the opt_bpf.h and opt_mroute.h
targe

Move most of the 15 variations on generating opt_inet.h and
opt_inet6.h into kmod.mk by forcing almost everybody to eat the same
dogfood. While at it, consolidate the opt_bpf.h and opt_mroute.h
targets here too.

show more ...

List one file per line in the Makefiles. This makes it easier to read
diffs when a file is added or removed.

MFC after: 2 weeks

Improve compliance with style.Makefile(5).

MFC after: 2 weeks

MFC @ r266724

An SVM update will follow this.

IFC @266473

Merge from head

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Merge fresh head.

IFC @247348.

Sync with HEAD.