cxgbe: Stop using bus_space_tag/handle directly

Reviewed by: np, imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D53030

cxgbe(4): Expanded interrupt handling for T7

- Catch up with the new cause/perr registers. The high level approach
is the same but the T7 has an extra top level INT_CAUSE concentrator
and a PER

cxgbe(4): Expanded interrupt handling for T7

- Catch up with the new cause/perr registers. The high level approach
is the same but the T7 has an extra top level INT_CAUSE concentrator
and a PERR_CAUSE concentrator with a changed layout.

- Add various flags to control the interrupt handlers' behavior.

- Implement a t4_intr_clear that internally use the slow handler as an
iterator over known cause/perr registers. This lets the driver clear
all of the interrupt sources that it knows about. The firmware sets
up the interrupt enables and clears the causes normally so this call
should be redundant.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe: Add support to the base driver for NVMe/TCP PDU offload

- Adds various per-queue counters similar to iSCSI PDU offload as well
as a hook in the adapter softc for a reference to the NVMe/TCP

cxgbe: Add support to the base driver for NVMe/TCP PDU offload

- Adds various per-queue counters similar to iSCSI PDU offload as well
as a hook in the adapter softc for a reference to the NVMe/TCP softc.

- Instruct the firmware to include a DDP indicator in the status field
for NVMe/TCP PDU completion messages. This flag indicates if the
payload data for a PDU has been received in the free list or if it
was placed directly into a kernel I/O data buffer via DDP.

Sponsored by: Chelsio Communications

show more ...

cxgbe tom: Add support for sending NVMe/TCP PDUs

These work requests are largely similar to the same work requests for
iSCSI offload but use a newer tx_data work request structure.

This includes in

cxgbe tom: Add support for sending NVMe/TCP PDUs

These work requests are largely similar to the same work requests for
iSCSI offload but use a newer tx_data work request structure.

This includes initial support for ISO where a large C2H_DATA or
H2C_DATA PDU is split into multiple PDUs on the wire.

Sponsored by: Chelsio Communications

show more ...

cxgbe: Use partial GCM mode for partial TLS records on T7

T7's crypto co-processor adds a new partial GCM mode. This permits an
AES-GCM operation to be split into multiple requests. After each
req

cxgbe: Use partial GCM mode for partial TLS records on T7

T7's crypto co-processor adds a new partial GCM mode. This permits an
AES-GCM operation to be split into multiple requests. After each
request, the partial GHASH state is returned to the host and must be
included in the subsequent request to continue the GHASH computation.

Make use of this when sending a TLS record that spans multiple TCP
"request" (where a request can use TSO to span multiple segments).
This permits computing the final GHASH value across multiple requests
without having to re-send the entire TLS record for the final request.

To ensure that intermediate GHASH results are available when needed,
mbuf chains are queued in the TLS pcb and only dispatched one at a
time to the NIC TXQ. Packets which do not request a GHASH result
queue the next mbuf from the connection as soon as they are written
into the TXQ. Packets which do request a GHASH result queue the next
mbuf after the the GHASH result is returned by a message on a NIC RXQ.

Note that partial GCM mode is only used for the in-order data at the
"tip" of a connection and not for retransmits of earlier data.
Retransmits will not request a GHASH result so will enqueue the next
mbuf from the connection to the NIC TXQ after being written.

Partial GCM mode is also not TLS-aware and only supports "plain"
AES-GCM. Thus, TLS AAD must be explicitly constructed and sent as
part of the work request in the TXQ at the start of each TLS record.
However, by avoiding the need to retransmit the entire TLS record's
contents for the last mbuf spanning a TLS record, the overall amount
of "wasted" DMA is reduced by a factor of about 30.

MFC after: 3 days
Sponsored by: Chelsio Communications

show more ...

cxgbe: Permit multiple handlers for CPL6_FW_PLD

Currently this CPL is only used to handle replies from lookaside
crypto requests submitted by ccr(4). However, in the future this
request will be ret

cxgbe: Permit multiple handlers for CPL6_FW_PLD

Currently this CPL is only used to handle replies from lookaside
crypto requests submitted by ccr(4). However, in the future this
request will be returned for other requests. Use the low bit in the
cookie field as a way to identify replies to ccr(4) vs other use
cases. This should be safe as 'struct cryptop' pointers should be
word-aligned.

MFC after: 3 days
Sponsored by: Chelsio Communications

show more ...

cxgbe: Support for NIC KTLS transmit on T7 adapters.

Unlike NIC KTLS support on T6, T7 is able to reuse the existing TSO
functionality directly, including trimming the output of the crypto
engine be

cxgbe: Support for NIC KTLS transmit on T7 adapters.

Unlike NIC KTLS support on T6, T7 is able to reuse the existing TSO
functionality directly, including trimming the output of the crypto
engine before it is passed on to TSO. This is much simpler and does
not require the use of bypass pseudo-connections in the TOE engine.
Among other things this permits arbitrary TCP options (including the
full range of possible TCP timestamp values) while also avoiding
various edge cases where parts of a requested TCP packet could not
always be transmitted (e.g. partial trailers). This implementation
also permits NIC KTLS to be used in parallel with TOE.

This version does not yet support connections over a VF (specifically
the ktls_tunnel_packet function needs to handle the VF work request),
nor does it support VxLAN offload.

MFC after: 3 days
Sponsored by: Chelsio Communications

show more ...

cxgbe: Move the STAG and PBL memory pool arenas to the base driver

Both RDMA (iw_cxgbe) and NVMe offloads use TPT table entries to map
transaction tags in incoming PDUs to buffers in host memory per

cxgbe: Move the STAG and PBL memory pool arenas to the base driver

Both RDMA (iw_cxgbe) and NVMe offloads use TPT table entries to map
transaction tags in incoming PDUs to buffers in host memory permitting
direct placement of received data into host memory buffers avoiding
copies (iSCSI offload uses a different scheme for mapping tags to host
memory). Move the vmem arenas for the supporting card memory regions
from iw_cxgbe to the main driver so they can be shared with the NVMe
offload driver. In addition, add some helper routines for
constructing work requests to update TPT table entries.

MFC after: 3 days
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Updates for T7 CIM multicore operation

T7 has a multicore microprocessor and each core has its own queue
configuration, inbound/outbound queues, and logic analyzer. A work
request involvi

cxgbe(4): Updates for T7 CIM multicore operation

T7 has a multicore microprocessor and each core has its own queue
configuration, inbound/outbound queues, and logic analyzer. A work
request involving a tid can only be handled on queues where
(tid & tid_qid_sel_mask) == (eq->cntxt_id & tid_qid_sel_mask).

MFC after: 3 days
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Recognize the new capabilities reported by T7 firmwares

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Use hw_port and tx_chan correctly all over the driver

The port's tx channel and its hardware index are different things that
happen to have the same value on NICs using T4 through T6. Thi

cxgbe(4): Use hw_port and tx_chan correctly all over the driver

The port's tx channel and its hardware index are different things that
happen to have the same value on NICs using T4 through T6. This is
going to change so use the correct spelling all over the driver.
Specifically, the firmware expects the hw port index in all of these:

* PCIe channel for queues
* TX interface for TX CPLs
* PORTID in FW_PORT commands
* PORTID in t4_alloc_vi
* flowid in all the firmware flowc WRs
* mbox commands that deal with the tx scheduler

Also, create a port_id lookup table for hw_port, just like the one for
tx_chan. Use it to lookup the port softc in the port_info handler.

MFC after: 3 days
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Rename lport to hw_port.

No functional change intended.

MFC after: 3 days
Sponsored by: Chelsio Communications

mbuf: Allow clusters to fill an entire jumbo page.

The assumption that MCLBYTES == MJUMPAGESIZE can only happen if pages
are small is incorrect: it can also happen if MCLSHIFT is adjusted to
increas

mbuf: Allow clusters to fill an entire jumbo page.

The assumption that MCLBYTES == MJUMPAGESIZE can only happen if pages
are small is incorrect: it can also happen if MCLSHIFT is adjusted to
increase the size of clusters. Restore the ability to have clusters
fill a jumbo page, while still disallowing them from exceeding them.

MFC after: 1 week
Fixes: 840327e5ddf3, 9c3ad5ba932b
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Reviewed by: siderop1_netapp.com, kevans, brooks
Differential Revision: https://reviews.freebsd.org/D50242

show more ...

cxgbe(4): Block most access to the hardware as soon as the adapter stops.

Add a new hw_all_ok() routine and use it to avoid hardware access in the
public control interfaces (ifnet ioctls, ifmedia ca

cxgbe(4): Block most access to the hardware as soon as the adapter stops.

Add a new hw_all_ok() routine and use it to avoid hardware access in the
public control interfaces (ifnet ioctls, ifmedia calls, etc.). Continue
to use hw_off_limits() in the private ioctls/sysctls and other debug
code. Retire adapter_stopped() as it's of no use by itself.

This fixes problems where ifnet slow-path operations would enter a
synch_op just before set_adapter_hwstatus(false) and touch the hardware
when it's not safe to do so.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Remove some unused PCI routines and associated headers.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Allow t4_tom to be unloaded safely.

* Disable IFCAP_TOE automatically on all ifnets on all adapters during
unload. This is user-friendly and avoids panics due to stale ifnet
state aft

cxgbe(4): Allow t4_tom to be unloaded safely.

* Disable IFCAP_TOE automatically on all ifnets on all adapters during
unload. This is user-friendly and avoids panics due to stale ifnet
state after t4_tom is unloaded.
* Do not allow unload if tids are in use by the TOE on any adapter.

Reported by: Bimal Abraham @ Chelsio
MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Export the core suspend/resume functionality to other modules.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Stop work request queues in a reliable manner.

Clear the EQ_HW_ALLOCATED flag with the wrq lock held and discard all
work requests, pending or new, when it's not set.

MFC after: 1 week
Sp

cxgbe(4): Stop work request queues in a reliable manner.

Clear the EQ_HW_ALLOCATED flag with the wrq lock held and discard all
work requests, pending or new, when it's not set.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Basic infrastructure for ULDs to participate in adapter reset.

The suspend/resume/reset implementation in the base driver (LLD)
currently works when only stateless features are in use. Th

cxgbe(4): Basic infrastructure for ULDs to participate in adapter reset.

The suspend/resume/reset implementation in the base driver (LLD)
currently works when only stateless features are in use. This commit
adds basic infrastructure for stateful upper layer drivers (ULDs) to
participate in suspend/resume/reset.

* Add a uld_restart to indicate that the adapter has been restarted
after a stop and the ULD should resume operations.
* Move the existing functionality in t4_suspend/t4_resume to stop_lld and
restart_lld. Use these and the new uld restart routines everywhere the
adapter has to be stopped abruptly and restarted, namely:
1. PCIE bus suspend/resume/reset methods invoked by the kernel.
2. Manual internal-reset using driver sysctl.
3. Automatic internal-reset on a fatal error.
* Implement an alternate internal-reset for use in VMs and for testing.

Typical reset sequence is:
stop_adapter(sc);
stop_lld(sc);
stop_all_uld(sc);
set_adapter_hwstatus(sc, false);

/* hw reset takes place here. */

restart_adapter(sc);
restart_lld(sc);
set_adapter_hwstatus(sc, true);
restart_all_uld(sc);

MFC after: 1 month
Sponsored by: Chelsio Communications

show more ...

cxgbev(4): Pay attention to the VLAN configuration for the VF.

Make sure that the transmit traffic is tagged correctly or else the
firmware will refuse to transmit and will report an ACL violation.

cxgbev(4): Pay attention to the VLAN configuration for the VF.

Make sure that the transmit traffic is tagged correctly or else the
firmware will refuse to transmit and will report an ACL violation.

On receive the hardware will make sure that tagged traffic is delivered
to the appropriate VM. The driver only asserts that the VLAN id that
was extracted from the wire traffic matches the VF's configuration.

All this works when associating a specific VLAN id with a VF. The
'trunk' setting likely needs more work.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): New knob to limit driver to the specified types of doorbells.

hw.cxgbe.doorbells_allowed="0xf"

The adapter's doorbells bitmap is clipped to the value specified in the
tunable, which is me

cxgbe(4): New knob to limit driver to the specified types of doorbells.

hw.cxgbe.doorbells_allowed="0xf"

The adapter's doorbells bitmap is clipped to the value specified in the
tunable, which is meant for debug and workarounds only. There is no
change in default behavior.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Rename rx_c_chan to rx_chan.

It is the equivalent of tx_chan but for receive so rx_chan is a better
name. Initialize both using helper functions and make sure both are
displayed in the sy

cxgbe(4): Rename rx_c_chan to rx_chan.

It is the equivalent of tx_chan but for receive so rx_chan is a better
name. Initialize both using helper functions and make sure both are
displayed in the sysctl MIB.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Allocate a taskqueue per port instead of per channel.

All the channels are not used on all boards and there's no point
allocating taskqueues that will never be used.

MFC after: 1 week
Spo

cxgbe(4): Allocate a taskqueue per port instead of per channel.

All the channels are not used on all boards and there's no point
allocating taskqueues that will never be used.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe: Support TCP_USE_DDP on offloaded TOE connections

When this socket option is enabled, relatively large contiguous
buffers are allocated and used to receive data from the remote
connection. Wh

cxgbe: Support TCP_USE_DDP on offloaded TOE connections

When this socket option is enabled, relatively large contiguous
buffers are allocated and used to receive data from the remote
connection. When data is received a wrapper M_EXT mbuf is queued to
the socket's receive buffer. This reduces the length of the linked
list of received mbufs and allows consumers to consume receive data in
larger chunks.

To minimize reprogramming the page pods in the adapter, receive
buffers for a given connection are recycled. When a buffer has been
fully consumed by the receiver and freed, the buffer is placed on a
per-connection free buffers list.

The size of the receive buffers defaults to 256k and can be set via
the hw.cxgbe.toe.ddp_rcvbuf_len sysctl. The
hw.cxgbe.toe.ddp_rcvbuf_cache sysctl (defaults to 4) determines the
maximum number of free buffers cached per connection. Note that this
limit does not apply to "in-flight" receive buffers that are
associated with mbufs in the socket's receive buffer.

Co-authored-by: Navdeep Parhar <np@FreeBSD.org>
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D44001

show more ...

cxgbe: Add counters for POSIX async I/O requests handled by the driver

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D43668