1.. SPDX-License-Identifier: GPL-2.0 2 3====== 4AF_XDP 5====== 6 7Overview 8======== 9 10AF_XDP is an address family that is optimized for high performance 11packet processing. 12 13This document assumes that the reader is familiar with BPF and XDP. If 14not, the Cilium project has an excellent reference guide at 15http://cilium.readthedocs.io/en/latest/bpf/. 16 17Using the XDP_REDIRECT action from an XDP program, the program can 18redirect ingress frames to other XDP enabled netdevs, using the 19bpf_redirect_map() function. AF_XDP sockets enable the possibility for 20XDP programs to redirect frames to a memory buffer in a user-space 21application. 22 23An AF_XDP socket (XSK) is created with the normal socket() 24syscall. Associated with each XSK are two rings: the RX ring and the 25TX ring. A socket can receive packets on the RX ring and it can send 26packets on the TX ring. These rings are registered and sized with the 27setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory 28to have at least one of these rings for each socket. An RX or TX 29descriptor ring points to a data buffer in a memory area called a 30UMEM. RX and TX can share the same UMEM so that a packet does not have 31to be copied between RX and TX. Moreover, if a packet needs to be kept 32for a while due to a possible retransmit, the descriptor that points 33to that packet can be changed to point to another and reused right 34away. This again avoids copying data. 35 36The UMEM consists of a number of equally sized chunks. A descriptor in 37one of the rings references a frame by referencing its addr. The addr 38is simply an offset within the entire UMEM region. The user space 39allocates memory for this UMEM using whatever means it feels is most 40appropriate (malloc, mmap, huge pages, etc). This memory area is then 41registered with the kernel using the new setsockopt XDP_UMEM_REG. The 42UMEM also has two rings: the FILL ring and the COMPLETION ring. The 43FILL ring is used by the application to send down addr for the kernel 44to fill in with RX packet data. References to these frames will then 45appear in the RX ring once each packet has been received. The 46COMPLETION ring, on the other hand, contains frame addr that the 47kernel has transmitted completely and can now be used again by user 48space, for either TX or RX. Thus, the frame addrs appearing in the 49COMPLETION ring are addrs that were previously transmitted using the 50TX ring. In summary, the RX and FILL rings are used for the RX path 51and the TX and COMPLETION rings are used for the TX path. 52 53The socket is then finally bound with a bind() call to a device and a 54specific queue id on that device, and it is not until bind is 55completed that traffic starts to flow. 56 57The UMEM can be shared between processes, if desired. If a process 58wants to do this, it simply skips the registration of the UMEM and its 59corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind 60call and submits the XSK of the process it would like to share UMEM 61with as well as its own newly created XSK socket. The new process will 62then receive frame addr references in its own RX ring that point to 63this shared UMEM. Note that since the ring structures are 64single-consumer / single-producer (for performance reasons), the new 65process has to create its own socket with associated RX and TX rings, 66since it cannot share this with the other process. This is also the 67reason that there is only one set of FILL and COMPLETION rings per 68UMEM. It is the responsibility of a single process to handle the UMEM. 69 70How is then packets distributed from an XDP program to the XSKs? There 71is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The 72user-space application can place an XSK at an arbitrary place in this 73map. The XDP program can then redirect a packet to a specific index in 74this map and at this point XDP validates that the XSK in that map was 75indeed bound to that device and ring number. If not, the packet is 76dropped. If the map is empty at that index, the packet is also 77dropped. This also means that it is currently mandatory to have an XDP 78program loaded (and one XSK in the XSKMAP) to be able to get any 79traffic to user space through the XSK. 80 81AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the 82driver does not have support for XDP, or XDP_SKB is explicitly chosen 83when loading the XDP program, XDP_SKB mode is employed that uses SKBs 84together with the generic XDP support and copies out the data to user 85space. A fallback mode that works for any network device. On the other 86hand, if the driver has support for XDP, it will be used by the AF_XDP 87code to provide better performance, but there is still a copy of the 88data into user space. 89 90Concepts 91======== 92 93In order to use an AF_XDP socket, a number of associated objects need 94to be setup. These objects and their options are explained in the 95following sections. 96 97For an overview on how AF_XDP works, you can also take a look at the 98Linux Plumbers paper from 2018 on the subject: 99http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do 100NOT consult the paper from 2017 on "AF_PACKET v4", the first attempt 101at AF_XDP. Nearly everything changed since then. Jonathan Corbet has 102also written an excellent article on LWN, "Accelerating networking 103with AF_XDP". It can be found at https://lwn.net/Articles/750845/. 104 105UMEM 106---- 107 108UMEM is a region of virtual contiguous memory, divided into 109equal-sized frames. An UMEM is associated to a netdev and a specific 110queue id of that netdev. It is created and configured (chunk size, 111headroom, start address and size) by using the XDP_UMEM_REG setsockopt 112system call. A UMEM is bound to a netdev and queue id, via the bind() 113system call. 114 115An AF_XDP is socket linked to a single UMEM, but one UMEM can have 116multiple AF_XDP sockets. To share an UMEM created via one socket A, 117the next socket B can do this by setting the XDP_SHARED_UMEM flag in 118struct sockaddr_xdp member sxdp_flags, and passing the file descriptor 119of A to struct sockaddr_xdp member sxdp_shared_umem_fd. 120 121The UMEM has two single-producer/single-consumer rings that are used 122to transfer ownership of UMEM frames between the kernel and the 123user-space application. 124 125Rings 126----- 127 128There are a four different kind of rings: FILL, COMPLETION, RX and 129TX. All rings are single-producer/single-consumer, so the user-space 130application need explicit synchronization of multiple 131processes/threads are reading/writing to them. 132 133The UMEM uses two rings: FILL and COMPLETION. Each socket associated 134with the UMEM must have an RX queue, TX queue or both. Say, that there 135is a setup with four sockets (all doing TX and RX). Then there will be 136one FILL ring, one COMPLETION ring, four TX rings and four RX rings. 137 138The rings are head(producer)/tail(consumer) based rings. A producer 139writes the data ring at the index pointed out by struct xdp_ring 140producer member, and increasing the producer index. A consumer reads 141the data ring at the index pointed out by struct xdp_ring consumer 142member, and increasing the consumer index. 143 144The rings are configured and created via the _RING setsockopt system 145calls and mmapped to user-space using the appropriate offset to mmap() 146(XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and 147XDP_UMEM_PGOFF_COMPLETION_RING). 148 149The size of the rings need to be of size power of two. 150 151UMEM Fill Ring 152~~~~~~~~~~~~~~ 153 154The FILL ring is used to transfer ownership of UMEM frames from 155user-space to kernel-space. The UMEM addrs are passed in the ring. As 156an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has 15716 chunks and can pass addrs between 0 and 64k. 158 159Frames passed to the kernel are used for the ingress path (RX rings). 160 161The user application produces UMEM addrs to this ring. Note that, if 162running the application with aligned chunk mode, the kernel will mask 163the incoming addr. E.g. for a chunk size of 2k, the log2(2048) LSB of 164the addr will be masked off, meaning that 2048, 2050 and 3000 refers 165to the same chunk. If the user application is run in the unaligned 166chunks mode, then the incoming addr will be left untouched. 167 168 169UMEM Completion Ring 170~~~~~~~~~~~~~~~~~~~~ 171 172The COMPLETION Ring is used transfer ownership of UMEM frames from 173kernel-space to user-space. Just like the FILL ring, UMEM indices are 174used. 175 176Frames passed from the kernel to user-space are frames that has been 177sent (TX ring) and can be used by user-space again. 178 179The user application consumes UMEM addrs from this ring. 180 181 182RX Ring 183~~~~~~~ 184 185The RX ring is the receiving side of a socket. Each entry in the ring 186is a struct xdp_desc descriptor. The descriptor contains UMEM offset 187(addr) and the length of the data (len). 188 189If no frames have been passed to kernel via the FILL ring, no 190descriptors will (or can) appear on the RX ring. 191 192The user application consumes struct xdp_desc descriptors from this 193ring. 194 195TX Ring 196~~~~~~~ 197 198The TX ring is used to send frames. The struct xdp_desc descriptor is 199filled (index, length and offset) and passed into the ring. 200 201To start the transfer a sendmsg() system call is required. This might 202be relaxed in the future. 203 204The user application produces struct xdp_desc descriptors to this 205ring. 206 207Libbpf 208====== 209 210Libbpf is a helper library for eBPF and XDP that makes using these 211technologies a lot simpler. It also contains specific helper functions 212in tools/testing/selftests/bpf/xsk.h for facilitating the use of 213AF_XDP. It contains two types of functions: those that can be used to 214make the setup of AF_XDP socket easier and ones that can be used in the 215data plane to access the rings safely and quickly. 216 217We recommend that you use this library unless you have become a power 218user. It will make your program a lot simpler. 219 220XSKMAP / BPF_MAP_TYPE_XSKMAP 221============================ 222 223On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that 224is used in conjunction with bpf_redirect_map() to pass the ingress 225frame to a socket. 226 227The user application inserts the socket into the map, via the bpf() 228system call. 229 230Note that if an XDP program tries to redirect to a socket that does 231not match the queue configuration and netdev, the frame will be 232dropped. E.g. an AF_XDP socket is bound to netdev eth0 and 233queue 17. Only the XDP program executing for eth0 and queue 17 will 234successfully pass data to the socket. Please refer to the sample 235application (samples/bpf/) in for an example. 236 237Configuration Flags and Socket Options 238====================================== 239 240These are the various configuration flags that can be used to control 241and monitor the behavior of AF_XDP sockets. 242 243XDP_COPY and XDP_ZEROCOPY bind flags 244------------------------------------ 245 246When you bind to a socket, the kernel will first try to use zero-copy 247copy. If zero-copy is not supported, it will fall back on using copy 248mode, i.e. copying all packets out to user space. But if you would 249like to force a certain mode, you can use the following flags. If you 250pass the XDP_COPY flag to the bind call, the kernel will force the 251socket into copy mode. If it cannot use copy mode, the bind call will 252fail with an error. Conversely, the XDP_ZEROCOPY flag will force the 253socket into zero-copy mode or fail. 254 255XDP_SHARED_UMEM bind flag 256------------------------- 257 258This flag enables you to bind multiple sockets to the same UMEM. It 259works on the same queue id, between queue ids and between 260netdevs/devices. In this mode, each socket has their own RX and TX 261rings as usual, but you are going to have one or more FILL and 262COMPLETION ring pairs. You have to create one of these pairs per 263unique netdev and queue id tuple that you bind to. 264 265Starting with the case were we would like to share a UMEM between 266sockets bound to the same netdev and queue id. The UMEM (tied to the 267fist socket created) will only have a single FILL ring and a single 268COMPLETION ring as there is only on unique netdev,queue_id tuple that 269we have bound to. To use this mode, create the first socket and bind 270it in the normal way. Create a second socket and create an RX and a TX 271ring, or at least one of them, but no FILL or COMPLETION rings as the 272ones from the first socket will be used. In the bind call, set he 273XDP_SHARED_UMEM option and provide the initial socket's fd in the 274sxdp_shared_umem_fd field. You can attach an arbitrary number of extra 275sockets this way. 276 277What socket will then a packet arrive on? This is decided by the XDP 278program. Put all the sockets in the XSK_MAP and just indicate which 279index in the array you would like to send each packet to. A simple 280round-robin example of distributing packets is shown below: 281 282.. code-block:: c 283 284 #include <linux/bpf.h> 285 #include "bpf_helpers.h" 286 287 #define MAX_SOCKS 16 288 289 struct { 290 __uint(type, BPF_MAP_TYPE_XSKMAP); 291 __uint(max_entries, MAX_SOCKS); 292 __uint(key_size, sizeof(int)); 293 __uint(value_size, sizeof(int)); 294 } xsks_map SEC(".maps"); 295 296 static unsigned int rr; 297 298 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) 299 { 300 rr = (rr + 1) & (MAX_SOCKS - 1); 301 302 return bpf_redirect_map(&xsks_map, rr, XDP_DROP); 303 } 304 305Note, that since there is only a single set of FILL and COMPLETION 306rings, and they are single producer, single consumer rings, you need 307to make sure that multiple processes or threads do not use these rings 308concurrently. There are no synchronization primitives in the 309libbpf code that protects multiple users at this point in time. 310 311Libbpf uses this mode if you create more than one socket tied to the 312same UMEM. However, note that you need to supply the 313XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the 314xsk_socket__create calls and load your own XDP program as there is no 315built in one in libbpf that will route the traffic for you. 316 317The second case is when you share a UMEM between sockets that are 318bound to different queue ids and/or netdevs. In this case you have to 319create one FILL ring and one COMPLETION ring for each unique 320netdev,queue_id pair. Let us say you want to create two sockets bound 321to two different queue ids on the same netdev. Create the first socket 322and bind it in the normal way. Create a second socket and create an RX 323and a TX ring, or at least one of them, and then one FILL and 324COMPLETION ring for this socket. Then in the bind call, set he 325XDP_SHARED_UMEM option and provide the initial socket's fd in the 326sxdp_shared_umem_fd field as you registered the UMEM on that 327socket. These two sockets will now share one and the same UMEM. 328 329There is no need to supply an XDP program like the one in the previous 330case where sockets were bound to the same queue id and 331device. Instead, use the NIC's packet steering capabilities to steer 332the packets to the right queue. In the previous example, there is only 333one queue shared among sockets, so the NIC cannot do this steering. It 334can only steer between queues. 335 336In libbpf, you need to use the xsk_socket__create_shared() API as it 337takes a reference to a FILL ring and a COMPLETION ring that will be 338created for you and bound to the shared UMEM. You can use this 339function for all the sockets you create, or you can use it for the 340second and following ones and use xsk_socket__create() for the first 341one. Both methods yield the same result. 342 343Note that a UMEM can be shared between sockets on the same queue id 344and device, as well as between queues on the same device and between 345devices at the same time. 346 347XDP_USE_NEED_WAKEUP bind flag 348----------------------------- 349 350This option adds support for a new flag called need_wakeup that is 351present in the FILL ring and the TX ring, the rings for which user 352space is a producer. When this option is set in the bind call, the 353need_wakeup flag will be set if the kernel needs to be explicitly 354woken up by a syscall to continue processing packets. If the flag is 355zero, no syscall is needed. 356 357If the flag is set on the FILL ring, the application needs to call 358poll() to be able to continue to receive packets on the RX ring. This 359can happen, for example, when the kernel has detected that there are no 360more buffers on the FILL ring and no buffers left on the RX HW ring of 361the NIC. In this case, interrupts are turned off as the NIC cannot 362receive any packets (as there are no buffers to put them in), and the 363need_wakeup flag is set so that user space can put buffers on the 364FILL ring and then call poll() so that the kernel driver can put these 365buffers on the HW ring and start to receive packets. 366 367If the flag is set for the TX ring, it means that the application 368needs to explicitly notify the kernel to send any packets put on the 369TX ring. This can be accomplished either by a poll() call, as in the 370RX path, or by calling sendto(). 371 372An example with the use of libbpf helpers would look like this for the 373TX path: 374 375.. code-block:: c 376 377 if (xsk_ring_prod__needs_wakeup(&my_tx_ring)) 378 sendto(xsk_socket__fd(xsk_handle), NULL, 0, MSG_DONTWAIT, NULL, 0); 379 380I.e., only use the syscall if the flag is set. 381 382We recommend that you always enable this mode as it usually leads to 383better performance especially if you run the application and the 384driver on the same core, but also if you use different cores for the 385application and the kernel driver, as it reduces the number of 386syscalls needed for the TX path. 387 388XDP_{RX|TX|UMEM_FILL|UMEM_COMPLETION}_RING setsockopts 389------------------------------------------------------ 390 391These setsockopts sets the number of descriptors that the RX, TX, 392FILL, and COMPLETION rings respectively should have. It is mandatory 393to set the size of at least one of the RX and TX rings. If you set 394both, you will be able to both receive and send traffic from your 395application, but if you only want to do one of them, you can save 396resources by only setting up one of them. Both the FILL ring and the 397COMPLETION ring are mandatory as you need to have a UMEM tied to your 398socket. But if the XDP_SHARED_UMEM flag is used, any socket after the 399first one does not have a UMEM and should in that case not have any 400FILL or COMPLETION rings created as the ones from the shared UMEM will 401be used. Note, that the rings are single-producer single-consumer, so 402do not try to access them from multiple processes at the same 403time. See the XDP_SHARED_UMEM section. 404 405In libbpf, you can create Rx-only and Tx-only sockets by supplying 406NULL to the rx and tx arguments, respectively, to the 407xsk_socket__create function. 408 409If you create a Tx-only socket, we recommend that you do not put any 410packets on the fill ring. If you do this, drivers might think you are 411going to receive something when you in fact will not, and this can 412negatively impact performance. 413 414XDP_UMEM_REG setsockopt 415----------------------- 416 417This setsockopt registers a UMEM to a socket. This is the area that 418contain all the buffers that packet can reside in. The call takes a 419pointer to the beginning of this area and the size of it. Moreover, it 420also has parameter called chunk_size that is the size that the UMEM is 421divided into. It can only be 2K or 4K at the moment. If you have an 422UMEM area that is 128K and a chunk size of 2K, this means that you 423will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM 424area and that your largest packet size can be 2K. 425 426There is also an option to set the headroom of each single buffer in 427the UMEM. If you set this to N bytes, it means that the packet will 428start N bytes into the buffer leaving the first N bytes for the 429application to use. The final option is the flags field, but it will 430be dealt with in separate sections for each UMEM flag. 431 432SO_BINDTODEVICE setsockopt 433-------------------------- 434 435This is a generic SOL_SOCKET option that can be used to tie AF_XDP 436socket to a particular network interface. It is useful when a socket 437is created by a privileged process and passed to a non-privileged one. 438Once the option is set, kernel will refuse attempts to bind that socket 439to a different interface. Updating the value requires CAP_NET_RAW. 440 441XDP_MAX_TX_SKB_BUDGET setsockopt 442-------------------------------- 443 444This setsockopt sets the maximum number of descriptors that can be handled 445and passed to the driver at one send syscall. It is applied in the copy 446mode to allow application to tune the per-socket maximum iteration for 447better throughput and less frequency of send syscall. 448Allowed range is [32, xs->tx->nentries]. 449 450XDP_STATISTICS getsockopt 451------------------------- 452 453Gets drop statistics of a socket that can be useful for debug 454purposes. The supported statistics are shown below: 455 456.. code-block:: c 457 458 struct xdp_statistics { 459 __u64 rx_dropped; /* Dropped for reasons other than invalid desc */ 460 __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */ 461 __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */ 462 }; 463 464XDP_OPTIONS getsockopt 465---------------------- 466 467Gets options from an XDP socket. The only one supported so far is 468XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not. 469 470Multi-Buffer Support 471==================== 472 473With multi-buffer support, programs using AF_XDP sockets can receive 474and transmit packets consisting of multiple buffers both in copy and 475zero-copy mode. For example, a packet can consist of two 476frames/buffers, one with the header and the other one with the data, 477or a 9K Ethernet jumbo frame can be constructed by chaining together 478three 4K frames. 479 480Some definitions: 481 482* A packet consists of one or more frames 483 484* A descriptor in one of the AF_XDP rings always refers to a single 485 frame. In the case the packet consists of a single frame, the 486 descriptor refers to the whole packet. 487 488To enable multi-buffer support for an AF_XDP socket, use the new bind 489flag XDP_USE_SG. If this is not provided, all multi-buffer packets 490will be dropped just as before. Note that the XDP program loaded also 491needs to be in multi-buffer mode. This can be accomplished by using 492"xdp.frags" as the section name of the XDP program used. 493 494To represent a packet consisting of multiple frames, a new flag called 495XDP_PKT_CONTD is introduced in the options field of the Rx and Tx 496descriptors. If it is true (1) the packet continues with the next 497descriptor and if it is false (0) it means this is the last descriptor 498of the packet. Why the reverse logic of end-of-packet (eop) flag found 499in many NICs? Just to preserve compatibility with non-multi-buffer 500applications that have this bit set to false for all packets on Rx, 501and the apps set the options field to zero for Tx, as anything else 502will be treated as an invalid descriptor. 503 504These are the semantics for producing packets onto AF_XDP Tx ring 505consisting of multiple frames: 506 507* When an invalid descriptor is found, all the other 508 descriptors/frames of this packet are marked as invalid and not 509 completed. The next descriptor is treated as the start of a new 510 packet, even if this was not the intent (because we cannot guess 511 the intent). As before, if your program is producing invalid 512 descriptors you have a bug that must be fixed. 513 514* Zero length descriptors are treated as invalid descriptors. 515 516* For copy mode, the maximum supported number of frames in a packet is 517 equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all 518 descriptors accumulated so far are dropped and treated as 519 invalid. To produce an application that will work on any system 520 regardless of this config setting, limit the number of frags to 18, 521 as the minimum value of the config is 17. 522 523* For zero-copy mode, the limit is up to what the NIC HW 524 supports. Usually at least five on the NICs we have checked. We 525 consciously chose to not enforce a rigid limit (such as 526 CONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would have 527 resulted in copy actions under the hood to fit into what limit the 528 NIC supports. Kind of defeats the purpose of zero-copy mode. How to 529 probe for this limit is explained in the "probe for multi-buffer 530 support" section. 531 532On the Rx path in copy-mode, the xsk core copies the XDP data into 533multiple descriptors, if needed, and sets the XDP_PKT_CONTD flag as 534detailed before. Zero-copy mode works the same, though the data is not 535copied. When the application gets a descriptor with the XDP_PKT_CONTD 536flag set to one, it means that the packet consists of multiple buffers 537and it continues with the next buffer in the following 538descriptor. When a descriptor with XDP_PKT_CONTD == 0 is received, it 539means that this is the last buffer of the packet. AF_XDP guarantees 540that only a complete packet (all frames in the packet) is sent to the 541application. If there is not enough space in the AF_XDP Rx ring, all 542frames of the packet will be dropped. 543 544If application reads a batch of descriptors, using for example the libxdp 545interfaces, it is not guaranteed that the batch will end with a full 546packet. It might end in the middle of a packet and the rest of the 547buffers of that packet will arrive at the beginning of the next batch, 548since the libxdp interface does not read the whole ring (unless you 549have an enormous batch size or a very small ring size). 550 551An example program each for Rx and Tx multi-buffer support can be found 552later in this document. 553 554Usage 555----- 556 557In order to use AF_XDP sockets two parts are needed. The user-space 558application and the XDP program. For a complete setup and usage example, 559please refer to the xdp-project at 560https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example. 561 562The XDP code sample is the following: 563 564.. code-block:: c 565 566 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) 567 { 568 int index = ctx->rx_queue_index; 569 570 // A set entry here means that the corresponding queue_id 571 // has an active AF_XDP socket bound to it. 572 if (bpf_map_lookup_elem(&xsks_map, &index)) 573 return bpf_redirect_map(&xsks_map, index, 0); 574 575 return XDP_PASS; 576 } 577 578A simple but not so performance ring dequeue and enqueue could look 579like this: 580 581.. code-block:: c 582 583 // struct xdp_rxtx_ring { 584 // __u32 *producer; 585 // __u32 *consumer; 586 // struct xdp_desc *desc; 587 // }; 588 589 // struct xdp_umem_ring { 590 // __u32 *producer; 591 // __u32 *consumer; 592 // __u64 *desc; 593 // }; 594 595 // typedef struct xdp_rxtx_ring RING; 596 // typedef struct xdp_umem_ring RING; 597 598 // typedef struct xdp_desc RING_TYPE; 599 // typedef __u64 RING_TYPE; 600 601 int dequeue_one(RING *ring, RING_TYPE *item) 602 { 603 __u32 entries = *ring->producer - *ring->consumer; 604 605 if (entries == 0) 606 return -1; 607 608 // read-barrier! 609 610 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)]; 611 (*ring->consumer)++; 612 return 0; 613 } 614 615 int enqueue_one(RING *ring, const RING_TYPE *item) 616 { 617 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer); 618 619 if (free_entries == 0) 620 return -1; 621 622 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item; 623 624 // write-barrier! 625 626 (*ring->producer)++; 627 return 0; 628 } 629 630But please use the libbpf functions as they are optimized and ready to 631use. Will make your life easier. 632 633Usage Multi-Buffer Rx 634--------------------- 635 636Here is a simple Rx path pseudo-code example (using libxdp interfaces 637for simplicity). Error paths have been excluded to keep it short: 638 639.. code-block:: c 640 641 void rx_packets(struct xsk_socket_info *xsk) 642 { 643 static bool new_packet = true; 644 u32 idx_rx = 0, idx_fq = 0; 645 static char *pkt; 646 647 int rcvd = xsk_ring_cons__peek(&xsk->rx, opt_batch_size, &idx_rx); 648 649 xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq); 650 651 for (int i = 0; i < rcvd; i++) { 652 struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++); 653 char *frag = xsk_umem__get_data(xsk->umem->buffer, desc->addr); 654 bool eop = !(desc->options & XDP_PKT_CONTD); 655 656 if (new_packet) 657 pkt = frag; 658 else 659 add_frag_to_pkt(pkt, frag); 660 661 if (eop) 662 process_pkt(pkt); 663 664 new_packet = eop; 665 666 *xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = desc->addr; 667 } 668 669 xsk_ring_prod__submit(&xsk->umem->fq, rcvd); 670 xsk_ring_cons__release(&xsk->rx, rcvd); 671 } 672 673Usage Multi-Buffer Tx 674--------------------- 675 676Here is an example Tx path pseudo-code (using libxdp interfaces for 677simplicity) ignoring that the umem is finite in size, and that we 678eventually will run out of packets to send. Also assumes pkts.addr 679points to a valid location in the umem. 680 681.. code-block:: c 682 683 void tx_packets(struct xsk_socket_info *xsk, struct pkt *pkts, 684 int batch_size) 685 { 686 u32 idx, i, pkt_nb = 0; 687 688 xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx); 689 690 for (i = 0; i < batch_size;) { 691 u64 addr = pkts[pkt_nb].addr; 692 u32 len = pkts[pkt_nb].size; 693 694 do { 695 struct xdp_desc *tx_desc; 696 697 tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx + i++); 698 tx_desc->addr = addr; 699 700 if (len > xsk_frame_size) { 701 tx_desc->len = xsk_frame_size; 702 tx_desc->options = XDP_PKT_CONTD; 703 } else { 704 tx_desc->len = len; 705 tx_desc->options = 0; 706 pkt_nb++; 707 } 708 len -= tx_desc->len; 709 addr += xsk_frame_size; 710 711 if (i == batch_size) { 712 /* Remember len, addr, pkt_nb for next iteration. 713 * Skipped for simplicity. 714 */ 715 break; 716 } 717 } while (len); 718 } 719 720 xsk_ring_prod__submit(&xsk->tx, i); 721 } 722 723Probing for Multi-Buffer Support 724-------------------------------- 725 726To discover if a driver supports multi-buffer AF_XDP in SKB or DRV 727mode, use the XDP_FEATURES feature of netlink in linux/netdev.h to 728query for NETDEV_XDP_ACT_RX_SG support. This is the same flag as for 729querying for XDP multi-buffer support. If XDP supports multi-buffer in 730a driver, then AF_XDP will also support that in SKB and DRV mode. 731 732To discover if a driver supports multi-buffer AF_XDP in zero-copy 733mode, use XDP_FEATURES and first check the NETDEV_XDP_ACT_XSK_ZEROCOPY 734flag. If it is set, it means that at least zero-copy is supported and 735you should go and check the netlink attribute 736NETDEV_A_DEV_XDP_ZC_MAX_SEGS in linux/netdev.h. An unsigned integer 737value will be returned stating the max number of frags that are 738supported by this device in zero-copy mode. These are the possible 739return values: 740 7411: Multi-buffer for zero-copy is not supported by this device, as max 742 one fragment supported means that multi-buffer is not possible. 743 744>=2: Multi-buffer is supported in zero-copy mode for this device. The 745 returned number signifies the max number of frags supported. 746 747For an example on how these are used through libbpf, please take a 748look at tools/testing/selftests/bpf/xskxceiver.c. 749 750Multi-Buffer Support for Zero-Copy Drivers 751------------------------------------------ 752 753Zero-copy drivers usually use the batched APIs for Rx and Tx 754processing. Note that the Tx batch API guarantees that it will provide 755a batch of Tx descriptors that ends with full packet at the end. This 756to facilitate extending a zero-copy driver with multi-buffer support. 757 758Sample application 759================== 760There is a xdpsock benchmarking/test application that can be found at 761https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example 762that demonstrates how to use AF_XDP sockets with private 763UMEMs. Say that you would like your UDP traffic from port 4242 to end 764up in queue 16, that we will enable AF_XDP on. Here, we use ethtool 765for this:: 766 767 ethtool -N p3p2 rx-flow-hash udp4 fn 768 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \ 769 action 16 770 771Running the rxdrop benchmark in XDP_DRV mode can then be done 772using:: 773 774 samples/bpf/xdpsock -i p3p2 -q 16 -r -N 775 776For XDP_SKB mode, use the switch "-S" instead of "-N" and all options 777can be displayed with "-h", as usual. 778 779This sample application uses libbpf to make the setup and usage of 780AF_XDP simpler. If you want to know how the raw uapi of AF_XDP is 781really used to make something more advanced, take a look at the libbpf 782code in tools/testing/selftests/bpf/xsk.[ch]. 783 784FAQ 785======= 786 787Q: I am not seeing any traffic on the socket. What am I doing wrong? 788 789A: When a netdev of a physical NIC is initialized, Linux usually 790 allocates one RX and TX queue pair per core. So on a 8 core system, 791 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP 792 bind call or the xsk_socket__create libbpf function call, you 793 specify a specific queue id to bind to and it is only the traffic 794 towards that queue you are going to get on you socket. So in the 795 example above, if you bind to queue 0, you are NOT going to get any 796 traffic that is distributed to queues 1 through 7. If you are 797 lucky, you will see the traffic, but usually it will end up on one 798 of the queues you have not bound to. 799 800 There are a number of ways to solve the problem of getting the 801 traffic you want to the queue id you bound to. If you want to see 802 all the traffic, you can force the netdev to only have 1 queue, queue 803 id 0, and then bind to queue 0. You can use ethtool to do this:: 804 805 sudo ethtool -L <interface> combined 1 806 807 If you want to only see part of the traffic, you can program the 808 NIC through ethtool to filter out your traffic to a single queue id 809 that you can bind your XDP socket to. Here is one example in which 810 UDP traffic to and from port 4242 are sent to queue 2:: 811 812 sudo ethtool -N <interface> rx-flow-hash udp4 fn 813 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \ 814 4242 action 2 815 816 A number of other ways are possible all up to the capabilities of 817 the NIC you have. 818 819Q: Can I use the XSKMAP to implement a switch between different umems 820 in copy mode? 821 822A: The short answer is no, that is not supported at the moment. The 823 XSKMAP can only be used to switch traffic coming in on queue id X 824 to sockets bound to the same queue id X. The XSKMAP can contain 825 sockets bound to different queue ids, for example X and Y, but only 826 traffic goming in from queue id Y can be directed to sockets bound 827 to the same queue id Y. In zero-copy mode, you should use the 828 switch, or other distribution mechanism, in your NIC to direct 829 traffic to the correct queue id and socket. 830 831Q: My packets are sometimes corrupted. What is wrong? 832 833A: Care has to be taken not to feed the same buffer in the UMEM into 834 more than one ring at the same time. If you for example feed the 835 same buffer into the FILL ring and the TX ring at the same time, the 836 NIC might receive data into the buffer at the same time it is 837 sending it. This will cause some packets to become corrupted. Same 838 thing goes for feeding the same buffer into the FILL rings 839 belonging to different queue ids or netdevs bound with the 840 XDP_SHARED_UMEM flag. 841 842Credits 843======= 844 845- Björn Töpel (AF_XDP core) 846- Magnus Karlsson (AF_XDP core) 847- Alexander Duyck 848- Alexei Starovoitov 849- Daniel Borkmann 850- Jesper Dangaard Brouer 851- John Fastabend 852- Jonathan Corbet (LWN coverage) 853- Michael S. Tsirkin 854- Qi Z Zhang 855- Willem de Bruijn 856