1========================================================
2OpenCAPI (Open Coherent Accelerator Processor Interface)
3========================================================
4
5OpenCAPI is an interface between processors and accelerators. It aims
6at being low-latency and high-bandwidth.
7
8The specification was developed by the OpenCAPI Consortium, and is now
9available from the `Compute Express Link Consortium
10<https://computeexpresslink.org/resource/opencapi-specification-archive/>`_.
11
12It allows an accelerator (which could be an FPGA, ASICs, ...) to access
13the host memory coherently, using virtual addresses. An OpenCAPI
14device can also host its own memory, that can be accessed from the
15host.
16
17OpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic
18evolution of 'cxl' (the driver for the IBM CAPI interface for
19powerpc), which was named that way to avoid confusion with the ISDN
20CAPI subsystem.
21
22
23High-level view
24===============
25
26OpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to
27be implemented on top of a physical link. Any processor or device
28implementing the DL and TL can start sharing memory.
29
30::
31
32  +-----------+                         +-------------+
33  |           |                         |             |
34  |           |                         | Accelerated |
35  | Processor |                         |  Function   |
36  |           |  +--------+             |    Unit     |  +--------+
37  |           |--| Memory |             |    (AFU)    |--| Memory |
38  |           |  +--------+             |             |  +--------+
39  +-----------+                         +-------------+
40       |                                       |
41  +-----------+                         +-------------+
42  |    TL     |                         |    TLX      |
43  +-----------+                         +-------------+
44       |                                       |
45  +-----------+                         +-------------+
46  |    DL     |                         |    DLX      |
47  +-----------+                         +-------------+
48       |                                       |
49       |                   PHY                 |
50       +---------------------------------------+
51
52
53
54Device discovery
55================
56
57OpenCAPI relies on a PCI-like configuration space, implemented on the
58device. So the host can discover AFUs by querying the config space.
59
60OpenCAPI devices in Linux are treated like PCI devices (with a few
61caveats). The firmware is expected to abstract the hardware as if it
62was a PCI link. A lot of the existing PCI infrastructure is reused:
63devices are scanned and BARs are assigned during the standard PCI
64enumeration. Commands like 'lspci' can therefore be used to see what
65devices are available.
66
67The configuration space defines the AFU(s) that can be found on the
68physical adapter, such as its name, how many memory contexts it can
69work with, the size of its MMIO areas, ...
70
71
72
73MMIO
74====
75
76OpenCAPI defines two MMIO areas for each AFU:
77
78* the global MMIO area, with registers pertinent to the whole AFU.
79* a per-process MMIO area, which has a fixed size for each context.
80
81
82
83AFU interrupts
84==============
85
86OpenCAPI includes the possibility for an AFU to send an interrupt to a
87host process. It is done through a 'intrp_req' defined in the
88Transaction Layer, specifying a 64-bit object handle which defines the
89interrupt.
90
91The driver allows a process to allocate an interrupt and obtain its
9264-bit object handle, that can be passed to the AFU.
93
94
95
96char devices
97============
98
99The driver creates one char device per AFU found on the physical
100device. A physical device may have multiple functions and each
101function can have multiple AFUs. At the time of this writing though,
102it has only been tested with devices exporting only one AFU.
103
104Char devices can be found in /dev/ocxl/ and are named as:
105/dev/ocxl/<AFU name>.<location>.<index>
106
107where <AFU name> is a max 20-character long name, as found in the
108config space of the AFU.
109<location> is added by the driver and can help distinguish devices
110when a system has more than one instance of the same OpenCAPI device.
111<index> is also to help distinguish AFUs in the unlikely case where a
112device carries multiple copies of the same AFU.
113
114
115
116Sysfs class
117===========
118
119An ocxl class is added for the devices representing the AFUs. See
120/sys/class/ocxl. The layout is described in
121Documentation/ABI/testing/sysfs-class-ocxl
122
123
124
125User API
126========
127
128open
129----
130
131Based on the AFU definition found in the config space, an AFU may
132support working with more than one memory context, in which case the
133associated char device may be opened multiple times by different
134processes.
135
136
137ioctl
138-----
139
140OCXL_IOCTL_ATTACH:
141
142  Attach the memory context of the calling process to the AFU so that
143  the AFU can access its memory.
144
145OCXL_IOCTL_IRQ_ALLOC:
146
147  Allocate an AFU interrupt and return an identifier.
148
149OCXL_IOCTL_IRQ_FREE:
150
151  Free a previously allocated AFU interrupt.
152
153OCXL_IOCTL_IRQ_SET_FD:
154
155  Associate an event fd to an AFU interrupt so that the user process
156  can be notified when the AFU sends an interrupt.
157
158OCXL_IOCTL_GET_METADATA:
159
160  Obtains configuration information from the card, such at the size of
161  MMIO areas, the AFU version, and the PASID for the current context.
162
163OCXL_IOCTL_ENABLE_P9_WAIT:
164
165  Allows the AFU to wake a userspace thread executing 'wait'. Returns
166  information to userspace to allow it to configure the AFU. Note that
167  this is only available on POWER9.
168
169OCXL_IOCTL_GET_FEATURES:
170
171  Reports on which CPU features that affect OpenCAPI are usable from
172  userspace.
173
174
175mmap
176----
177
178A process can mmap the per-process MMIO area for interactions with the
179AFU.
180