system/s390x/vfio-ap.rst

7 ------------
11 These AP devices provide cryptographic functions to all CPUs assigned to a
19 -------------------------
51   An AP queue is the means by which an AP command-request message is sent to an
57   which the AP command-request message is to be sent for processing.
63   * NQAP: to enqueue an AP command-request message to a queue
64   * DQAP: to dequeue an AP command-reply message from a queue
73 ----------------------------------------------
84   an APID from 0-255. If a bit is set, the corresponding adapter is valid for
89   corresponds to  an AP queue index (APQI) from 0-255. If a bit is set, the
94   changed by an AP command-request message sent to a usage domain from the
96   0-255. If a bit is set, the corresponding domain can be modified by an AP
97   command-request message sent to a usage domain.
100 an APQN to identify the AP adapter and AP queue to which an AP command-request
102 command-reply message is to be received (DQAP instruction). The validity of an
109 The APQNs can provide secure key functionality - i.e., a private key is stored
110 on the adapter card for each of its domains - so each APQN must be assigned to
116 +----------+--------+--------+
120 +----------+--------+--------+
122 +----------+--------+--------+
132 +----------+--------+--------+
136 +----------+--------+--------+
138 +----------+--------+--------+
148 +----------+--------+--------+
152 +----------+--------+--------+
154 +----------+--------+--------+
160 -------------------------------------
165 linux system is started, the AP bus will detect the AP devices assigned to the
169   ... [devices]
188   ... [devices]
200 A set of default device drivers are also created to control each type of AP
216 Binding AP devices to device drivers
220 range as 'usable by the default AP queue device drivers' or 'not usable by the
221 default device drivers' and thus available for use by the alternate device
227 The ``apmask`` is a 256-bit mask that identifies a set of AP adapter IDs
230 0-255. If a bit is set, the APID is marked as usable only by the default AP
234 The ``aqmask`` is a 256-bit mask that identifies a set of AP queue indexes
237 0-255. If a bit is set, the APQI is marked as usable only by the default AP
247       1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
252 and APQIs marked as usable only by the default AP queue device drivers. If a
253 match is detected,  only the default AP queue device drivers will be probed;
256 By default, the two masks are set to reserve all APQNs for use by the default
257 AP queue device drivers. There are two ways the default masks can be changed:
262     * An absolute hex string starting with 0x - like "0x12345678" - sets
278       number string must be prepended with a (``+``) or minus (``-``) to indicate
279       the corresponding bit is to be switched on (``+``) or off (``-``). Some
283            "-13"   switches bit 13 off
285            "-0xff" switches bit 255 off
289               +0,-6,+0x47,-0xf0
314             default drivers pool:    adapter 0-15, domain 1
315             alternate drivers pool:  adapter 16-255, domains 0, 2-255
322 mediated matrix device must first be created for the ``/sys/devices/vfio_ap/matrix``
325 creating mediated matrix devices is created::
327   /sys/devices
331   ............ [vfio_ap-passthrough]
333   ............... [devices]
347 the UUID is created in the ``devices`` subdirectory::
349   /sys/devices
353   ............ [vfio_ap-passthrough]
355   ............... [devices]
362   /sys/devices
366   ............ [vfio_ap-passthrough]
368   ............... [devices]
481    -device vfio_ap,sysfsdev=$path-to-mdev
486   /sys/devices/vfio_ap/matrix/$uuid
487   /sys/bus/mdev/devices/$uuid
489   /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
517    are installed on the host system. The feature is s390-specific and is
518    represented as a parameter of the -cpu option on the QEMU command line::
520       qemu-system-s390x -cpu $model,ap=on|off
530         (off). The default for CPU models zEC12 or newer
532         vfio-ap device (``-device vfio-ap,sysfsdev=$path``) is configured
540    s390-specific and is represented as a parameter of the -cpu option on the
543       qemu-system-s390x -cpu $model,apqci=on|off
552         not (off). The default for CPU models zEC12 or newer
570    s390-specific and is represented as a parameter of the -cpu option on the
573       qemu-system-s390x -cpu $model,apft=on|off
583         not (off). The default for CPU models zEC12 and
594         running on the guest will not detect CEX4 and newer devices
595         without it. Since only CEX4 and newer devices are supported
596         for guest usage, no AP devices can be made accessible to a
597         guest started without APFT installed.
599 Hot plug a vfio-ap device into a running guest
602 Only one vfio-ap device can be attached to the virtual machine's ap-bus, so a
603 vfio-ap device can be hot plugged if and only if no vfio-ap device is attached
607 To hot plug a vfio-ap device, use the QEMU ``device_add`` command::
609     (qemu) device_add vfio-ap,sysfsdev="$path-to-mdev",id="$id"
611 Where the ``$path-to-mdev`` value specifies the absolute path to a mediated
615 Note that on Linux guests, the AP devices will be created in the
616 ``/sys/bus/ap/devices`` directory when the AP bus subsequently performs its periodic
617 scan, so there may be a short delay before the AP devices are accessible on the
622 * A vfio-ap device has already been attached to the virtual machine's ap-bus.
627 Hot unplug a vfio-ap device from a running guest
630 A vfio-ap device can be unplugged from a running KVM guest if a vfio-ap device
631 has been attached to the virtual machine's ap-bus via the QEMU command line
634 To hot unplug a vfio-ap device, use the QEMU ``device_del`` command::
640 On a Linux guest, the AP devices will be removed from the ``/sys/bus/ap/devices``
642 so there may be a short delay before the AP devices are no longer accessible by
645 The command will fail if the ``$path-to-mdev`` specified on the ``device_del`` command
646 does not match the value specified when the vfio-ap device was attached to
647 the virtual machine's ap-bus.
650 -----------------------------------------------------
660   ------------------------------
661   05          CEX5C CCA-Coproc
662   05.0004     CEX5C CCA-Coproc
663   05.00ab     CEX5C CCA-Coproc
666   06.00ab     CEX5C CCA-Coproc
671   ------------------------------
679   ------------------------------
697    To build the vfio_ap module, the kernel build must be configured with the
709    If using make menuconfig select the following to build the vfio_ap module::
710      -> Device Drivers
711         -> IOMMU Hardware Support
713         -> VFIO Non-Privileged userspace driver framework
714            -> Mediated device driver framework
715               -> VFIO driver for Mediated devices
716      -> I/O subsystem
717         -> VFIO support for AP devices
722    the corresponding APQNs must be removed from the default queue drivers pool
725       echo -5,-6 > /sys/bus/ap/apmask
727       echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
732    to the AP queue devices bound to it::
749    supporting older devices that will go out of service in the relatively near
757      cat /sys/bus/ap/devices/card05/hwtype
762 3. Create the mediated devices needed to configure the AP matrixes for the
766      /sys/devices/vfio_ap/matrix/
768      ...... [vfio_ap-passthrough] (passthrough mediated matrix device type)
770      ......... [devices]
772    To create the mediated devices for the three guests::
786    This will create three mediated devices in the [devices] subdirectory named
790      /sys/devices/vfio_ap/matrix/
792      ...... [vfio_ap-passthrough]
793      ......... [devices]
822    devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
865 …/usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on -device vfio-ap,sysfsdev=/sys/devi…
869 …/usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on -device vfio-ap,sysfsdev=/sys/devi…
873 …/usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on -device vfio-ap,sysfsdev=/sys/devi…
875 When the guest is shut down, the mediated matrix devices may be removed.
879    /sys/devices/vfio_ap/matrix/
881    ...... [vfio_ap-passthrough]
882    ......... [devices]
897 the pool of adapters and queues reserved for use by the default drivers.
900 -----------
903   to the default drivers pool of a queue that is still assigned to a mediated
910   mediated matrix device - see `Configuring an AP matrix for a linux guest`_
911   section above - while a running guest is using it is currently not supported.
913 * Live guest migration is not supported for guests using AP devices. If a guest
914   is using AP devices, the vfio-ap device configured for the guest must be
915   unplugged before migrating the guest (see `Hot unplug a vfio-ap device from a