Lines Matching full:the
9 The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
14 On s390x, AP adapter cards are exposed via the AP bus. This document
15 describes how those cards may be made available to KVM guests using the
21 In order understand the terminology used in the rest of this document, let's
28 on the machine model. Adapters assigned to the LPAR in which a linux host is
29 running will be available to the linux host. Each adapter is identified by a
30 number from 0 to 255; however, the maximum adapter number allowed is
38 hold up to 256 domains; however, the maximum domain number allowed is
46 commands sent to a usage domain to control or change the domain; for
47 example, to set a secure private key for the domain.
51 An AP queue is the means by which an AP command-request message is sent to an
53 comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
54 APQI corresponds to a given usage domain number within the adapter. This tuple
56 instructions include a field containing the APQN to identify the AP queue to
57 which the AP command-request message is to be sent for processing.
65 * PQAP: to administer the queues
67 AP instructions identify the domain that is targeted to process the AP
68 command; this must be one of the usage domains. An AP command may modify a
69 domain that is not one of the usage domains, but the modified domain
70 must be one of the control domains.
75 A KVM guest is started by executing the Start Interpretive Execution (SIE)
76 instruction. The SIE state description is a control block that contains the
77 state information for a KVM guest and is supplied as input to the SIE
78 instruction. The SIE state description contains a satellite control block called
79 the Crypto Control Block (CRYCB). The CRYCB contains three fields to identify
80 the adapters, usage domains and control domains assigned to the KVM guest:
82 * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
83 to the KVM guest. Each bit in the mask, from left to right, corresponds to
84 an APID from 0-255. If a bit is set, the corresponding adapter is valid for
85 use by the KVM guest.
87 * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
88 assigned to the KVM guest. Each bit in the mask, from left to right,
89 corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the
90 corresponding queue is valid for use by the KVM guest.
92 * The AP Domain Mask field is a bit mask that identifies the AP control domains
93 assigned to the KVM guest. The ADM bit mask controls which domains can be
94 changed by an AP command-request message sent to a usage domain from the
95 guest. Each bit in the mask, from left to right, corresponds to a domain from
96 0-255. If a bit is set, the corresponding domain can be modified by an AP
99 If you recall from the description of an AP Queue, AP instructions include
100 an APQN to identify the AP adapter and AP queue to which an AP command-request
102 command-reply message is to be received (DQAP instruction). The validity of an
103 APQN is defined by the matrix calculated from the APM and AQM; it is the
106 assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
107 the guest.
109 The APQNs can provide secure key functionality - i.e., a private key is stored
110 on the adapter card for each of its domains - so each APQN must be assigned to
111 at most one guest or the linux host.
162 A linux system is a guest of the LPAR in which it is running and has access to
163 the AP resources configured for the LPAR. The LPAR's AP matrix is
164 configured via its Activation Profile which can be edited on the HMC. When the
165 linux system is started, the AP bus will detect the AP devices assigned to the
166 LPAR and create the following in sysfs::
181 is an APQN with xx specifying the APID and yyyy specifying the APQI
184 255 (0xff) are configured for the LPAR, the sysfs representation on the linux
201 device that can be assigned to the LPAR on which a linux host is running::
219 There are two sysfs files that specify bitmasks marking a subset of the APQN
220 range as 'usable by the default AP queue device drivers' or 'not usable by the
221 default device drivers' and thus available for use by the alternate device
222 driver(s). The sysfs locations of the masks are::
227 The ``apmask`` is a 256-bit mask that identifies a set of AP adapter IDs
228 (APID). Each bit in the mask, from left to right (i.e., from most significant
230 0-255. If a bit is set, the APID is marked as usable only by the default AP
231 queue device drivers; otherwise, the APID is usable by the vfio_ap
234 The ``aqmask`` is a 256-bit mask that identifies a set of AP queue indexes
235 (APQI). Each bit in the mask, from left to right (i.e., from most significant
237 0-255. If a bit is set, the APQI is marked as usable only by the default AP
238 queue device drivers; otherwise, the APQI is usable by the vfio_ap device
241 Take, for example, the following mask::
247 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
248 belong to the vfio_ap device driver's pool.
250 The APQN of each AP queue device assigned to the linux host is checked by the
251 AP bus against the set of APQNs derived from the cross product of APIDs
252 and APQIs marked as usable only by the default AP queue device drivers. If a
253 match is detected, only the default AP queue device drivers will be probed;
254 otherwise, the vfio_ap device driver will be probed.
256 By default, the two masks are set to reserve all APQNs for use by the default
257 AP queue device drivers. There are two ways the default masks can be changed:
259 1. The sysfs mask files can be edited by echoing a string into the
263 the mask. If the given string is shorter than the mask, it is padded
264 with 0s on the right; for example, specifying a mask value of 0x41 is
265 the same as specifying::
269 Keep in mind that the mask reads from left to right (i.e., most
270 significant to least significant bit in big endian order), so the mask
273 If the string is longer than the mask, the operation is terminated with
276 * Individual bits in the mask can be switched on and off by specifying
279 the corresponding bit is to be switched on (``+``) or off (``-``). Some
287 The following example::
294 Note that the bits not specified in the list remain as they were before
295 the operation.
297 2. The masks can also be changed at boot time via parameters on the kernel
302 This would create the following masks:
320 The sysfs interfaces for configuring an AP matrix for a guest are built on the
322 mediated matrix device must first be created for the ``/sys/devices/vfio_ap/matrix``
323 device. When the vfio_ap device driver is loaded, it registers with the VFIO
324 mediated device framework. When the driver registers, the sysfs interfaces for
335 A mediated AP matrix device is created by writing a UUID to the attribute file
347 the UUID is created in the ``devices`` subdirectory::
358 There will also be three sets of attribute files created in the mediated
359 matrix device's sysfs directory to configure an AP matrix for the
379 To assign an AP adapter to the mediated matrix device, its APID is written
380 to the ``assign_adapter`` file. This may be done multiple times to assign more
381 than one adapter. The APID may be specified using conventional semantics
392 * The adapter number specified must represent a value from 0 up to the
393 maximum adapter number allowed by the machine model. If an adapter number
394 higher than the maximum is specified, the operation will terminate with
397 * All APQNs that can be derived from the adapter ID being assigned and the
398 IDs of the previously assigned domains must be bound to the vfio_ap device
400 one APQN with the specified APID bound to the vfio_ap driver. If no such
401 APQNs are bound to the driver, the operation will terminate with an
404 * No APQN that can be derived from the adapter ID and the IDs of the
406 device. If an APQN is assigned to another mediated matrix device, the
410 To unassign an AP adapter, its APID is written to the ``unassign_adapter``
414 To assign a usage domain, the domain number is written into the
416 usage domain. The domain number is specified using conventional semantics as
427 * The domain number specified must represent a value from 0 up to the
428 maximum domain number allowed by the machine model. If a domain number
429 higher than the maximum is specified, the operation will terminate with
432 * All APQNs that can be derived from the domain ID being assigned and the IDs
433 of the previously assigned adapters must be bound to the vfio_ap device
435 one APQN with the specified APQI bound to the vfio_ap driver. If no such
436 APQNs are bound to the driver, the operation will terminate with an
439 * No APQN that can be derived from the domain ID being assigned and the IDs
440 of the previously assigned adapters can be assigned to another mediated
442 the operation will terminate with an error (EADDRINUSE).
445 To unassign a usage domain, the domain number is written into the
450 To assign a control domain, the domain number is written into the
452 assign more than one control domain. The domain number may be specified using
461 In order to successfully assign a control domain, the domain number
462 specified must represent a value from 0 up to the maximum domain number
463 allowed by the machine model. If a control domain number higher than the
464 maximum is specified, the operation will terminate with an error (ENODEV).
467 To unassign a control domain, the domain number is written into the
471 Notes: No changes to the AP matrix will be allowed while a guest using
472 the mediated matrix device is running. Attempts to assign an adapter,
478 To provide a mediated matrix device for use by a guest, the following option
479 must be specified on the QEMU command line::
483 The sysfsdev parameter specifies the path to the mediated matrix device.
491 When the linux guest is started, the guest will open the mediated
492 matrix device's file descriptor to get information about the mediated matrix
493 device. The ``vfio_ap`` device driver will update the APM, AQM, and ADM fields in
494 the guest's CRYCB with the adapter, usage domain and control domains assigned
495 via the mediated matrix device's sysfs attribute files. Programs running on the
498 1. Have direct access to the APQNs derived from the cross product of the AP
499 adapter numbers (APID) and queue indexes (APQI) specified in the APM and AQM
500 fields of the guests's CRYCB respectively. These APQNs identify the AP queues
501 that are valid for use by the guest; meaning, AP commands can be sent by the
505 identified in the ADM field of the guest's CRYCB. The AP command must be sent
515 The AP facilities feature indicates that AP facilities are installed on the
516 guest. This feature will be exposed for use only if the AP facilities
517 are installed on the host system. The feature is s390-specific and is
518 represented as a parameter of the -cpu option on the QEMU command line::
525 is the CPU model defined for the guest (defaults to the model of
526 the host system if not specified).
530 (off). The default for CPU models zEC12 or newer
531 is ``ap=on``. AP facilities must be installed on the guest if a
533 for the guest, or the guest will fail to start.
537 The QCI facility is used by the AP bus running on the guest to query the
538 configuration of the AP facilities. This facility will be available
539 only if the QCI facility is installed on the host system. The feature is
540 s390-specific and is represented as a parameter of the -cpu option on the
548 is the CPU model defined for the guest
551 indicates whether the QCI facility is installed (on) or
552 not (off). The default for CPU models zEC12 or newer
556 (``ap=off``), an error message will be logged, but the guest
558 installed if the AP facilities are not; this is considered
561 If the QCI facility is not installed, APQNs with an APQI
562 greater than 15 will not be detected by the AP bus
563 running on the guest.
567 The APFT facility is used by the AP bus running on the guest to test the
569 only if the APFT facility is installed on the host system. The feature is
570 s390-specific and is represented as a parameter of the -cpu option on the
578 is the CPU model defined for the guest (defaults to the model of
579 the host system if not specified).
582 indicates whether the APFT facility is installed (on) or
583 not (off). The default for CPU models zEC12 and
588 (``ap=off``), an error message will be logged, but the guest
590 installed if the AP facilities are not; this is considered
593 It also makes no sense to turn APFT off because the AP bus
594 running on the guest will not detect CEX4 and newer devices
602 Only one vfio-ap device can be attached to the virtual machine's ap-bus, so a
604 to the bus already, whether via the QEMU command line or a prior hot plug
607 To hot plug a vfio-ap device, use the QEMU ``device_add`` command::
611 Where the ``$path-to-mdev`` value specifies the absolute path to a mediated
612 device to which AP resources to be used by the guest have been assigned.
613 ``$id`` is the name value for the optional id parameter.
615 Note that on Linux guests, the AP devices will be created in the
616 ``/sys/bus/ap/devices`` directory when the AP bus subsequently performs its periodic
617 scan, so there may be a short delay before the AP devices are accessible on the
620 The command will fail if:
622 * A vfio-ap device has already been attached to the virtual machine's ap-bus.
624 * The CPU model features for controlling guest access to AP facilities are not
625 enabled (see 'CPU model features' subsection in the previous section).
631 has been attached to the virtual machine's ap-bus via the QEMU command line
634 To hot unplug a vfio-ap device, use the QEMU ``device_del`` command::
638 Where ``$id`` is the same id that was specified at device creation.
640 On a Linux guest, the AP devices will be removed from the ``/sys/bus/ap/devices``
641 directory on the guest when the AP bus subsequently performs its periodic scan,
642 so there may be a short delay before the AP devices are no longer accessible by
643 the guest.
645 The command will fail if the ``$path-to-mdev`` specified on the ``device_del`` command
646 does not match the value specified when the vfio-ap device was attached to
647 the virtual machine's ap-bus.
654 three guests such that executing the lszcrypt command on the guests would
684 These are the steps:
686 1. Install the vfio_ap module on the linux host. The dependency chain for the
697 To build the vfio_ap module, the kernel build must be configured with the
709 If using make menuconfig select the following to build the vfio_ap module::
719 2. Secure the AP queues to be used by the three guests so that the host can not
720 access them. To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff,
721 06.0004, 06.0047, 06.00ab, and 06.00ff for use by the vfio_ap device driver,
722 the corresponding APQNs must be removed from the default queue drivers pool
730 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
731 sysfs directory for the vfio_ap device driver will now contain symbolic links
732 to the AP queue devices bound to it::
747 can be bound to the vfio_ap device driver. The reason for this is to
748 simplify the implementation by not needlessly complicating the design by
749 supporting older devices that will go out of service in the relatively near
752 The administrator, therefore, must take care to secure only AP queues that
753 can be bound to the vfio_ap device driver. The device type for a given AP
754 queue device can be read from the parent card's sysfs directory. For example,
755 to see the hardware type of the queue 05.0004::
759 The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
762 3. Create the mediated devices needed to configure the AP matrixes for the
763 three guests and to provide an interface to the vfio_ap driver for
764 use by the guests::
772 To create the mediated devices for the three guests::
786 This will create three mediated devices in the [devices] subdirectory named
787 after the UUID used to create the mediated device. We'll call them $uuid1,
788 $uuid2 and $uuid3 and this is the sysfs directory structure after creation::
821 4. The administrator now needs to configure the matrixes for the mediated
824 This is how the matrix is configured for Guest1::
831 Control domains can similarly be assigned using the assign_control_domain
835 you can use the ``unassign_xxx`` interfaces to unassign the adapter, domain or
838 To display the matrix configuration for Guest1::
842 The output will display the APQNs in the format ``xx.yyyy``, where xx is
843 the adapter number and yyyy is the domain number. The output for Guest1
851 This is how the matrix is configured for Guest2::
857 This is how the matrix is configured for Guest3::
875 When the guest is shut down, the mediated matrix devices may be removed.
877 Using our example again, to remove the mediated matrix device $uuid1::
889 This will remove all of the mdev matrix device's sysfs structures including
890 the mdev device itself. To recreate and reconfigure the mdev matrix device,
891 all of the steps starting with step 3 will have to be performed again. Note
892 that the remove will fail if a guest using the mdev is still running.
895 remove it if no guest will use it during the remaining lifetime of the linux
896 host. If the mdev matrix device is removed, one may want to also reconfigure
897 the pool of adapters and queues reserved for use by the default drivers.
902 * The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
903 to the default drivers pool of a queue that is still assigned to a mediated
904 device in use by a guest. It is incumbent upon the administrator to
905 ensure there is no mediated device in use by a guest to which the APQN is
906 assigned lest the host be given access to the private data of the AP queue
907 device, such as a private key configured specifically for the guest.
914 is using AP devices, the vfio-ap device configured for the guest must be
915 unplugged before migrating the guest (see `Hot unplug a vfio-ap device from a