1.. SPDX-License-Identifier: GPL-2.0 2 3==================================== 4Nested KVM on POWER 5==================================== 6 7Introduction 8============ 9 10This document explains how a guest operating system can act as a 11hypervisor and run nested guests through the use of hypercalls, if the 12hypervisor has implemented them. The terms L0, L1, and L2 are used to 13refer to different software entities. L0 is the hypervisor mode entity 14that would normally be called the "host" or "hypervisor". L1 is a 15guest virtual machine that is directly run under L0 and is initiated 16and controlled by L0. L2 is a guest virtual machine that is initiated 17and controlled by L1 acting as a hypervisor. 18 19Existing API 20============ 21 22Linux/KVM has had support for Nesting as an L0 or L1 since 2018 23 24The L0 code was added:: 25 26 commit 8e3f5fc1045dc49fd175b978c5457f5f51e7a2ce 27 Author: Paul Mackerras <paulus@ozlabs.org> 28 Date: Mon Oct 8 16:31:03 2018 +1100 29 KVM: PPC: Book3S HV: Framework and hcall stubs for nested virtualization 30 31The L1 code was added:: 32 33 commit 360cae313702cdd0b90f82c261a8302fecef030a 34 Author: Paul Mackerras <paulus@ozlabs.org> 35 Date: Mon Oct 8 16:31:04 2018 +1100 36 KVM: PPC: Book3S HV: Nested guest entry via hypercall 37 38This API works primarily using a single hcall h_enter_nested(). This 39call made by the L1 to tell the L0 to start an L2 vCPU with the given 40state. The L0 then starts this L2 and runs until an L2 exit condition 41is reached. Once the L2 exits, the state of the L2 is given back to 42the L1 by the L0. The full L2 vCPU state is always transferred from 43and to L1 when the L2 is run. The L0 doesn't keep any state on the L2 44vCPU (except in the short sequence in the L0 on L1 -> L2 entry and L2 45-> L1 exit). 46 47The only state kept by the L0 is the partition table. The L1 registers 48it's partition table using the h_set_partition_table() hcall. All 49other state held by the L0 about the L2s is cached state (such as 50shadow page tables). 51 52The L1 may run any L2 or vCPU without first informing the L0. It 53simply starts the vCPU using h_enter_nested(). The creation of L2s and 54vCPUs is done implicitly whenever h_enter_nested() is called. 55 56In this document, we call this existing API the v1 API. 57 58New PAPR API 59=============== 60 61The new PAPR API changes from the v1 API such that the creating L2 and 62associated vCPUs is explicit. In this document, we call this the v2 63API. 64 65h_enter_nested() is replaced with H_GUEST_VCPU_RUN(). Before this can 66be called the L1 must explicitly create the L2 using h_guest_create() 67and any associated vCPUs() created with h_guest_create_vCPU(). Getting 68and setting vCPU state can also be performed using h_guest_{g|s}et 69hcall. 70 71The basic execution flow is for an L1 to create an L2, run it, and 72delete it is: 73 74- L1 and L0 negotiate capabilities with H_GUEST_{G,S}ET_CAPABILITIES() 75 (normally at L1 boot time). 76 77- L1 requests the L0 create an L2 with H_GUEST_CREATE() and receives a token 78 79- L1 requests the L0 create an L2 vCPU with H_GUEST_CREATE_VCPU() 80 81- L1 and L0 communicate the vCPU state using the H_GUEST_{G,S}ET() hcall 82 83- L1 requests the L0 runs the vCPU running H_GUEST_VCPU_RUN() hcall 84 85- L1 deletes L2 with H_GUEST_DELETE() 86 87More details of the individual hcalls follows: 88 89HCALL Details 90============= 91 92This documentation is provided to give an overall understating of the 93API. It doesn't aim to provide all the details required to implement 94an L1 or L0. Latest version of PAPR can be referred to for more details. 95 96All these HCALLs are made by the L1 to the L0. 97 98H_GUEST_GET_CAPABILITIES() 99-------------------------- 100 101This is called to get the capabilities of the L0 nested 102hypervisor. This includes capabilities such the CPU versions (eg 103POWER9, POWER10) that are supported as L2s:: 104 105 H_GUEST_GET_CAPABILITIES(uint64 flags) 106 107 Parameters: 108 Input: 109 flags: Reserved 110 Output: 111 R3: Return code 112 R4: Hypervisor Supported Capabilities bitmap 1 113 114H_GUEST_SET_CAPABILITIES() 115-------------------------- 116 117This is called to inform the L0 of the capabilities of the L1 118hypervisor. The set of flags passed here are the same as 119H_GUEST_GET_CAPABILITIES() 120 121Typically, GET will be called first and then SET will be called with a 122subset of the flags returned from GET. This process allows the L0 and 123L1 to negotiate an agreed set of capabilities:: 124 125 H_GUEST_SET_CAPABILITIES(uint64 flags, 126 uint64 capabilitiesBitmap1) 127 Parameters: 128 Input: 129 flags: Reserved 130 capabilitiesBitmap1: Only capabilities advertised through 131 H_GUEST_GET_CAPABILITIES 132 Output: 133 R3: Return code 134 R4: If R3 = H_P2: The number of invalid bitmaps 135 R5: If R3 = H_P2: The index of first invalid bitmap 136 137H_GUEST_CREATE() 138---------------- 139 140This is called to create an L2. A unique ID of the L2 created 141(similar to an LPID) is returned, which can be used on subsequent HCALLs to 142identify the L2:: 143 144 H_GUEST_CREATE(uint64 flags, 145 uint64 continueToken); 146 Parameters: 147 Input: 148 flags: Reserved 149 continueToken: Initial call set to -1. Subsequent calls, 150 after H_Busy or H_LongBusyOrder has been 151 returned, value that was returned in R4. 152 Output: 153 R3: Return code. Notable: 154 H_Not_Enough_Resources: Unable to create Guest VCPU due to not 155 enough Hypervisor memory. See H_GUEST_CREATE_GET_STATE(flags = 156 takeOwnershipOfVcpuState) 157 R4: If R3 = H_Busy or_H_LongBusyOrder -> continueToken 158 159H_GUEST_CREATE_VCPU() 160--------------------- 161 162This is called to create a vCPU associated with an L2. The L2 id 163(returned from H_GUEST_CREATE()) should be passed it. Also passed in 164is a unique (for this L2) vCPUid. This vCPUid is allocated by the 165L1:: 166 167 H_GUEST_CREATE_VCPU(uint64 flags, 168 uint64 guestId, 169 uint64 vcpuId); 170 Parameters: 171 Input: 172 flags: Reserved 173 guestId: ID obtained from H_GUEST_CREATE 174 vcpuId: ID of the vCPU to be created. This must be within the 175 range of 0 to 2047 176 Output: 177 R3: Return code. Notable: 178 H_Not_Enough_Resources: Unable to create Guest VCPU due to not 179 enough Hypervisor memory. See H_GUEST_CREATE_GET_STATE(flags = 180 takeOwnershipOfVcpuState) 181 182H_GUEST_GET_STATE() 183------------------- 184 185This is called to get state associated with an L2 (Guest-wide or vCPU specific). 186This info is passed via the Guest State Buffer (GSB), a standard format as 187explained later in this doc, necessary details below: 188 189This can get either L2 wide or vcpu specific information. Examples of 190L2 wide is the timebase offset or process scoped page table 191info. Examples of vCPU specific are GPRs or VSRs. A bit in the flags 192parameter specifies if this call is L2 wide or vCPU specific and the 193IDs in the GSB must match this. 194 195The L1 provides a pointer to the GSB as a parameter to this call. Also 196provided is the L2 and vCPU IDs associated with the state to set. 197 198The L1 writes only the IDs and sizes in the GSB. L0 writes the 199associated values for each ID in the GSB:: 200 201 H_GUEST_GET_STATE(uint64 flags, 202 uint64 guestId, 203 uint64 vcpuId, 204 uint64 dataBuffer, 205 uint64 dataBufferSizeInBytes); 206 Parameters: 207 Input: 208 flags: 209 Bit 0: getGuestWideState: Request state of the Guest instead 210 of an individual VCPU. 211 Bit 1: getHostWideState: Request stats of the Host. This causes 212 the guestId and vcpuId parameters to be ignored and attempting 213 to get the VCPU/Guest state will cause an error. 214 Bits 2-63: Reserved 215 guestId: ID obtained from H_GUEST_CREATE 216 vcpuId: ID of the vCPU pass to H_GUEST_CREATE_VCPU 217 dataBuffer: A L1 real address of the GSB. 218 If takeOwnershipOfVcpuState, size must be at least the size 219 returned by ID=0x0001 220 dataBufferSizeInBytes: Size of dataBuffer 221 Output: 222 R3: Return code 223 R4: If R3 = H_Invalid_Element_Id: The array index of the bad 224 element ID. 225 If R3 = H_Invalid_Element_Size: The array index of the bad 226 element size. 227 If R3 = H_Invalid_Element_Value: The array index of the bad 228 element value. 229 230H_GUEST_SET_STATE() 231------------------- 232 233This is called to set L2 wide or vCPU specific L2 state. This info is 234passed via the Guest State Buffer (GSB), necessary details below: 235 236This can set either L2 wide or vcpu specific information. Examples of 237L2 wide is the timebase offset or process scoped page table 238info. Examples of vCPU specific are GPRs or VSRs. A bit in the flags 239parameter specifies if this call is L2 wide or vCPU specific and the 240IDs in the GSB must match this. 241 242The L1 provides a pointer to the GSB as a parameter to this call. Also 243provided is the L2 and vCPU IDs associated with the state to set. 244 245The L1 writes all values in the GSB and the L0 only reads the GSB for 246this call:: 247 248 H_GUEST_SET_STATE(uint64 flags, 249 uint64 guestId, 250 uint64 vcpuId, 251 uint64 dataBuffer, 252 uint64 dataBufferSizeInBytes); 253 Parameters: 254 Input: 255 flags: 256 Bit 0: getGuestWideState: Request state of the Guest instead 257 of an individual VCPU. 258 Bit 1: returnOwnershipOfVcpuState Return Guest VCPU state. See 259 GET_STATE takeOwnershipOfVcpuState 260 Bits 2-63: Reserved 261 guestId: ID obtained from H_GUEST_CREATE 262 vcpuId: ID of the vCPU pass to H_GUEST_CREATE_VCPU 263 dataBuffer: A L1 real address of the GSB. 264 If takeOwnershipOfVcpuState, size must be at least the size 265 returned by ID=0x0001 266 dataBufferSizeInBytes: Size of dataBuffer 267 Output: 268 R3: Return code 269 R4: If R3 = H_Invalid_Element_Id: The array index of the bad 270 element ID. 271 If R3 = H_Invalid_Element_Size: The array index of the bad 272 element size. 273 If R3 = H_Invalid_Element_Value: The array index of the bad 274 element value. 275 276H_GUEST_RUN_VCPU() 277------------------ 278 279This is called to run an L2 vCPU. The L2 and vCPU IDs are passed in as 280parameters. The vCPU runs with the state set previously using 281H_GUEST_SET_STATE(). When the L2 exits, the L1 will resume from this 282hcall. 283 284This hcall also has associated input and output GSBs. Unlike 285H_GUEST_{S,G}ET_STATE(), these GSB pointers are not passed in as 286parameters to the hcall (This was done in the interest of 287performance). The locations of these GSBs must be preregistered using 288the H_GUEST_SET_STATE() call with ID 0x0c00 and 0x0c01 (see table 289below). 290 291The input GSB may contain only VCPU specific elements to be set. This 292GSB may also contain zero elements (ie 0 in the first 4 bytes of the 293GSB) if nothing needs to be set. 294 295On exit from the hcall, the output buffer is filled with elements 296determined by the L0. The reason for the exit is contained in GPR4 (ie 297NIP is put in GPR4). The elements returned depend on the exit 298type. For example, if the exit reason is the L2 doing a hcall (GPR4 = 2990xc00), then GPR3-12 are provided in the output GSB as this is the 300state likely needed to service the hcall. If additional state is 301needed, H_GUEST_GET_STATE() may be called by the L1. 302 303To synthesize interrupts in the L2, when calling H_GUEST_RUN_VCPU() 304the L1 may set a flag (as a hcall parameter) and the L0 will 305synthesize the interrupt in the L2. Alternatively, the L1 may 306synthesize the interrupt itself using H_GUEST_SET_STATE() or the 307H_GUEST_RUN_VCPU() input GSB to set the state appropriately:: 308 309 H_GUEST_RUN_VCPU(uint64 flags, 310 uint64 guestId, 311 uint64 vcpuId, 312 uint64 dataBuffer, 313 uint64 dataBufferSizeInBytes); 314 Parameters: 315 Input: 316 flags: 317 Bit 0: generateExternalInterrupt: Generate an external interrupt 318 Bit 1: generatePrivilegedDoorbell: Generate a Privileged Doorbell 319 Bit 2: sendToSystemReset”: Generate a System Reset Interrupt 320 Bits 3-63: Reserved 321 guestId: ID obtained from H_GUEST_CREATE 322 vcpuId: ID of the vCPU pass to H_GUEST_CREATE_VCPU 323 Output: 324 R3: Return code 325 R4: If R3 = H_Success: The reason L1 VCPU exited (ie. NIA) 326 0x000: The VCPU stopped running for an unspecified reason. An 327 example of this is the Hypervisor stopping a VCPU running 328 due to an outstanding interrupt for the Host Partition. 329 0x980: HDEC 330 0xC00: HCALL 331 0xE00: HDSI 332 0xE20: HISI 333 0xE40: HEA 334 0xF80: HV Fac Unavail 335 If R3 = H_Invalid_Element_Id, H_Invalid_Element_Size, or 336 H_Invalid_Element_Value: R4 is offset of the invalid element 337 in the input buffer. 338 339H_GUEST_DELETE() 340---------------- 341 342This is called to delete an L2. All associated vCPUs are also 343deleted. No specific vCPU delete call is provided. 344 345A flag may be provided to delete all guests. This is used to reset the 346L0 in the case of kdump/kexec:: 347 348 H_GUEST_DELETE(uint64 flags, 349 uint64 guestId) 350 Parameters: 351 Input: 352 flags: 353 Bit 0: deleteAllGuests: deletes all guests 354 Bits 1-63: Reserved 355 guestId: ID obtained from H_GUEST_CREATE 356 Output: 357 R3: Return code 358 359Guest State Buffer 360================== 361 362The Guest State Buffer (GSB) is the main method of communicating state 363about the L2 between the L1 and L0 via H_GUEST_{G,S}ET() and 364H_GUEST_VCPU_RUN() calls. 365 366State may be associated with a whole L2 (eg timebase offset) or a 367specific L2 vCPU (eg. GPR state). Only L2 VCPU state maybe be set by 368H_GUEST_VCPU_RUN(). 369 370All data in the GSB is big endian (as is standard in PAPR) 371 372The Guest state buffer has a header which gives the number of 373elements, followed by the GSB elements themselves. 374 375GSB header: 376 377+----------+----------+-------------------------------------------+ 378| Offset | Size | Purpose | 379| Bytes | Bytes | | 380+==========+==========+===========================================+ 381| 0 | 4 | Number of elements | 382+----------+----------+-------------------------------------------+ 383| 4 | | Guest state buffer elements | 384+----------+----------+-------------------------------------------+ 385 386GSB element: 387 388+----------+----------+-------------------------------------------+ 389| Offset | Size | Purpose | 390| Bytes | Bytes | | 391+==========+==========+===========================================+ 392| 0 | 2 | ID | 393+----------+----------+-------------------------------------------+ 394| 2 | 2 | Size of Value | 395+----------+----------+-------------------------------------------+ 396| 4 | As above | Value | 397+----------+----------+-------------------------------------------+ 398 399The ID in the GSB element specifies what is to be set. This includes 400archtected state like GPRs, VSRs, SPRs, plus also some meta data about 401the partition like the timebase offset and partition scoped page 402table information. 403 404+--------+-------+----+--------+----------------------------------+ 405| ID | Size | RW |(H)ost | Details | 406| | Bytes | |(G)uest | | 407| | | |(T)hread| | 408| | | |Scope | | 409+========+=======+====+========+==================================+ 410| 0x0000 | | RW | TG | NOP element | 411+--------+-------+----+--------+----------------------------------+ 412| 0x0001 | 0x08 | R | G | Size of L0 vCPU state. See: | 413| | | | | H_GUEST_GET_STATE: | 414| | | | | flags = takeOwnershipOfVcpuState | 415+--------+-------+----+--------+----------------------------------+ 416| 0x0002 | 0x08 | R | G | Size Run vCPU out buffer | 417+--------+-------+----+--------+----------------------------------+ 418| 0x0003 | 0x04 | RW | G | Logical PVR | 419+--------+-------+----+--------+----------------------------------+ 420| 0x0004 | 0x08 | RW | G | TB Offset (L1 relative) | 421+--------+-------+----+--------+----------------------------------+ 422| 0x0005 | 0x18 | RW | G |Partition scoped page tbl info: | 423| | | | | | 424| | | | |- 0x00 Addr part scope table | 425| | | | |- 0x08 Num addr bits | 426| | | | |- 0x10 Size root dir | 427+--------+-------+----+--------+----------------------------------+ 428| 0x0006 | 0x10 | RW | G |Process Table Information: | 429| | | | | | 430| | | | |- 0x0 Addr proc scope table | 431| | | | |- 0x8 Table size. | 432+--------+-------+----+--------+----------------------------------+ 433| 0x0007-| | | | Reserved | 434| 0x07FF | | | | | 435+--------+-------+----+--------+----------------------------------+ 436| 0x0800 | 0x08 | R | H | Current usage in bytes of the | 437| | | | | L0's Guest Management Space | 438| | | | | for an L1-Lpar. | 439+--------+-------+----+--------+----------------------------------+ 440| 0x0801 | 0x08 | R | H | Max bytes available in the | 441| | | | | L0's Guest Management Space for | 442| | | | | an L1-Lpar | 443+--------+-------+----+--------+----------------------------------+ 444| 0x0802 | 0x08 | R | H | Current usage in bytes of the | 445| | | | | L0's Guest Page Table Management | 446| | | | | Space for an L1-Lpar | 447+--------+-------+----+--------+----------------------------------+ 448| 0x0803 | 0x08 | R | H | Max bytes available in the L0's | 449| | | | | Guest Page Table Management | 450| | | | | Space for an L1-Lpar | 451+--------+-------+----+--------+----------------------------------+ 452| 0x0804 | 0x08 | R | H | Cumulative Reclaimed bytes from | 453| | | | | L0 Guest's Page Table Management | 454| | | | | Space due to overcommit | 455+--------+-------+----+--------+----------------------------------+ 456| 0x0805-| | | | Reserved | 457| 0x0BFF | | | | | 458+--------+-------+----+--------+----------------------------------+ 459| 0x0C00 | 0x10 | RW | T |Run vCPU Input Buffer: | 460| | | | | | 461| | | | |- 0x0 Addr of buffer | 462| | | | |- 0x8 Buffer Size. | 463+--------+-------+----+--------+----------------------------------+ 464| 0x0C01 | 0x10 | RW | T |Run vCPU Output Buffer: | 465| | | | | | 466| | | | |- 0x0 Addr of buffer | 467| | | | |- 0x8 Buffer Size. | 468+--------+-------+----+--------+----------------------------------+ 469| 0x0C02 | 0x08 | RW | T | vCPU VPA Address | 470+--------+-------+----+--------+----------------------------------+ 471| 0x0C03-| | | | Reserved | 472| 0x0FFF | | | | | 473+--------+-------+----+--------+----------------------------------+ 474| 0x1000-| 0x08 | RW | T | GPR 0-31 | 475| 0x101F | | | | | 476+--------+-------+----+--------+----------------------------------+ 477| 0x1020 | 0x08 | T | T | HDEC expiry TB | 478+--------+-------+----+--------+----------------------------------+ 479| 0x1021 | 0x08 | RW | T | NIA | 480+--------+-------+----+--------+----------------------------------+ 481| 0x1022 | 0x08 | RW | T | MSR | 482+--------+-------+----+--------+----------------------------------+ 483| 0x1023 | 0x08 | RW | T | LR | 484+--------+-------+----+--------+----------------------------------+ 485| 0x1024 | 0x08 | RW | T | XER | 486+--------+-------+----+--------+----------------------------------+ 487| 0x1025 | 0x08 | RW | T | CTR | 488+--------+-------+----+--------+----------------------------------+ 489| 0x1026 | 0x08 | RW | T | CFAR | 490+--------+-------+----+--------+----------------------------------+ 491| 0x1027 | 0x08 | RW | T | SRR0 | 492+--------+-------+----+--------+----------------------------------+ 493| 0x1028 | 0x08 | RW | T | SRR1 | 494+--------+-------+----+--------+----------------------------------+ 495| 0x1029 | 0x08 | RW | T | DAR | 496+--------+-------+----+--------+----------------------------------+ 497| 0x102A | 0x08 | RW | T | DEC expiry TB | 498+--------+-------+----+--------+----------------------------------+ 499| 0x102B | 0x08 | RW | T | VTB | 500+--------+-------+----+--------+----------------------------------+ 501| 0x102C | 0x08 | RW | T | LPCR | 502+--------+-------+----+--------+----------------------------------+ 503| 0x102D | 0x08 | RW | T | HFSCR | 504+--------+-------+----+--------+----------------------------------+ 505| 0x102E | 0x08 | RW | T | FSCR | 506+--------+-------+----+--------+----------------------------------+ 507| 0x102F | 0x08 | RW | T | FPSCR | 508+--------+-------+----+--------+----------------------------------+ 509| 0x1030 | 0x08 | RW | T | DAWR0 | 510+--------+-------+----+--------+----------------------------------+ 511| 0x1031 | 0x08 | RW | T | DAWR1 | 512+--------+-------+----+--------+----------------------------------+ 513| 0x1032 | 0x08 | RW | T | CIABR | 514+--------+-------+----+--------+----------------------------------+ 515| 0x1033 | 0x08 | RW | T | PURR | 516+--------+-------+----+--------+----------------------------------+ 517| 0x1034 | 0x08 | RW | T | SPURR | 518+--------+-------+----+--------+----------------------------------+ 519| 0x1035 | 0x08 | RW | T | IC | 520+--------+-------+----+--------+----------------------------------+ 521| 0x1036-| 0x08 | RW | T | SPRG 0-3 | 522| 0x1039 | | | | | 523+--------+-------+----+--------+----------------------------------+ 524| 0x103A | 0x08 | W | T | PPR | 525+--------+-------+----+--------+----------------------------------+ 526| 0x103B | 0x08 | RW | T | MMCR 0-3 | 527| 0x103E | | | | | 528+--------+-------+----+--------+----------------------------------+ 529| 0x103F | 0x08 | RW | T | MMCRA | 530+--------+-------+----+--------+----------------------------------+ 531| 0x1040 | 0x08 | RW | T | SIER | 532+--------+-------+----+--------+----------------------------------+ 533| 0x1041 | 0x08 | RW | T | SIER 2 | 534+--------+-------+----+--------+----------------------------------+ 535| 0x1042 | 0x08 | RW | T | SIER 3 | 536+--------+-------+----+--------+----------------------------------+ 537| 0x1043 | 0x08 | RW | T | BESCR | 538+--------+-------+----+--------+----------------------------------+ 539| 0x1044 | 0x08 | RW | T | EBBHR | 540+--------+-------+----+--------+----------------------------------+ 541| 0x1045 | 0x08 | RW | T | EBBRR | 542+--------+-------+----+--------+----------------------------------+ 543| 0x1046 | 0x08 | RW | T | AMR | 544+--------+-------+----+--------+----------------------------------+ 545| 0x1047 | 0x08 | RW | T | IAMR | 546+--------+-------+----+--------+----------------------------------+ 547| 0x1048 | 0x08 | RW | T | AMOR | 548+--------+-------+----+--------+----------------------------------+ 549| 0x1049 | 0x08 | RW | T | UAMOR | 550+--------+-------+----+--------+----------------------------------+ 551| 0x104A | 0x08 | RW | T | SDAR | 552+--------+-------+----+--------+----------------------------------+ 553| 0x104B | 0x08 | RW | T | SIAR | 554+--------+-------+----+--------+----------------------------------+ 555| 0x104C | 0x08 | RW | T | DSCR | 556+--------+-------+----+--------+----------------------------------+ 557| 0x104D | 0x08 | RW | T | TAR | 558+--------+-------+----+--------+----------------------------------+ 559| 0x104E | 0x08 | RW | T | DEXCR | 560+--------+-------+----+--------+----------------------------------+ 561| 0x104F | 0x08 | RW | T | HDEXCR | 562+--------+-------+----+--------+----------------------------------+ 563| 0x1050 | 0x08 | RW | T | HASHKEYR | 564+--------+-------+----+--------+----------------------------------+ 565| 0x1051 | 0x08 | RW | T | HASHPKEYR | 566+--------+-------+----+--------+----------------------------------+ 567| 0x1052 | 0x08 | RW | T | CTRL | 568+--------+-------+----+--------+----------------------------------+ 569| 0x1053 | 0x08 | RW | T | DPDES | 570+--------+-------+----+--------+----------------------------------+ 571| 0x1054-| | | | Reserved | 572| 0x1FFF | | | | | 573+--------+-------+----+--------+----------------------------------+ 574| 0x2000 | 0x04 | RW | T | CR | 575+--------+-------+----+--------+----------------------------------+ 576| 0x2001 | 0x04 | RW | T | PIDR | 577+--------+-------+----+--------+----------------------------------+ 578| 0x2002 | 0x04 | RW | T | DSISR | 579+--------+-------+----+--------+----------------------------------+ 580| 0x2003 | 0x04 | RW | T | VSCR | 581+--------+-------+----+--------+----------------------------------+ 582| 0x2004 | 0x04 | RW | T | VRSAVE | 583+--------+-------+----+--------+----------------------------------+ 584| 0x2005 | 0x04 | RW | T | DAWRX0 | 585+--------+-------+----+--------+----------------------------------+ 586| 0x2006 | 0x04 | RW | T | DAWRX1 | 587+--------+-------+----+--------+----------------------------------+ 588| 0x2007-| 0x04 | RW | T | PMC 1-6 | 589| 0x200c | | | | | 590+--------+-------+----+--------+----------------------------------+ 591| 0x200D | 0x04 | RW | T | WORT | 592+--------+-------+----+--------+----------------------------------+ 593| 0x200E | 0x04 | RW | T | PSPB | 594+--------+-------+----+--------+----------------------------------+ 595| 0x200F-| | | | Reserved | 596| 0x2FFF | | | | | 597+--------+-------+----+--------+----------------------------------+ 598| 0x3000-| 0x10 | RW | T | VSR 0-63 | 599| 0x303F | | | | | 600+--------+-------+----+--------+----------------------------------+ 601| 0x3040-| | | | Reserved | 602| 0xEFFF | | | | | 603+--------+-------+----+--------+----------------------------------+ 604| 0xF000 | 0x08 | R | T | HDAR | 605+--------+-------+----+--------+----------------------------------+ 606| 0xF001 | 0x04 | R | T | HDSISR | 607+--------+-------+----+--------+----------------------------------+ 608| 0xF002 | 0x04 | R | T | HEIR | 609+--------+-------+----+--------+----------------------------------+ 610| 0xF003 | 0x08 | R | T | ASDR | 611+--------+-------+----+--------+----------------------------------+ 612 613 614Miscellaneous info 615================== 616 617State not in ptregs/hvregs 618-------------------------- 619 620In the v1 API, some state is not in the ptregs/hvstate. This includes 621the vector register and some SPRs. For the L1 to set this state for 622the L2, the L1 loads up these hardware registers before the 623h_enter_nested() call and the L0 ensures they end up as the L2 state 624(by not touching them). 625 626The v2 API removes this and explicitly sets this state via the GSB. 627 628L1 Implementation details: Caching state 629---------------------------------------- 630 631In the v1 API, all state is sent from the L1 to the L0 and vice versa 632on every h_enter_nested() hcall. If the L0 is not currently running 633any L2s, the L0 has no state information about them. The only 634exception to this is the location of the partition table, registered 635via h_set_partition_table(). 636 637The v2 API changes this so that the L0 retains the L2 state even when 638it's vCPUs are no longer running. This means that the L1 only needs to 639communicate with the L0 about L2 state when it needs to modify the L2 640state, or when it's value is out of date. This provides an opportunity 641for performance optimisation. 642 643When a vCPU exits from a H_GUEST_RUN_VCPU() call, the L1 internally 644marks all L2 state as invalid. This means that if the L1 wants to know 645the L2 state (say via a kvm_get_one_reg() call), it needs call 646H_GUEST_GET_STATE() to get that state. Once it's read, it's marked as 647valid in L1 until the L2 is run again. 648 649Also, when an L1 modifies L2 vcpu state, it doesn't need to write it 650to the L0 until that L2 vcpu runs again. Hence when the L1 updates 651state (say via a kvm_set_one_reg() call), it writes to an internal L1 652copy and only flushes this copy to the L0 when the L2 runs again via 653the H_GUEST_VCPU_RUN() input buffer. 654 655This lazy updating of state by the L1 avoids unnecessary 656H_GUEST_{G|S}ET_STATE() calls. 657