arm/arm64: Remove dcache_line_size global variable

Compute the dcache line size when doing dcache maintenance instead of using
a global variable computed in setup(), which allows us to do dcache
mai

arm/arm64: Remove dcache_line_size global variable

Compute the dcache line size when doing dcache maintenance instead of using
a global variable computed in setup(), which allows us to do dcache
maintenance at any point in the boot process. This will be useful for
running as an EFI app and it also aligns our implementation to that of the
Linux kernel. As a result, the dcache_by_line_op assembly has been modified
to take a range described by start address and size, instead of start and
end addresses.

For consistency, the arm code has been similary modified.

Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: Read system registers to get the state of the MMU

When we are in EL1 we can directly tell if the local cpu's MMU is on
by reading a system register (SCTRL/SCTRL_EL1). In EL0, we use the
r

arm/arm64: Read system registers to get the state of the MMU

When we are in EL1 we can directly tell if the local cpu's MMU is on
by reading a system register (SCTRL/SCTRL_EL1). In EL0, we use the
relevant cpumask. This way we don't have to rely on the cpu id in
thread_info when we are in setup executing in EL1.

In addition, this change:
* Removes mmu_disabled_cpu_count as it is no longer necessary and
assumed that calls to mmu_mark_enabled()/mmu_mark_disabled() were
serialized. This is currently true but a future change could
easily break that assumption.
* Changes mmu_disabled_mask to mmu_enabled_mask and inverts the
logic to track in a more intuitive way that all CPUs start with
the MMU OFF and at some point, we turn them ON.

Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
[ Applied some code cleanups suggested by Alexandru Elisei ]
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

lib/asm: Fix definitions of memory areas

Fix the definitions of the memory areas.

Bring the headers in line with the rest of the asm headers, by having the
appropriate #ifdef _ASM$ARCH_ guarding th

lib/asm: Fix definitions of memory areas

Fix the definitions of the memory areas.

Bring the headers in line with the rest of the asm headers, by having the
appropriate #ifdef _ASM$ARCH_ guarding the headers.

Fixes: d74708246bd9 ("lib/asm: Add definitions of memory areas")

Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Krish Sadhukhan <krish.sadhukhan@oracle.com>
Message-Id: <20210115123730.381612-5-imbrenda@linux.ibm.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

lib/asm: Add definitions of memory areas

Add definitions and boundaries of memory areas for some architectures.
This is needed by the next patch.

Most architectures only get one generic memory area

lib/asm: Add definitions of memory areas

Add definitions and boundaries of memory areas for some architectures.
This is needed by the next patch.

Most architectures only get one generic memory area, wherease x86 and
s390x get some more attention:

x86 gets
* low area (32-bit addresses)
* high PAE area (36-bit addresses)
* the rest

s390x gets
* low area (31-bit addresses)
* the rest

Notice that the number indicates the order in which the areas are
scanned when more than one area is indicated. The default order tries
to get allocations from higher address ranges before trying lower ones.
This tries to keep the precious lower addresses as free as possible.

Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Message-Id: <20201002154420.292134-4-imbrenda@linux.ibm.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

arm/arm64: Change dcache_line_size to ulong

dcache_line_size is treated like a long in assembly, so make it one.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Message-Id: <20201014191444.136782-

arm/arm64: Change dcache_line_size to ulong

dcache_line_size is treated like a long in assembly, so make it one.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Message-Id: <20201014191444.136782-4-drjones@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

arm: Add support for the DEVICE_nGRE and NORMAL_WT memory types

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Alexandru Elisei

arm: Add support for the DEVICE_nGRE and NORMAL_WT memory types

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm64: Add support for configuring the translation granule

Make the translation granule configurable for arm64. arm64 supports
page sizes of 4K, 16K and 64K. By default, arm64 is configured with
64K

arm64: Add support for configuring the translation granule

Make the translation granule configurable for arm64. arm64 supports
page sizes of 4K, 16K and 64K. By default, arm64 is configured with
64K pages. configure has been extended with a new argument:

--page-size=PAGE_SIZE

which allows the user to set the page shift and therefore the page
size for arm64. Using the --page-size for any other architecture
results an error message.

In addition, bump the VA size from 42 to 48 bits. To enable 48 bit
VAs for all translation granules, add support for 4-level and 3-level
page tables. At compile time, we determine how many levels in the page
tables we needed.

Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Tested-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm64: Check if the configured translation granule is supported

As arm64 implementations may support a subset of the architecturally
defined granules, we need to check and warn the user if the confi

arm64: Check if the configured translation granule is supported

As arm64 implementations may support a subset of the architecturally
defined granules, we need to check and warn the user if the configured
translation granule is not supported.

Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Tested-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

lib/arm64/spinlock: Fix inline assembly for Clang

/home/travis/build/huth/kvm-unit-tests/lib/arm64/spinlock.c:29:12: error:
value size does not match register size specified by the constraint and

lib/arm64/spinlock: Fix inline assembly for Clang

/home/travis/build/huth/kvm-unit-tests/lib/arm64/spinlock.c:29:12: error:
value size does not match register size specified by the constraint and
modifier [-Werror,-Wasm-operand-widths]
: "=&r" (val), "=&r" (fail)
^
/home/travis/build/huth/kvm-unit-tests/lib/arm64/spinlock.c:27:9: note: use
constraint modifier "w"
" mov %0, #1\n"
^~
%w0

Use the "w" modifier as suggested to fix the issue.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201001072234.143703-7-thuth@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Tested-by: Andrew Jones <drjones@redhat.com>

show more ...

arm64: its: Handle its command queue wrapping

Because micro-bench may send a large number of ITS commands, we
should handle ITS command queue wrapping as kernel instead of just
failing the test.

Si

arm64: its: Handle its command queue wrapping

Because micro-bench may send a large number of ITS commands, we
should handle ITS command queue wrapping as kernel instead of just
failing the test.

Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

arm/arm64: timer: Extract irqs at setup time

The timer can be useful for other tests besides the timer test.
Extract the DT parsing of the irqs out of the timer test into
setup and provide them alon

arm/arm64: timer: Extract irqs at setup time

The timer can be useful for other tests besides the timer test.
Extract the DT parsing of the irqs out of the timer test into
setup and provide them along with some defines in a new timer.h
file.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

arm/arm64: ITS: migration tests

This test maps LPIs (populates the device table, the collection table,
interrupt translation tables, configuration table), migrates and make
sure the translation is c

arm/arm64: ITS: migration tests

This test maps LPIs (populates the device table, the collection table,
interrupt translation tables, configuration table), migrates and make
sure the translation is correct on the destination.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
[ Complete migration even when the test is skipped. Otherwise the
migration scripts hang. ]
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: ITS: Commands

Implement main ITS commands. The code is largely inherited from
the ITS driver.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.

arm/arm64: ITS: Commands

Implement main ITS commands. The code is largely inherited from
the ITS driver.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: ITS: Device and collection Initialization

Introduce an helper functions to register
- a new device, characterized by its device id and the
max number of event IDs that dimension its ITT

arm/arm64: ITS: Device and collection Initialization

Introduce an helper functions to register
- a new device, characterized by its device id and the
max number of event IDs that dimension its ITT (Interrupt
Translation Table). The function allocates the ITT.

- a new collection, characterized by its ID and the
target processing engine (PE).

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: ITS: its_enable_defaults

its_enable_defaults() enable LPIs at redistributor level
and ITS level.

gicv3_enable_defaults must be called before.

Signed-off-by: Eric Auger <eric.auger@redha

arm/arm64: ITS: its_enable_defaults

its_enable_defaults() enable LPIs at redistributor level
and ITS level.

gicv3_enable_defaults must be called before.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: ITS: Introspection tests

Detect the presence of an ITS as part of the GICv3 init
routine, initialize its base address and read few registers
the IIDR, the TYPER to store its dimensioning

arm/arm64: ITS: Introspection tests

Detect the presence of an ITS as part of the GICv3 init
routine, initialize its base address and read few registers
the IIDR, the TYPER to store its dimensioning parameters.
Parse the BASER registers. As part of the init sequence we
also init all the requested tables.

This is our first ITS test, belonging to a new "its" group.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm: pmu: Basic event counter Tests

Adds the following tests:
- event-counter-config: test event counter configuration
- basic-event-count:
- programs counters #0 and #1 to count 2 required events

arm: pmu: Basic event counter Tests

Adds the following tests:
- event-counter-config: test event counter configuration
- basic-event-count:
- programs counters #0 and #1 to count 2 required events
(resp. CPU_CYCLES and INST_RETIRED). Counter #0 is preset
to a value close enough to the 32b
overflow limit so that we check the overflow bit is set
after the execution of the asm loop.
- mem-access: counts MEM_ACCESS event on counters #0 and #1
with and without 32-bit overflow.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm64: Provide read/write_sysreg_s

Sometimes we need to test access to system registers which are
missing assembler mnemonics.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Alexandr

arm64: Provide read/write_sysreg_s

Sometimes we need to test access to system registers which are
missing assembler mnemonics.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>

show more ...

arm/arm64: Perform dcache clean + invalidate after turning MMU off

When the MMU is off, data accesses are to Device nGnRnE memory on arm64 [1]
or to Strongly-Ordered memory on arm [2]. This means th

arm/arm64: Perform dcache clean + invalidate after turning MMU off

When the MMU is off, data accesses are to Device nGnRnE memory on arm64 [1]
or to Strongly-Ordered memory on arm [2]. This means that the accesses are
non-cacheable.

Perform a dcache clean to PoC so we can read the newer values from the
cache after we turn the MMU off, instead of the stale values from memory.

Perform an invalidation so we can access the data written to memory after
we turn the MMU back on. This prevents reading back the stale values we
cleaned from the cache when we turned the MMU off.

Data caches are PIPT and the VAs are translated using the current
translation tables, or an identity mapping (what Arm calls a "flat
mapping") when the MMU is off [1, 2]. Do the clean + invalidate when the
MMU is off so we don't depend on the current translation tables and we can
make sure that the operation applies to the entire physical memory.

The patch was tested by hacking arm/selftest.c:

+#include <alloc_page.h>
+#include <asm/mmu.h>
int main(int argc, char **argv)
{
+ int *x = alloc_page();
+
report_prefix_push("selftest");

+ *x = 0x42;
+ mmu_disable();
+ report(*x == 0x42, "read back value written with MMU on");
+ *x = 0x50;
+ mmu_enable(current_thread_info()->pgtable);
+ report(*x == 0x50, "read back value written with MMU off");
+
if (argc < 2)
report_abort("no test specified");

Without the fix, the first report fails, and the test usually hangs before
the second report. This is because mmu_enable pushes the LR register on the
stack when the MMU is off, which means that the value will be written to
memory. However, after asm_mmu_enable, the MMU is enabled, and we read it
back from the dcache, thus getting garbage.

With the fix, the two reports pass.

[1] ARM DDI 0487E.a, section D5.2.9
[2] ARM DDI 0406C.d, section B3.2.1

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: selftest: Add prefetch abort test

When a guest tries to execute code from an invalid physical address
KVM should inject an external abort. This test is based on a test
originally posted b

arm/arm64: selftest: Add prefetch abort test

When a guest tries to execute code from an invalid physical address
KVM should inject an external abort. This test is based on a test
originally posted by Alexandru Elisei. This version avoids hard
coding the invalid physical address used.

Cc: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

lib: arm/arm64: Teach mmu_clear_user about block mappings

kvm-unit-tests uses block mappings, so let's expand the mmu_clear_user
function to handle those as well.

Now that the function knows about

lib: arm/arm64: Teach mmu_clear_user about block mappings

kvm-unit-tests uses block mappings, so let's expand the mmu_clear_user
function to handle those as well.

Now that the function knows about block mappings, we cannot simply
assume that if an address isn't mapped we can map it as a regular page.
Change the semantics of the function to fail quite loudly if the address
isn't mapped, and shift the burden on the caller to map the address as a
page or block mapping before calling mmu_clear_user.

Also make mmu_clear_user more flexible by adding a pgtable parameter,
instead of assuming that the change always applies to the current
translation tables.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

lib: arm/arm64: Add missing include for alloc_page.h in pgtable.h

pgtable.h is used only by mmu.c, where it is included after alloc_page.h.

Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off

lib: arm/arm64: Add missing include for alloc_page.h in pgtable.h

pgtable.h is used only by mmu.c, where it is included after alloc_page.h.

Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

lib: arm/arm64: Use WRITE_ONCE to update the translation tables

Use WRITE_ONCE to prevent store tearing when updating an entry in the
translation tables. Without WRITE_ONCE, the compiler, even thoug

lib: arm/arm64: Use WRITE_ONCE to update the translation tables

Use WRITE_ONCE to prevent store tearing when updating an entry in the
translation tables. Without WRITE_ONCE, the compiler, even though it is
unlikely, can emit several stores when changing the table, and we might
end up with bogus TLB entries.

It's worth noting that the existing code is mostly fine without any
changes because the translation tables are updated in one of the
following situations:

- When the tables are being created with the MMU off, which means no TLB
caching is being performed.

- When new page table entries are added as a result of vmalloc'ing a
stack for a secondary CPU, which doesn't happen very often.

- When clearing the PTE_USER bit for the cache test, and store tearing
has no effect on the table walker because there are no intermediate
values between bit values 0 and 1. We still use WRITE_ONCE in this case
for consistency.

However, the functions are global and there is nothing preventing someone
from writing a test that uses them in a different scenario. Let's make
sure that when that happens, there will be no breakage once in a blue
moon.

Reported-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

lib: arm64: Add missing ISB in flush_tlb_page

Linux commit d0b7a302d58a made it abundantly clear that certain CPU
implementations require an ISB after a DSB. Add the missing ISB to
flush_tlb_page. N

lib: arm64: Add missing ISB in flush_tlb_page

Linux commit d0b7a302d58a made it abundantly clear that certain CPU
implementations require an ISB after a DSB. Add the missing ISB to
flush_tlb_page. No changes are required for flush_tlb_all, as the function
already had the ISB.

Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm: Remove redundant page zeroing

Now that alloc_page() zeros the page, remove the redundant page zeroing.

Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Nadav Amit <nadav.amit@gma

arm: Remove redundant page zeroing

Now that alloc_page() zeros the page, remove the redundant page zeroing.

Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Nadav Amit <nadav.amit@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...