lib: Use __ASSEMBLER__ instead of __ASSEMBLY__

Convert all non-x86 #ifdefs from __ASSEMBLY__ to __ASSEMBLER__, and remove
all manual __ASSEMBLY__ #defines. __ASSEMBLY_ was inherited blindly from
th

lib: Use __ASSEMBLER__ instead of __ASSEMBLY__

Convert all non-x86 #ifdefs from __ASSEMBLY__ to __ASSEMBLER__, and remove
all manual __ASSEMBLY__ #defines. __ASSEMBLY_ was inherited blindly from
the Linux kernel, and must be manually defined, e.g. through build rules
or with the aforementioned explicit #defines in assembly code.

__ASSEMBLER__ on the other hand is automatically defined by the compiler
when preprocessing assembly, i.e. doesn't require manually #defines for
the code to function correctly.

Ignore x86, as x86 doesn't actually rely on __ASSEMBLY__ at the moment,
and is undergoing a parallel cleanup.

Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Andrew Jones <andrew.jones@linux.dev>
Message-ID: <20250222014526.2302653-1-seanjc@google.com>
[thuth: Fix three more occurances in libfdt.h and sbi-tests.h]
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

arm64: Expand SMCCC arguments and return values

PSCI uses the SMC Calling Convention (SMCCC) to communicate with the higher
level software. PSCI uses at most 4 arguments and expend only one return
v

arm64: Expand SMCCC arguments and return values

PSCI uses the SMC Calling Convention (SMCCC) to communicate with the higher
level software. PSCI uses at most 4 arguments and expend only one return
value. However, SMCCC has provisions for more arguments (upto 17 depending
on the SMCCC version) and upto 10 distinct return values.

We are going to be adding tests that make use of it, so add support for the
extended number of arguments and return values.

Also rename the SMCCC functions to generic, non-PSCI names, so they
can be used for Realm services.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Co-developed-by: Joey Gouly <joey.gouly@arm.com>
Signed-off-by: Joey Gouly <joey.gouly@arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
[Fixed EFI compile error.]
Signed-off-by: Andrew Jones <andrew.jones@linux.dev>

show more ...

arm: Add necessary header files in asm/pgtable.h

We use memalign() and other symbols defined elsewhere without explicitly including
them, which could cause build failures. Add the necessary header f

arm: Add necessary header files in asm/pgtable.h

We use memalign() and other symbols defined elsewhere without explicitly including
them, which could cause build failures. Add the necessary header files.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Andrew Jones <andrew.jones@linux.dev>

show more ...

arm/arm64: Share memregions

arm/arm64 (and to a small extent powerpc) have memory regions which
get built from hardware descriptions (DT/ACPI/EFI) and then used to
build page tables. Move memregions

arm/arm64: Share memregions

arm/arm64 (and to a small extent powerpc) have memory regions which
get built from hardware descriptions (DT/ACPI/EFI) and then used to
build page tables. Move memregions to common code, tweaking the API
a bit at the same time, e.g. change 'mem_region' to 'memregions'.
The biggest change is there is now a default number of memory regions
which, if too small, should be overridden at setup time with a new
init function, memregions_init().

Signed-off-by: Andrew Jones <andrew.jones@linux.dev>
Acked-by: Thomas Huth <thuth@redhat.com>

show more ...

arm/arm64: Share on_cpus

Now that the previous patches have cleaned up Arm's on_cpus
implementation we can move it to the common lib where riscv
will share it.

Signed-off-by: Andrew Jones <andrew.j

arm/arm64: Share on_cpus

Now that the previous patches have cleaned up Arm's on_cpus
implementation we can move it to the common lib where riscv
will share it.

Signed-off-by: Andrew Jones <andrew.jones@linux.dev>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

show more ...

arm/arm64: Remove spinlocks from on_cpu_async

Remove spinlocks from on_cpu_async() by pulling some of their
use into a new function and also by narrowing the locking to a
single on_cpu_info structur

arm/arm64: Remove spinlocks from on_cpu_async

Remove spinlocks from on_cpu_async() by pulling some of their
use into a new function and also by narrowing the locking to a
single on_cpu_info structure by introducing yet another cpumask.

Signed-off-by: Andrew Jones <andrew.jones@linux.dev>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

show more ...

arm/arm64: Generalize wfe/sev names in smp.c

Most of Arm's on_cpus() implementation can be shared by any
architecture which has the possible, present, and idle cpumasks,
like riscv does. Rename the

arm/arm64: Generalize wfe/sev names in smp.c

Most of Arm's on_cpus() implementation can be shared by any
architecture which has the possible, present, and idle cpumasks,
like riscv does. Rename the exceptions (wfe/sve) to something
more generic in order to prepare to share the functions.

Signed-off-by: Andrew Jones <andrew.jones@linux.dev>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

show more ...

arm/arm64: Share cpu online, present and idle masks

RISC-V will also use Arm's three cpumasks. These were in smp.h,
but they can be in cpumask.h instead, so move them there, which
is now shared.

Si

arm/arm64: Share cpu online, present and idle masks

RISC-V will also use Arm's three cpumasks. These were in smp.h,
but they can be in cpumask.h instead, so move them there, which
is now shared.

Signed-off-by: Andrew Jones <andrew.jones@linux.dev>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

show more ...

arm/arm64: Move cpumask.h to common lib

RISC-V will also make use of cpumask.h, so move it to the arch-common
directory.

Signed-off-by: Andrew Jones <andrew.jones@linux.dev>
Reviewed-by: Thomas Hut

arm/arm64: Move cpumask.h to common lib

RISC-V will also make use of cpumask.h, so move it to the arch-common
directory.

Signed-off-by: Andrew Jones <andrew.jones@linux.dev>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

show more ...

arm64: Add a setup sequence for systems that boot through EFI

This change implements an alternative setup sequence for the system
when we are booting through EFI. The memory map is discovered throug

arm64: Add a setup sequence for systems that boot through EFI

This change implements an alternative setup sequence for the system
when we are booting through EFI. The memory map is discovered through
EFI boot services and devices through ACPI.

This change is based on a change initially proposed by
Andrew Jones <drjones@redhat.com>

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Shaoqin Huang <shahuang@redhat.com>
[Changed __ALIGN to ALIGN as pointed out by Nadav Amit.]
Signed-off-by: Andrew Jones <andrew.jones@linux.dev>

show more ...

arm64: Add a new type of memory type flag MR_F_RESERVED

This will be used by future change to add PTE entries for special EFI
memory regions.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com

arm64: Add a new type of memory type flag MR_F_RESERVED

This will be used by future change to add PTE entries for special EFI
memory regions.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
Signed-off-by: Andrew Jones <andrew.jones@linux.dev>

show more ...

arm64: Add support for timer initialization through ACPI

For systems with ACPI support, we can discover timers through the ACPI
GTDT table. This change implements the code to discover timers through

arm64: Add support for timer initialization through ACPI

For systems with ACPI support, we can discover timers through the ACPI
GTDT table. This change implements the code to discover timers through
the GTDT and adds ACPI support in timer_save_state. This change
retains the default behavior; we check if a valid DT is provided, if
not, we try to discover timers using ACPI.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Andrew Jones <andrew.jones@linux.dev>

show more ...

arm/arm64: mmu: Rename mmu_get_pte() -> follow_pte()

The function get_pte() from mmu.c returns a pointer to the PTE
associated with the requested virtual address, mapping the virtual
address in the

arm/arm64: mmu: Rename mmu_get_pte() -> follow_pte()

The function get_pte() from mmu.c returns a pointer to the PTE
associated with the requested virtual address, mapping the virtual
address in the process if it's not already mapped.

mmu_get_pte() returns a pointer to the PTE if and only if the virtual is
mapped in pgtable, otherwise returns NULL. Rename it to follow_pte() to
avoid any confusion with get_pte(). follow_pte() also matches the name
of Linux kernel function with a similar purpose.

Also remove the mmu_enabled() check from the function, as the purpose of
the function is to get the mapping for the virtual address in the pgtable
supplied as the argument, not to translate the virtual address to a
physical address using the current translation; that's what
virt_to_phys() does.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <andrew.jones@linux.dev>

show more ...

arm/psci: Test that CPU 1 has been successfully brought online

For the PSCI CPU_ON function test, all other CPUs perform a CPU_ON call
that target CPU 1. The test is considered a success if CPU_ON r

arm/psci: Test that CPU 1 has been successfully brought online

For the PSCI CPU_ON function test, all other CPUs perform a CPU_ON call
that target CPU 1. The test is considered a success if CPU_ON returns PSCI
SUCCESS exactly once, and for the rest of the calls PSCI ALREADY_ON.

Enhance the test by checking that CPU 1 is actually online and able to
execute code. Also make the test more robust by checking that the CPU_ON
call returns, instead of assuming that it will always succeed and
hanging indefinitely if it doesn't.

Since the CPU 1 thread is now being set up properly by kvm-unit-tests
when being brought online, it becomes possible to add other tests in the
future that require all CPUs.

The include header order in arm/psci.c has been changed to be in
alphabetic order. This means moving the errata.h include before
libcflat.h, which causes compilation to fail because of missing includes
in errata.h. Fix that also by including the needed header in errata.h.

Reviewed-by: Andrew Jones <andrew.jones@linux.dev>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <andrew.jones@linux.dev>

show more ...

header guards: clean up some stragglers

Some headers had been missed during the initial header guard
standardization.

Signed-off-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Laurent Vivier <l

header guards: clean up some stragglers

Some headers had been missed during the initial header guard
standardization.

Signed-off-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210610135937.94375-2-cohuck@redhat.com>

show more ...

arm: unify header guards

The assembler.h files were the only ones not already following
the convention.

Signed-off-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: David Hildenbrand <david@redhat

arm: unify header guards

The assembler.h files were the only ones not already following
the convention.

Signed-off-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Message-Id: <20210609143712.60933-5-cohuck@redhat.com>

show more ...

arm/arm64: psci: Don't assume method is hvc

The method can be smc in addition to hvc, and it will be when running
on bare metal. Additionally, we move the invocations to assembly so
we don't have to

arm/arm64: psci: Don't assume method is hvc

The method can be smc in addition to hvc, and it will be when running
on bare metal. Additionally, we move the invocations to assembly so
we don't have to rely on compiler assumptions. We also fix the
prototype of psci_invoke. function_id should be an unsigned int, not
an unsigned long.

Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Tested-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: setup: Consolidate memory layout assumptions

Keep as much memory layout assumptions as possible in init::start
and a single setup function. This prepares us for calling setup()
from diffe

arm/arm64: setup: Consolidate memory layout assumptions

Keep as much memory layout assumptions as possible in init::start
and a single setup function. This prepares us for calling setup()
from different start functions which have been linked with different
linker scripts. To do this, stacktop is only referenced from
init::start, making freemem_start a parameter to setup(). We also
split mem_init() into three parts, one that populates the mem regions
per the DT, one that populates the mem regions per assumptions,
and one that does the mem init. The concept of a primary region
is dropped, but we add a sanity check for the absence of memory
holes, because we don't know how to deal with them yet.

Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Tested-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: mmu: Remove memory layout assumptions

Rather than making too many assumptions about the memory layout
in mmu code, just set up the page tables per the memory regions
(which means putting

arm/arm64: mmu: Remove memory layout assumptions

Rather than making too many assumptions about the memory layout
in mmu code, just set up the page tables per the memory regions
(which means putting all the memory layout assumptions in setup).
To ensure we get the right default flags set we need to split the
primary region into two regions for code and data.

We still only expect the primary regions to be present, but the
next patch will remove that assumption too.

(Unfortunately we still have an assumption in setup_mmu. We assume
the range 3G-4G is available for the virtual memory allocator. We'll
need to remove that assumption as well with another patch in order
to support arbitrary memory maps.)

Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Tested-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: mmu: Stop mapping an assumed IO region

By providing a proper ioremap function, we can just rely on devices
calling it for each region they need (as they already do) instead of
mapping a b

arm/arm64: mmu: Stop mapping an assumed IO region

By providing a proper ioremap function, we can just rely on devices
calling it for each region they need (as they already do) instead of
mapping a big assumed I/O range. We don't require the MMU to be
enabled at the time of the ioremap. In that case, we add the mapping
to the identity map anyway. This allows us to call setup_vm after
io_init. Why don't we just call setup_vm before io_init, I hear you
ask? Well, that's because tests like sieve want to start with the MMU
off, later call setup_vm, and all the while have working I/O. Some
unit tests are just really demanding...

While at it, ensure we map the I/O regions with XN (execute never),
as suggested by Alexandru Elisei.

Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Tested-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: Remove dcache_line_size global variable

Compute the dcache line size when doing dcache maintenance instead of using
a global variable computed in setup(), which allows us to do dcache
mai

arm/arm64: Remove dcache_line_size global variable

Compute the dcache line size when doing dcache maintenance instead of using
a global variable computed in setup(), which allows us to do dcache
maintenance at any point in the boot process. This will be useful for
running as an EFI app and it also aligns our implementation to that of the
Linux kernel. As a result, the dcache_by_line_op assembly has been modified
to take a range described by start address and size, instead of start and
end addresses.

For consistency, the arm code has been similary modified.

Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: Add sanity checks to the cpumask API

Make sure that any calls to the cpumask API target a valid CPU. The
CPU is identified by an int in the range [0, nr_cpus), where nr_cpus
is set based

arm/arm64: Add sanity checks to the cpumask API

Make sure that any calls to the cpumask API target a valid CPU. The
CPU is identified by an int in the range [0, nr_cpus), where nr_cpus
is set based on information in the DT.

Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: Read system registers to get the state of the MMU

When we are in EL1 we can directly tell if the local cpu's MMU is on
by reading a system register (SCTRL/SCTRL_EL1). In EL0, we use the
r

arm/arm64: Read system registers to get the state of the MMU

When we are in EL1 we can directly tell if the local cpu's MMU is on
by reading a system register (SCTRL/SCTRL_EL1). In EL0, we use the
relevant cpumask. This way we don't have to rely on the cpu id in
thread_info when we are in setup executing in EL1.

In addition, this change:
* Removes mmu_disabled_cpu_count as it is no longer necessary and
assumed that calls to mmu_mark_enabled()/mmu_mark_disabled() were
serialized. This is currently true but a future change could
easily break that assumption.
* Changes mmu_disabled_mask to mmu_enabled_mask and inverts the
logic to track in a more intuitive way that all CPUs start with
the MMU OFF and at some point, we turn them ON.

Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
[ Applied some code cleanups suggested by Alexandru Elisei ]
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

arm/arm64: Avoid calling cpumask_test_cpu for CPUs above nr_cpus

Prior to this change, a call of cpumask_next(cpu, mask) where
cpu=nr_cpus - 1 (assuming all cpus are enumerated in the range
0..nr_cp

arm/arm64: Avoid calling cpumask_test_cpu for CPUs above nr_cpus

Prior to this change, a call of cpumask_next(cpu, mask) where
cpu=nr_cpus - 1 (assuming all cpus are enumerated in the range
0..nr_cpus - 1) would make an out-of-bounds access to the mask.
In many cases, this is still a valid memory location due the
implementation of cpumask_t, however, in certain configurations
(for example, nr_cpus == sizeof(long)) this would cause an access
outside the bounds of the mask too.

This patch changes the way we guard calls to cpumask_test_cpu() in
cpumask_next() to avoid the above condition. A following change adds
assertions to catch out-of-bounds accesses to cpumask_t.

Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>

show more ...

lib/asm: Fix definitions of memory areas

Fix the definitions of the memory areas.

Bring the headers in line with the rest of the asm headers, by having the
appropriate #ifdef _ASM$ARCH_ guarding th

lib/asm: Fix definitions of memory areas

Fix the definitions of the memory areas.

Bring the headers in line with the rest of the asm headers, by having the
appropriate #ifdef _ASM$ARCH_ guarding the headers.

Fixes: d74708246bd9 ("lib/asm: Add definitions of memory areas")

Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Krish Sadhukhan <krish.sadhukhan@oracle.com>
Message-Id: <20210115123730.381612-5-imbrenda@linux.ibm.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...