Changes in aes-modes.S

11d6bc70fff310cf0c4bbfa740144b0e350cd706 - lib/crypto: arm64/aes: Remove obsolete chunking logic

Wed, 01 Apr 2026 00:05:40 +0000

lib/crypto: arm64/aes: Remove obsolete chunking logicSince commit aefbab8e77eb ("arm64: fpsimd: Preserve/restore kernel modeNEON at context switch"), kernel-mode NEON sections have beenpreemptible on arm64. And since commit 7dadeaa6e851 ("sched: Furtherrestrict the preemption modes"), voluntary preemption is no longersupported on arm64 either. Therefore, there's no longer any need tolimit the length of kernel-mode NEON sections on arm64.Simplify the AES-CBC-MAC code accordingly.Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260401000548.133151-2-ebiggers@kernel.orgSigned-off-by: Eric Biggers List of files: /linux/lib/crypto/arm64/aes-modes.S

58286738b159ca93d41438a6ddcc2ea5333191b4 - lib/crypto: arm64/aes: Migrate optimized CBC-based MACs into library

Wed, 18 Feb 2026 21:34:51 +0000

lib/crypto: arm64/aes: Migrate optimized CBC-based MACs into libraryInstead of exposing the arm64-optimized CMAC, XCBC-MAC, and CBC-MAC codevia arm64-specific crypto_shash algorithms, instead just implement theaes_cbcmac_blocks_arch() library function. This is much simpler, itmakes the corresponding library functions be arm64-optimized, and itfixes the longstanding issue where this optimized code was disabled bydefault. The corresponding algorithms still remain available throughcrypto_shash, but individual architectures no longer need to handle it.Note that to be compatible with the library using 'size_t' lengths, thetype of the return value and 'blocks' parameter to the assemblyfunctions had to be changed to 'size_t', and the assembly code had to beupdated accordingly to use the corresponding 64-bit registers.Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260218213501.136844-6-ebiggers@kernel.orgSigned-off-by: Eric Biggers List of files: /linux/lib/crypto/arm64/aes-modes.S

4b908403209252e59ecad4c068bf967fa3f07525 - lib/crypto: arm64/aes: Move assembly code for AES modes into libaes

Wed, 18 Feb 2026 21:34:50 +0000

lib/crypto: arm64/aes: Move assembly code for AES modes into libaesTo migrate the support for CBC-based MACs into libaes, the correspondingarm64 assembly code needs to be moved there. However, the arm64 AESassembly code groups many AES modes together; individual modes aren'teasily separable. (This isn't unique to arm64; other architecturesorganize their AES modes similarly.)Since the other AES modes will be migrated into the library eventuallytoo, just move the full assembly files for the AES modes into thelibrary. (This is similar to what I already did for PowerPC and SPARC.)Specifically: move the assembly files aes-ce.S, aes-modes.S, andaes-neon.S and their build rules; declare the assembly functions in; and export the assembly functions from libaes.Note that the exports and public declarations of the assembly functionsare temporary. They exist only to keep arch/arm64/crypto/ working untilthe AES modes are fully moved into the library.Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260218213501.136844-5-ebiggers@kernel.orgSigned-off-by: Eric Biggers List of files: /linux/lib/crypto/arm64/aes-modes.S

f8f08d7cc43237e91e3aedf7b67d015d24c38fcc - crypto: arm64/aes - Fix 32-bit aes_mac_update() arg treated as 64-bit

Wed, 18 Feb 2026 21:34:49 +0000

crypto: arm64/aes - Fix 32-bit aes_mac_update() arg treated as 64-bitSince the 'enc_after' argument to neon_aes_mac_update() andce_aes_mac_update() has type 'int', it needs to be accessed using thecorresponding 32-bit register, not the 64-bit register. The upper halfof the corresponding 64-bit register may contain garbage.Fixes: 4860620da7e5 ("crypto: arm64/aes - add NEON/Crypto Extensions CBCMAC/CMAC/XCBC driver")Cc: stable@vger.kernel.orgReviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260218213501.136844-4-ebiggers@kernel.orgSigned-off-by: Eric Biggers List of files: /linux/lib/crypto/arm64/aes-modes.S

7d709af18054bc9e2043499bb35eb1809c2a316f - crypto: arm64/aes-modes - use frame_push/pop macros consistently

Tue, 29 Nov 2022 16:48:50 +0000

crypto: arm64/aes-modes - use frame_push/pop macros consistentlyUse the frame_push and frame_pop macros to create the stack frames inthe AES chaining mode wrappers so that they will get PAC and/or shadowcall stack protection when configured.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

c0eb7591c1ed9cbdb0ad796bb56aed13748b55fa - crypto: arm64/aes-xctr - Improve readability of XCTR and CTR modes

Fri, 20 May 2022 18:14:58 +0000

crypto: arm64/aes-xctr - Improve readability of XCTR and CTR modesAdded some clarifying comments, changed the register allocations to makethe code clearer, and added register aliases.Signed-off-by: Nathan Huckleberry Reviewed-by: Eric Biggers Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

23a251cc1696e1bf68df1dbba569d2fe12469d22 - crypto: arm64/aes-xctr - Add accelerated implementation of XCTR

Fri, 20 May 2022 18:14:57 +0000

crypto: arm64/aes-xctr - Add accelerated implementation of XCTRAdd hardware accelerated version of XCTR for ARM64 CPUs with ARMv8Crypto Extension support. This XCTR implementation is based on the CTRimplementation in aes-modes.S.More information on XCTR can be found inthe HCTR2 paper: "Length-preserving encryption with HCTR2":https://eprint.iacr.org/2021/1441.pdfSigned-off-by: Nathan Huckleberry Reviewed-by: Ard Biesheuvel Reviewed-by: Eric Biggers Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

8daa399edeed4cfa792ccea12beda50d445ab6a0 - crypto: arm64/aes-neon-ctr - improve handling of single tail block

Thu, 27 Jan 2022 09:52:11 +0000

crypto: arm64/aes-neon-ctr - improve handling of single tail blockInstead of falling back to C code to do a memcpy of the output of thelast block, handle this in the asm code directly if possible, which isthe case if the entire input is longer than 16 bytes.Cc: Nathan Huckleberry Cc: Eric Biggers Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

9be34be87cc8d1afe3c3bc2e645b4dee512d9eda - arm64: Add macro version of the BTI instruction

Tue, 14 Dec 2021 15:27:12 +0000

arm64: Add macro version of the BTI instructionBTI is only available from v8.5 so we need to encode it using HINT ingeneric code and for older toolchains. Add an assembler macro based onone written by Mark Rutland which lets us use the mnemonic and updatethe existing users.Suggested-by: Mark Rutland Acked-by: Ard Biesheuvel Acked-by: Will Deacon Signed-off-by: Mark Brown Acked-by: Mark Rutland Link: https://lore.kernel.org/r/20211214152714.2380849-2-broonie@kernel.orgSigned-off-by: Catalin Marinas List of files: /linux/lib/crypto/arm64/aes-modes.S

31a24ae89c92d5533c049046a76c6a2d649efb72 - Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux

Mon, 26 Apr 2021 17:25:03 +0000

Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linuxPull arm64 updates from Catalin Marinas: - MTE asynchronous support for KASan. Previously only synchronous (slower) mode was supported. Asynchronous is faster but does not allow precise identification of the illegal access. - Run kernel mode SIMD with softirqs disabled. This allows using NEON in softirq context for crypto performance improvements. The conditional yield support is modified to take softirqs into account and reduce the latency. - Preparatory patches for Apple M1: handle CPUs that only have the VHE mode available (host kernel running at EL2), add FIQ support. - arm64 perf updates: support for HiSilicon PA and SLLC PMU drivers, new functions for the HiSilicon HHA and L3C PMU, cleanups. - Re-introduce support for execute-only user permissions but only when the EPAN (Enhanced Privileged Access Never) architecture feature is available. - Disable fine-grained traps at boot and improve the documented boot requirements. - Support CONFIG_KASAN_VMALLOC on arm64 (only with KASAN_GENERIC). - Add hierarchical eXecute Never permissions for all page tables. - Add arm64 prctl(PR_PAC_{SET,GET}_ENABLED_KEYS) allowing user programs to control which PAC keys are enabled in a particular task. - arm64 kselftests for BTI and some improvements to the MTE tests. - Minor improvements to the compat vdso and sigpage. - Miscellaneous cleanups.* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (86 commits) arm64/sve: Add compile time checks for SVE hooks in generic functions arm64/kernel/probes: Use BUG_ON instead of if condition followed by BUG. arm64: pac: Optimize kernel entry/exit key installation code paths arm64: Introduce prctl(PR_PAC_{SET,GET}_ENABLED_KEYS) arm64: mte: make the per-task SCTLR_EL1 field usable elsewhere arm64/sve: Remove redundant system_supports_sve() tests arm64: fpsimd: run kernel mode NEON with softirqs disabled arm64: assembler: introduce wxN aliases for wN registers arm64: assembler: remove conditional NEON yield macros kasan, arm64: tests supports for HW_TAGS async mode arm64: mte: Report async tag faults before suspend arm64: mte: Enable async tag check fault arm64: mte: Conditionally compile mte_enable_kernel_*() arm64: mte: Enable TCO in functions that can read beyond buffer limits kasan: Add report for async mode arm64: mte: Drop arch_enable_tagging() kasan: Add KASAN mode kernel parameter arm64: mte: Add asynchronous mode support arm64: Get rid of CONFIG_ARM64_VHE arm64: Cope with CPUs stuck in VHE mode ... List of files: /linux/lib/crypto/arm64/aes-modes.S

0f19dbc994dcb7f7137f2e056e813c84530b7538 - crypto: arm64/aes-ce - deal with oversight in new CTR carry code

Tue, 06 Apr 2021 14:25:23 +0000

crypto: arm64/aes-ce - deal with oversight in new CTR carry codeThe new carry handling code in the CTR driver can deal with a carryoccurring in the 4x/5x parallel code path, by using a computed goto tojump into the carry sequence at the right place as to only apply thecarry to a subset of the blocks being processed.If the lower half of the counter wraps and ends up at exactly 0x0, acarry needs to be applied to the counter, but not to the counter valuestaken for the 4x/5x parallel sequence. In this case, the computed gotoskips all register assignments, and branches straight to the jumpinstruction that gets us back to the fast path. This produces thecorrect result, but due to the fact that this branch target does notcarry the correct BTI annotation, this fails when BTI is enabled.Let's omit the computed goto entirely in this case, and jump straightback to the fast path after applying the carry to the main counter.Fixes: 5318d3db465d ("crypto: arm64/aes-ctr - improve tail handling")Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

13150149aa6ded1e6bbe0025beac6e12604dd87c - arm64: fpsimd: run kernel mode NEON with softirqs disabled

Tue, 02 Mar 2021 09:01:12 +0000

arm64: fpsimd: run kernel mode NEON with softirqs disabledKernel mode NEON can be used in task or softirq context, but only ina non-nesting manner, i.e., softirq context is only permitted if theinterrupt was not taken at a point where the kernel was using the NEONin task context.This means all users of kernel mode NEON have to be aware of thislimitation, and either need to provide scalar fallbacks that may be muchslower (up to 20x for AES instructions) and potentially less safe, oruse an asynchronous interface that defers processing to a later timewhen the NEON is guaranteed to be available.Given that grabbing and releasing the NEON is cheap, we can relax thisrestriction, by increasing the granularity of kernel mode NEON code, andalways disabling softirq processing while the NEON is being used in taskcontext.Signed-off-by: Ard Biesheuvel Acked-by: Will Deacon Link: https://lore.kernel.org/r/20210302090118.30666-4-ardb@kernel.orgSigned-off-by: Catalin Marinas List of files: /linux/lib/crypto/arm64/aes-modes.S

f0070f4a7934e4deba83fdde70c79d9798b2366b - crypto: arm64/aes-ce-mac - simplify NEON yield

Wed, 03 Feb 2021 11:36:24 +0000

crypto: arm64/aes-ce-mac - simplify NEON yieldSigned-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

5318d3db465d29efe97b0e18da29ad95156e6142 - crypto: arm64/aes-ctr - improve tail handling

Thu, 17 Dec 2020 18:55:16 +0000

crypto: arm64/aes-ctr - improve tail handlingCounter mode is a stream cipher chaining mode that is typically usedwith inputs that are of arbitrarily length, and so a tail block whichis smaller than a full AES block is rule rather than exception.The current ctr(aes) implementation for arm64 always makes a separatecall into the assembler routine to process this tail block, which issuboptimal, given that it requires reloading of the AES round keys,and prevents us from handling this tail block using the 5-way stridethat we use for better performance on deep pipelines.So let's update the assembler routine so it can handle any input size,and uses NEON permutation instructions and overlapping loads and storesto handle the tail block. This results in a ~16% speedup for 1420 byteblocks on cores with deep pipelines such as ThunderX2.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

b8e505484e376322cb1e12540e8b52dc31b73b6e - arm64: crypto: Modernize names for AES function macros

Tue, 18 Feb 2020 19:58:26 +0000

arm64: crypto: Modernize names for AES function macrosNow that the rest of the code has been converted to the modern START/ENDmacros the AES_ENTRY() and AES_ENDPROC() macros look out of place andlike they need updating. Rename them to AES_FUNC_START() and AES_FUNC_END()to line up with the modern style assembly macros.Signed-off-by: Mark Brown Reviewed-by: Ard Biesheuvel Signed-off-by: Catalin Marinas List of files: /linux/lib/crypto/arm64/aes-modes.S

0e89640b640d7f726bcbf6903c78257a28e56f3c - crypto: arm64 - Use modern annotations for assembly functions

Fri, 13 Dec 2019 15:49:10 +0000

crypto: arm64 - Use modern annotations for assembly functionsIn an effort to clarify and simplify the annotation of assembly functionsin the kernel new macros have been introduced. These replace ENTRY andENDPROC and also add a new annotation for static functions which previouslyhad no ENTRY equivalent. Update the annotations in the crypto code to thenew macros.There are a small number of files imported from OpenSSL where the assemblyis generated using perl programs, these are not currently annotated at alland have not been modified.Signed-off-by: Mark Brown Acked-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

67cfa5d3b7214ce944747908f9a1a3cba8b989b9 - crypto: arm64/aes-neonbs - implement ciphertext stealing for XTS

Tue, 03 Sep 2019 16:43:34 +0000

crypto: arm64/aes-neonbs - implement ciphertext stealing for XTSUpdate the AES-XTS implementation based on NEON instructions so that itcan deal with inputs whose size is not a multiple of the cipher blocksize. This is part of the original XTS specification, but was neverimplemented before in the Linux kernel.Since the bit slicing driver is only faster if it can operate on atleast 7 blocks of input at the same time, let's reuse the alternatepath we are adding for CTS to process any data tail whose size isnot a multiple of 128 bytes.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

7cceca8b25b39486c647ceaf6a29f833af16f7c0 - crypto: arm64/aes - implement support for XTS ciphertext stealing

Tue, 03 Sep 2019 16:43:33 +0000

crypto: arm64/aes - implement support for XTS ciphertext stealingAdd the missing support for ciphertext stealing in the implementationof AES-XTS, which is part of the XTS specification but was omitted upuntil now due to lack of a need for it.The asm helpers are updated so they can deal with any input size, aslong as the last full block and the final partial block are presentedat the same time. The glue code is updated so that the common case ofoperating on a sector or page is mostly as before. When CTS is needed,the walk is split up into two pieces, unless the entire input is coveredby a single step.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

0cfd507c83469b07f037100a6389b02ce99b546f - crypto: arm64/aes-cts-cbc-ce - performance tweak

Tue, 03 Sep 2019 16:43:31 +0000

crypto: arm64/aes-cts-cbc-ce - performance tweakOptimize away one of the tbl instructions in the decryption path,which turns out to be unnecessary.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S

735177ca148af5049e9e98b38eaba0c769182655 - crypto: arm64/aes - implement accelerated ESSIV/CBC mode

Mon, 19 Aug 2019 14:17:36 +0000

crypto: arm64/aes - implement accelerated ESSIV/CBC modeAdd an accelerated version of the 'essiv(cbc(aes),sha256)' skcipher,which is used by fscrypt or dm-crypt on systems where CBC mode issignficantly more performant than XTS mode (e.g., when using a h/waccelerator which supports the former but not the latter) This avoidsa separate call into the AES cipher for every invocation.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-modes.S