Changes in aes-ce.S

4b908403209252e59ecad4c068bf967fa3f07525 - lib/crypto: arm64/aes: Move assembly code for AES modes into libaes

Wed, 18 Feb 2026 21:34:50 +0000

lib/crypto: arm64/aes: Move assembly code for AES modes into libaesTo migrate the support for CBC-based MACs into libaes, the correspondingarm64 assembly code needs to be moved there. However, the arm64 AESassembly code groups many AES modes together; individual modes aren'teasily separable. (This isn't unique to arm64; other architecturesorganize their AES modes similarly.)Since the other AES modes will be migrated into the library eventuallytoo, just move the full assembly files for the AES modes into thelibrary. (This is similar to what I already did for PowerPC and SPARC.)Specifically: move the assembly files aes-ce.S, aes-modes.S, andaes-neon.S and their build rules; declare the assembly functions in; and export the assembly functions from libaes.Note that the exports and public declarations of the assembly functionsare temporary. They exist only to keep arch/arm64/crypto/ working untilthe AES modes are fully moved into the library.Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260218213501.136844-5-ebiggers@kernel.orgSigned-off-by: Eric Biggers List of files: /linux/lib/crypto/arm64/aes-ce.S

571e557cbaf748124aaf0f0ac26772d7380e78fc - crypto: arm64/aes-ce - Simplify round key load sequence

Mon, 15 Apr 2024 13:04:26 +0000

crypto: arm64/aes-ce - Simplify round key load sequenceTweak the round key logic so that they can be loaded using a singlebranchless sequence using overlapping loads. This is shorter andsimpler, and puts the conditional branches based on the key size furtherapart, which might benefit microarchitectures that cannot record takenbranches at every instruction. For these branches, use test-bit-branchinstructions that don't clobber the condition flags.Note that none of this has any impact on performance, positive orotherwise (and the branch prediction benefit would only benefit AES-192which nobody uses). It does make for nicer code, though.While at it, use \@ to generate the labels inside the macros, which ismore robust than using fixed numbers, which could clash inadvertently.Also, bring aes-neon.S in line with these changes, including the switchto test-and-branch instructions, to avoid surprises in the future whenwe might start relying on the condition flags being preserved in thechaining mode wrappers in aes-modes.SSigned-off-by: Ard Biesheuvel Reviewed-by: Eric Biggers Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

b8e505484e376322cb1e12540e8b52dc31b73b6e - arm64: crypto: Modernize names for AES function macros

Tue, 18 Feb 2020 19:58:26 +0000

arm64: crypto: Modernize names for AES function macrosNow that the rest of the code has been converted to the modern START/ENDmacros the AES_ENTRY() and AES_ENDPROC() macros look out of place andlike they need updating. Rename them to AES_FUNC_START() and AES_FUNC_END()to line up with the modern style assembly macros.Signed-off-by: Mark Brown Reviewed-by: Ard Biesheuvel Signed-off-by: Catalin Marinas List of files: /linux/lib/crypto/arm64/aes-ce.S

0e89640b640d7f726bcbf6903c78257a28e56f3c - crypto: arm64 - Use modern annotations for assembly functions

Fri, 13 Dec 2019 15:49:10 +0000

crypto: arm64 - Use modern annotations for assembly functionsIn an effort to clarify and simplify the annotation of assembly functionsin the kernel new macros have been introduced. These replace ENTRY andENDPROC and also add a new annotation for static functions which previouslyhad no ENTRY equivalent. Update the annotations in the crypto code to thenew macros.There are a small number of files imported from OpenSSL where the assemblyis generated using perl programs, these are not currently annotated at alland have not been modified.Signed-off-by: Mark Brown Acked-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

67cfa5d3b7214ce944747908f9a1a3cba8b989b9 - crypto: arm64/aes-neonbs - implement ciphertext stealing for XTS

Tue, 03 Sep 2019 16:43:34 +0000

crypto: arm64/aes-neonbs - implement ciphertext stealing for XTSUpdate the AES-XTS implementation based on NEON instructions so that itcan deal with inputs whose size is not a multiple of the cipher blocksize. This is part of the original XTS specification, but was neverimplemented before in the Linux kernel.Since the bit slicing driver is only faster if it can operate on atleast 7 blocks of input at the same time, let's reuse the alternatepath we are adding for CTS to process any data tail whose size isnot a multiple of 128 bytes.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

4d2fa8b44b891f0da5ceda3e5a1402ccf0ab6f26 - Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

Tue, 09 Jul 2019 03:57:08 +0000

Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6Pull crypto updates from Herbert Xu: "Here is the crypto update for 5.3: API: - Test shash interface directly in testmgr - cra_driver_name is now mandatory Algorithms: - Replace arc4 crypto_cipher with library helper - Implement 5 way interleave for ECB, CBC and CTR on arm64 - Add xxhash - Add continuous self-test on noise source to drbg - Update jitter RNG Drivers: - Add support for SHA204A random number generator - Add support for 7211 in iproc-rng200 - Fix fuzz test failures in inside-secure - Fix fuzz test failures in talitos - Fix fuzz test failures in qat"* 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (143 commits) crypto: stm32/hash - remove interruptible condition for dma crypto: stm32/hash - Fix hmac issue more than 256 bytes crypto: stm32/crc32 - rename driver file crypto: amcc - remove memset after dma_alloc_coherent crypto: ccp - Switch to SPDX license identifiers crypto: ccp - Validate the the error value used to index error messages crypto: doc - Fix formatting of new crypto engine content crypto: doc - Add parameter documentation crypto: arm64/aes-ce - implement 5 way interleave for ECB, CBC and CTR crypto: arm64/aes-ce - add 5 way interleave routines crypto: talitos - drop icv_ool crypto: talitos - fix hash on SEC1. crypto: talitos - move struct talitos_edesc into talitos.h lib/scatterlist: Fix mapping iterator when sg->offset is greater than PAGE_SIZE crypto/NX: Set receive window credits to max number of CRBs in RxFIFO crypto: asymmetric_keys - select CRYPTO_HASH where needed crypto: serpent - mark __serpent_setkey_sbox noinline crypto: testmgr - dynamically allocate crypto_shash crypto: testmgr - dynamically allocate testvec_config crypto: talitos - eliminate unneeded 'done' functions at build time ... List of files: /linux/lib/crypto/arm64/aes-ce.S

7367bfeb2c141fb3ddff6b09bb5dfeb739b3d245 - crypto: arm64/aes-ce - implement 5 way interleave for ECB, CBC and CTR

Mon, 24 Jun 2019 17:38:31 +0000

crypto: arm64/aes-ce - implement 5 way interleave for ECB, CBC and CTRThis implements 5-way interleaving for ECB, CBC decryption and CTR,resulting in a speedup of ~11% on Marvell ThunderX2, which has avery deep pipeline and therefore a high issue latency for NEONinstructions operating on the same registers.Note that XTS is left alone: implementing 5-way interleave therewould either involve spilling of the calculated tweaks to thestack, or recalculating them after the encryption operation, anddoing either of those would most likely penalize low end cores.For ECB, this is not a concern at all, given that we have plentyof spare registers. For CTR and CBC decryption, we take advantageof the fact that v16 is not used by the CE version of the code(which is the only one targeted by the optimization), and so wecan reshuffle the code a bit and avoid having to spill to memory(with the exception of one extra reload in the CBC routine)Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

e217413964a453fc2eeb437c32deb00581cf899d - crypto: arm64/aes-ce - add 5 way interleave routines

Mon, 24 Jun 2019 17:38:30 +0000

crypto: arm64/aes-ce - add 5 way interleave routinesIn preparation of tweaking the accelerated AES chaining mode routinesto be able to use a 5-way stride, implement the core routines tosupport processing 5 blocks of input at a time. While at it, dropthe 2 way versions, which have been unused for a while now.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

d2912cb15bdda8ba4a5dd73396ad62641af2f520 - treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500

Tue, 04 Jun 2019 08:11:33 +0000

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500Based on 2 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation #extracted by the scancode license scanner the SPDX license identifier GPL-2.0-onlyhas been chosen to replace the boilerplate/reference in 4122 file(s).Signed-off-by: Thomas Gleixner Reviewed-by: Enrico Weigelt Reviewed-by: Kate Stewart Reviewed-by: Allison Randal Cc: linux-spdx@vger.kernel.orgLink: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.deSigned-off-by: Greg Kroah-Hartman List of files: /linux/lib/crypto/arm64/aes-ce.S

2e5d2f33d1dbd551f14634483c7ea81e5119d689 - crypto: arm64/aes-blk - improve XTS mask handling

Mon, 10 Sep 2018 14:41:15 +0000

crypto: arm64/aes-blk - improve XTS mask handlingThe Crypto Extension instantiation of the aes-modes.S collection ofskciphers uses only 15 NEON registers for the round key array, whereasthe pure NEON flavor uses 16 NEON registers for the AES S-box.This means we have a spare register available that we can use to holdthe XTS mask vector, removing the need to reload it at every iterationof the inner loop.Since the pure NEON version does not permit this optimization, tweakthe macros so we can factor out this functionality. Also, replace theliteral load with a short sequence to compose the mask vector.On Cortex-A53, this results in a ~4% speedup.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

0c8f838a52fe9fd82761861a934f16ef9896b4e5 - crypto: arm64/aes-blk - yield NEON after every block of input

Mon, 30 Apr 2018 16:18:24 +0000

crypto: arm64/aes-blk - yield NEON after every block of inputAvoid excessive scheduling delays under a preemptible kernel byyielding the NEON after every block of input.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

f402e3115e20b345bd6fbfcf463a506d958c7bf6 - crypto: arm64/aes-ce-cipher - match round key endianness with generic code

Mon, 24 Jul 2017 10:28:10 +0000

crypto: arm64/aes-ce-cipher - match round key endianness with generic codeIn order to be able to reuse the generic AES code as a fallback forsituations where the NEON may not be used, update the key handlingto match the byte order of the generic code: it stores round keysas sequences of 32-bit quantities rather than streams of bytes, andso our code needs to be updated to reflect that.Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

caf4b9e2b326cc2a5005a5c557274306536ace61 - crypto: arm64/aes-xts-ce: fix for big endian

Tue, 11 Oct 2016 18:15:19 +0000

crypto: arm64/aes-xts-ce: fix for big endianEmit the XTS tweak literal constants in the appropriate order for asingle 128-bit scalar literal load.Fixes: 49788fe2a128 ("arm64/crypto: AES-ECB/CBC/CTR/XTS using ARMv8 NEON and Crypto Extensions")Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S

4a97abd44329bf7b9c57f020224da5f823c9c9ea - arm64/crypto: issue aese/aesmc instructions in pairs

Tue, 17 Mar 2015 18:05:13 +0000

arm64/crypto: issue aese/aesmc instructions in pairsThis changes the AES core transform implementations to issue aese/aesmc(and aesd/aesimc) in pairs. This enables a micro-architectural optimizationin recent Cortex-A5x cores that improves performance by 50-90%.Measured performance in cycles per byte (Cortex-A57): CBC enc CBC dec CTR before 3.64 1.34 1.32 after 1.95 0.85 0.93Note that this results in a ~5% performance decrease for older cores.Signed-off-by: Ard Biesheuvel Signed-off-by: Will Deacon List of files: /linux/lib/crypto/arm64/aes-ce.S

49788fe2a128217f78a21ee4edbe6e92e988f222 - arm64/crypto: AES-ECB/CBC/CTR/XTS using ARMv8 NEON and Crypto Extensions

Fri, 21 Mar 2014 09:19:17 +0000

arm64/crypto: AES-ECB/CBC/CTR/XTS using ARMv8 NEON and Crypto ExtensionsThis adds ARMv8 implementations of AES in ECB, CBC, CTR and XTS modes,both for ARMv8 with Crypto Extensions and for plain ARMv8 NEON.The Crypto Extensions version can only run on ARMv8 implementations thathave support for these optional extensions.The plain NEON version is a table based yet time invariant implementation.All S-box substitutions are performed in parallel, leveraging the wide rangeof ARMv8's tbl/tbx instructions, and the huge NEON register file, which cancomfortably hold the entire S-box and still have room to spare for doing theactual computations.The key expansion routines were borrowed from aes_generic.Signed-off-by: Ard Biesheuvel Acked-by: Herbert Xu List of files: /linux/lib/crypto/arm64/aes-ce.S