Adding crc32 specialization for aarch64 which uses both PMULL and crc32
instructions to perform 192 bytes fold in one iteration, performing
9x PMULL and 6 crc32 in one loop iteration, obtaining higher performance for
large inputs >8kB. This approach is based on zlib-ng implementation which
is also described at https://github.com/corsix/fast-crc32.
For smaller buffer size, it was observed to be slightly slower, thus only
for input size >8192 this logic is used, for smaller sizes otherwise the
4x PMULL folding method is used along with scalar crc32 instructions for
processing the remainder input size.
On a MediaTek Dimensity 9400 Android device in termux environment,
with normal checkasm seed 0 which picks random buffer size and max buffer size
of 16kB, the data observed on Cortex X925, A720 and X4:
X925 Before:
crc_32_IEEE_LE_c: 12762.0
crc_32_IEEE_LE_crc: 667.5 (19.11x)
crc_32_IEEE_LE_pmull_eor3: 346.9 (26.30x)
X925 After:
crc_32_IEEE_LE_c: 12707.6
crc_32_IEEE_LE_crc: 665.2 (19.10x)
crc_32_IEEE_LE_pmull_eor3: 292.8 (41.90x)
A720 Before:
crc_32_IEEE_LE_c: 23059.1
crc_32_IEEE_LE_crc: 1220.7 (18.89x)
crc_32_IEEE_LE_pmull_eor3: 1198.9 (19.23x)
A720 After:
crc_32_IEEE_LE_c: 23293.3
crc_32_IEEE_LE_crc: 1209.1 (19.26x)
crc_32_IEEE_LE_pmull_eor3: 1150.4 (20.24x)
X4 Before:
crc_32_IEEE_LE_c: 12405.5
crc_32_IEEE_LE_crc: 664.5 (18.67x)
crc_32_IEEE_LE_pmull_eor3: 498.1 (24.90x)
X4 After:
crc_32_IEEE_LE_c: 12457.2
crc_32_IEEE_LE_crc: 665.5 (18.72x)
crc_32_IEEE_LE_pmull_eor3: 468.8 (26.57x)
So it seems to work well on high performance core like X925, and results in about
20% better performance, while having tiny gains on other cores.
Testing for input size of 160 kB after modifying the checkasm crc test to
have buffer size increased to 160kB and always using full capacity instead of
a random size results in below observations:
X925 Before:
crc_32_IEEE_LE_c: 210177.1
crc_32_IEEE_LE_crc: 10313.7 (20.35x)
crc_32_IEEE_LE_pmull_eor3: 6580.9 (31.83x)
X925 After:
crc_32_IEEE_LE_c: 210869.3
crc_32_IEEE_LE_crc: 10304.8 (20.36x)
crc_32_IEEE_LE_pmull_eor3: 3098.5 (68.05x)
A720 Before:
crc_32_IEEE_LE_c: 387502.5
crc_32_IEEE_LE_crc: 19196.7 (19.54x)
crc_32_IEEE_LE_pmull_eor3: 18717.1 (20.63x)
A720 After:
crc_32_IEEE_LE_c: 392090.8
crc_32_IEEE_LE_crc: 19795.1 (18.68x)
crc_32_IEEE_LE_pmull_eor3: 14971.4 (24.97x)
X4 Before:
crc_32_IEEE_LE_c: 196232.0
crc_32_IEEE_LE_crc: 10378.7 (18.68x)
crc_32_IEEE_LE_pmull_eor3: 7742.0 (25.29x)
X4 After:
crc_32_IEEE_LE_c: 199632.9
crc_32_IEEE_LE_crc: 10495.8 (18.32x)
crc_32_IEEE_LE_pmull_eor3: 5448.9 (24.69x)
Seems to result in about 2x gains on X925, 25% on A70 and 40% on X4.
In general the performance gains depends on the CPU Core and input size,
and this optimization benefits large input size especially on high performance
cores like X925 and Apple M series.
Unroll to 16 floats per iteration with four independent accumulators
and reduce them once after the loop.
scalarproduct_float_neon: before after
Apple M1 (clang 16): 0.9 (3.56x) 0.4 (9.18x)
Cortex-A76 (gcc 12.4): 118.7 (4.43x) 85.3 (6.15x)
Signed-off-by: Zhao Zhili <quinkblack@foxmail.com>
This adds a NEON-optimized function for computing 32x32 Sum of Absolute
Differences (SAD) on AArch64, addressing a gap where x86 had SSE2/AVX2
implementations but AArch64 lacked equivalent coverage.
The implementation mirrors the existing sad8 and sad16 NEON functions,
employing a 4-row unrolled loop with UABAL and UABAL2 instructions for
efficient load-compute interleaving, and four 8x16-bit accumulators to
handle the wider 32-byte rows.
Benchmarks on AWS Graviton3 (Neoverse V1, c7g.xlarge) using checkasm:
sad_32x32_0: C 146.4 cycles -> NEON 98.1 cycles (1.49x speedup)
sad_32x32_1: C 141.4 cycles -> NEON 98.9 cycles (1.43x speedup)
sad_32x32_2: C 140.7 cycles -> NEON 95.0 cycles (1.48x speedup)
Signed-off-by: Jeongkeun Kim <variety0724@gmail.com>
We currently don't have any cases where this is needed, but include
it for completeness and clarity.
These macros for BTI were added in
08b4716a9e.
A later comment in this file, added in
248986a0db, referenced the macro
AARCH64_VALID_JUMP_CALL_TARGET which never was added here before.
The function macro emits AARCH64_VALID_CALL_TARGET for exported symbols,
marking them as valid destinations for indirect _calls_. Functions that
are reached by indirect _branches_ (i.e. tail-call dispatch chains
where the link register is not set) require AARCH64_VALID_JUMP_TARGET
instead.
This commit adds a "jumpable" parameter to the function macro that, when
set, emits AARCH64_VALID_JUMP_TARGET instead of AARCH64_VALID_CALL_TARGET.
Sponsored-by: Sovereign Tech Fund
Signed-off-by: Ramiro Polla <ramiro.polla@gmail.com>
Whenever the link register is stored on the stack, sign it
before storing it and validate at a symmetrical point (with the
stack at the same level as when it was signed).
These macros only have an effect if built with PAC enabled (e.g.
through -mbranch-protection=standard), otherwise they don't
generate any extra instructions.
None of these cases were present when PAC support was added
in 248986a0db in 2022.
Without these changes, PAC still had an effect in the compiler
generated code and in the existing cases where we these macros were
used - but make it apply to the remaining cases of link register
on the stack.
Signal that our assembly is compliant with the GCS feature, if
the GCS feature is enabled in the compiler (available since Clang
18 and GCC 15) - this is enabled by -mbranch-protection=standard
with a new enough compiler.
GCS doesn't require any specific modifications to the assembly
code, but requires that all functions return to the expected call
address (checked through a shadow stack).
Newer revisions of WinSDK 10.0.26100.0 have exposed more flags for
IsProcessorFeaturePresent; now there is a separate one for
detecting specifically I8MM and not just SVE-I8MM. Switch to using
this flag instead.
Name the feature "arm_crc" rather than plain "crc", to make it
clear that this is about a CPU feature extension, not CRC
implementations in general.
This requires dealing with the extension slightly differently
than other extensions, as the name of the feature and the
".arch_extension" extension name differ.
Naming it with an "arm" prefix rather than "aarch64", as the
CPU extension also is available in 32 bit ARM form, even though
we don't intend to use it there.
Use 16-byte alignment (align=4) instead of 4-byte (align=2) in the function and
const macros. This improves instruction fetch and NEON load performance on
modern AArch64 CPUs.
There's no direct processor feature constant for I8MM alone, but
there is a flag for SVE-I8MM (added in WinSDK 10.0.26100 and
recent versions of mingw-w64). If SVE-I8MM is available, we can
assume that I8MM is available.
While HW supporting these features isn't yet commonly running
Windows, this at least allows detecting and running the I8MM codepaths
in Windows builds in Wine (possibly running in QEMU).
Signed-off-by: Martin Storsjö <martin@martin.st>
The default timer register pmccntr_el0 usually requires enabling
access with e.g. a kernel module (while it is accessible by
default on Windows). On Linux, the default for checkasm benchmarks
is to use perf (if suitable headers are available) though.
On macOS, using cntvct_el0 gives measurements with the same
magnitude as mach_absolute_time (which is used currently), but
possibly with a little less overhead/noise.
Signed-off-by: Martin Storsjö <martin@martin.st>
C code or compiler built-ins are preferable over inline assembler for
byte-swaps as it allows for better optimisations (e.g. instruction
scheduling) which would otherwise be impossible.
As with f64c2e710f for x86 and Arm,
this removes the inline assembler on GCC (and Clang) since we now
require recent enough compiler versions. This indeed seems to work on
AArch64, SuperH and, if Zbb is enabled, RISC-V. (AVR32 was not tested
since it has no known working compilers at this time.)
It will fallback to mach_absolute_time inside libavutil/timer.h
Reviewed-by: Martin Storsjö <martin@martin.st>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
This makes the code much simpler (especially for adding support
for other instruction set extensions), avoids needing inline
assembly for this feature, and generally is more of the canonical
way to do this.
The CPU feature detection was added in
493fcde50a, using HWCAP_CPUID.
The argument for using that, was that HWCAP_CPUID was added much
earlier in the kernel (in Linux v4.11), while the HWCAP flags for
individual features always come later. This allows detecting support
for new CPU extensions before the kernel exposes information about
them via hwcap flags.
However in practice, there's probably quite little advantage in this.
E.g. HWCAP2_I8MM was added in Linux v5.10 - long after HWCAP_CPUID,
but there's probably very little practical cases where one would
run a kernel older than that on a CPU that supports those instructions.
Additionally, we provide our own definitions of the flag values to
check (as they are fixed constants anyway), with names not conflicting
with the ones from system headers. This reduces the number of ifdefs
needed, and allows detecting those features even if building with
userland headers that are lacking the definitions of those flags.
Also, slightly older versions of QEMU, e.g. 6.2 in Ubuntu 22.04,
do expose support for these features via HWCAP flags, but the
emulated cpuid registers are missing the bits for exposing e.g. I8MM.
(This issue is fixed in later versions of QEMU though.)
Signed-off-by: Martin Storsjö <martin@martin.st>
This eases actual development of the assembly functions, by only
allowing extension instructions within the sections that explicitly
enable them, instead of having all extensions enabled everywhere.
Signed-off-by: Martin Storsjö <martin@martin.st>
Favour left aligned columns over right aligned columns.
In principle either style should be ok, but some of the cases
easily lead to incorrect indentation in the surrounding code (see
a couple of cases fixed up in the preceding patch), and show up in
automatic indentation correction attempts.
Signed-off-by: Martin Storsjö <martin@martin.st>
Some functions have slightly different indentation styles; try
to match the surrounding code.
libavcodec/aarch64/vc1dsp_neon.S is skipped here, as it intentionally
uses a layered indentation style to visually show how different
unrolled/interleaved phases fit together.
Signed-off-by: Martin Storsjö <martin@martin.st>
This is not actually used for anything. The configure check causes the
CPU feature flag to be set, but nothing consumes it at all.
While AArch64 does have VFP, it is only used for the scalar C code.
Conversely, it is still possible to disable VFP, by changing the
C compiler flags as before (though that only makes sense for an
hypothetical non-standard Armv8 platform without VFP).
Note that this retains the "vfp" option flag, for backward
compatibility and on the very remote but theoretically possible chance
that FFmpeg actually makes use of it in the future.
AV_CPU_FLAG_VFP is retained as it is actually used by AArch32.
For now, there's not much value in this since Clang don't support
enabling the dotprod or i8mm features with either .arch_extension
or .arch (it has to be enabled by the base arch flags passed to
the compiler). But it may be supported in the future.
Signed-off-by: Martin Storsjö <martin@martin.st>
Shreesh Adiga
Zhao Zhili
Zhao Zhili
Jeongkeun Kim
Martin Storsjö
Ramiro Polla
Andreas Rheinhardt
Georgii Zagoruiko
Brad Smith
James Almer
Krzysztof Pyrkosz via ffmpeg-devel
Rémi Denis-Courmont
Reimar Döffinger