Jing Fang 13348c572a [ARM CPU] hgemm optimized for gqa (#23107) ... ### Description Add fp16 kernels for GQA matmul on ARM CPU. The kernels are mlas hgemm for C = alpha * A x B' + beta * C ### Motivation and Context Add fp16 support for GQA, speed up the operator and reduce memory usage. __Token Generation__ | | HGEMM Runtime (ns) | SGEMM Runtime (ns) | Speed-up (%) | |---------------------------------|--------------------|--------------------|--------------| | M:1/N:4096/K:4096 | 251551 | 1775905 | 85.84 | | M:1/N:11008/K:4096 | 892507 | 4649145 | 80.80 | | M:1/N:4096/K:11008 | 866860 | 3240015 | 73.25 | | M:1/N:11008/K:11008 | 2631615 |8783877 | 70.04 | __Prompting__ | | HGEMM Runtime (ns) | SGEMM Runtime (ns) | Speed-up (%) | |---------------------------------|--------------------|--------------------|--------------| | M:1024/N:4096/K:4096 | 90508701 | 111283029 | 18.67 | | M:2048/N:4096/K:4096 | 181307522 | 240211107 | 24.52 | | M:1024/N:11008/K:4096 | 241120234 | 307707933 | 21.64 | | M:2048/N:11008/K:4096 | 481091232 | 648921367 | 25.86 | | M:1024/N:4096/K:11008 | 241736343 | 310129880 | 22.05 | | M:2048/N:4096/K:11008 | 480456703 | 644814999 | 25.49 | | M:1024/N:11008/K:11008 | 642121440 | 847925766 | 24.27 | | M:2048/N:11008/K:11008 | 1276097154 | 1731314509 | 26.29		2025-01-24 15:25:24 -08:00
..
external	Make ORT and Dawn use the same protobuf/abseil source code (#23447)	2025-01-21 17:17:47 -08:00
patches	[WebGPU] allow build WebGPU EP for WebAssembly (#23364)	2025-01-16 10:52:17 -08:00
tensorboard	Improve dependency management (#13523)	2022-12-01 09:51:59 -08:00
vcpkg-ports	Add custom vcpkg ports (#23456)	2025-01-22 11:49:16 -08:00
vcpkg-triplets	Update the compile flags for vcpkg packages (#23455)	2025-01-22 11:48:38 -08:00
adjust_global_compile_flags.cmake	Update ORT extension to the latest (#23314)	2025-01-13 18:59:42 -08:00
arm64x.cmake
CMakeLists.txt	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
CMakePresets.json	Correct ONNX and Protobuf version in vcpkg build (#23285)	2025-01-08 12:25:17 -08:00
CMakeSettings.json
codeconv.runsettings
deps.txt	Update xnnpack, cpuinfo and pthreadpool (#23362)	2025-01-15 09:42:15 -08:00
deps_update_and_upload.py
EnableVisualStudioCodeAnalysis.props	Fix SDL warnings in CPU EP (#9975)	2021-12-19 20:54:29 -08:00
gdk_toolchain.cmake
hip_fatbin_insert
Info.plist.in
libonnxruntime.pc.cmake.in
linux_arm32_crosscompile_toolchain.cmake
linux_arm64_crosscompile_toolchain.cmake
maccatalyst_prepare_objects_for_prelink.py
nuget_helpers.cmake
onnxruntime.cmake	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
onnxruntime_common.cmake
onnxruntime_compile_triton_kernel.cmake
onnxruntime_config.h.in	Pre-requisites of upgrading EMSDK (#23347)	2025-01-14 11:07:21 -08:00
onnxruntime_csharp.cmake
onnxruntime_flatbuffers.cmake
onnxruntime_framework.cmake
onnxruntime_framework.natvis
onnxruntime_fuzz_test.cmake
onnxruntime_graph.cmake
onnxruntime_ios.toolchain.cmake
onnxruntime_java.cmake	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
onnxruntime_java_unittests.cmake
onnxruntime_kernel_explorer.cmake
onnxruntime_lora.cmake
onnxruntime_mlas.cmake	[ARM CPU] hgemm optimized for gqa (#23107)	2025-01-24 15:25:24 -08:00
onnxruntime_nodejs.cmake
onnxruntime_objectivec.cmake
onnxruntime_opschema_lib.cmake
onnxruntime_optimizer.cmake	Update Linux docker images (#23244)	2025-01-09 10:20:33 -08:00
onnxruntime_providers.cmake	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
onnxruntime_providers_acl.cmake
onnxruntime_providers_armnn.cmake
onnxruntime_providers_azure.cmake
onnxruntime_providers_cann.cmake
onnxruntime_providers_coreml.cmake	Update xnnpack, cpuinfo and pthreadpool (#23362)	2025-01-15 09:42:15 -08:00
onnxruntime_providers_cpu.cmake	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
onnxruntime_providers_cuda.cmake
onnxruntime_providers_dml.cmake
onnxruntime_providers_dnnl.cmake
onnxruntime_providers_js.cmake
onnxruntime_providers_migraphx.cmake
onnxruntime_providers_nnapi.cmake
onnxruntime_providers_openvino.cmake
onnxruntime_providers_qnn.cmake	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
onnxruntime_providers_rknpu.cmake
onnxruntime_providers_rocm.cmake
onnxruntime_providers_tensorrt.cmake
onnxruntime_providers_vitisai.cmake
onnxruntime_providers_vsinpu.cmake
onnxruntime_providers_webgpu.cmake	[WebGPU] allow build WebGPU EP for WebAssembly (#23364)	2025-01-16 10:52:17 -08:00
onnxruntime_providers_webnn.cmake
onnxruntime_providers_xnnpack.cmake
onnxruntime_python.cmake	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
onnxruntime_rocm_hipify.cmake
onnxruntime_session.cmake
onnxruntime_snpe_provider.cmake
onnxruntime_training.cmake
onnxruntime_unittests.cmake	[QNN EP] Make QNN EP a shared library (#23120)	2025-01-22 12:11:00 -08:00
onnxruntime_util.cmake
onnxruntime_visionos.toolchain.cmake
onnxruntime_webassembly.cmake	[WebGPU] allow build WebGPU EP for WebAssembly (#23364)	2025-01-16 10:52:17 -08:00
precompiled_header.cmake
riscv64.toolchain.cmake
Sdl.ruleset
set_winapi_family_desktop.h
target_delayload.cmake	Refactor the cmake code that is related to delay loading (#22646)	2024-11-04 16:30:50 -08:00
uwp_stubs.h
vcpkg-configuration.json	Correct ONNX and Protobuf version in vcpkg build (#23285)	2025-01-08 12:25:17 -08:00
vcpkg.json	Correct ONNX and Protobuf version in vcpkg build (#23285)	2025-01-08 12:25:17 -08:00
wcos_rules_override.cmake
winml.cmake	Fix a tiny problem in winml.cmake (#23173)	2024-12-20 11:48:43 -08:00
winml_cppwinrt.cmake
winml_sdk_helpers.cmake
winml_unittests.cmake