saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-05-22 22:01:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/cmake
History
Chen Fu 90142899bd
Supporting Intel AMX instructions in quantized GEMM (#14042) 
### Description
Using Intel AMX int8 instructions to accelerate quantized GEMM


### Motivation and Context
AMX instructions accelerate quantized GEMM significantly:

Prepacked B perf numbers (latency in ns)

GEMM Config | AVX512Vnni | AMX
-- | --: | --:
M:384/N:1024/K:1024/Batch:1/Threads:4 | 1057511 | 285393
M:384/N:1024/K:3072/Batch:1/Threads:4 | 2643929 | 700397
M:384/N:1024/K:4096/Batch:1/Threads:4 | 3784750 | 890701
M:384/N:4096/K:1024/Batch:1/Threads:4 | 2378139 | 887251
M:384/N:1024/K:1024/Batch:1/Threads:16 | 307137 | 138481
M:384/N:1024/K:3072/Batch:1/Threads:16 | 855730 | 295027
M:384/N:1024/K:4096/Batch:1/Threads:16 | 1126878 | 317395
M:384/N:4096/K:1024/Batch:1/Threads:16 | 781963 | 237014
M:1536/N:1024/K:1024/Batch:1/Threads:16 | 538864 | 181459
M:1536/N:1024/K:3072/Batch:1/Threads:16 | 1681002 | 561600
M:1536/N:1024/K:4096/Batch:1/Threads:16 | 2158127 | 717470
M:1536/N:4096/K:1024/Batch:1/Threads:16 | 2428622 | 896140
M:3072/N:1024/K:1024/Batch:1/Threads:16 | 1058029 | 357031
M:3072/N:1024/K:3072/Batch:1/Threads:16 | 3138504 | 1095857
M:3072/N:1024/K:4096/Batch:1/Threads:16 | 4155640 | 1386183
M:3072/N:4096/K:1024/Batch:1/Threads:16 | 4679030 | 1778624

Co-authored-by: Yi-Hong Lyu <yilyu@microsoft.com>
Co-authored-by: Chen Fu <fuchen@microsoft.com>
2023-01-10 12:16:27 -08:00
..
external pin ort-ext to 81e7799c69044c745239202085eb0a98f102937b (#14044) 2023-01-10 10:10:17 -08:00
patches [ROCm] Add GemmFastGelu CK implementation (#13759) 2023-01-05 17:53:30 +08:00
tensorboard Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
adjust_global_compile_flags.cmake Address follow-up comments on multistream pr #13495 (#13992) 2023-01-03 16:33:36 -08:00
CMakeLists.txt Enable cache for msbuild (#14085) 2023-01-06 11:19:57 +08:00
CMakeSettings.json
codeconv.runsettings
deps.txt to work with onnx 1.13 rc, implement ver 18 reduce and optioanl ops, … (#13765) 2023-01-09 10:26:16 -08:00
EnableVisualStudioCodeAnalysis.props Fix SDL warnings in CPU EP (#9975) 2021-12-19 20:54:29 -08:00
gdk_toolchain.cmake Enable building with a GDK (#11126) 2022-04-07 15:06:31 -07:00
Info.plist.in
libonnxruntime.pc.cmake.in
nuget_helpers.cmake
onnxruntime.cmake CloudEP (#13855) 2023-01-03 10:03:15 -08:00
onnxruntime_codegen_tvm.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_common.cmake Enabling thread pool to be numa-aware (#13778) 2022-12-12 10:33:55 -08:00
onnxruntime_config.h.in [wasm] update emscripten v2.0.34 (#10391) 2022-01-26 14:46:02 -08:00
onnxruntime_csharp.cmake Refactor training build options (#13964) 2023-01-03 13:28:16 -08:00
onnxruntime_eager.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_flatbuffers.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_framework.cmake CloudEP (#13855) 2023-01-03 10:03:15 -08:00
onnxruntime_fuzz_test.cmake
onnxruntime_graph.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_ios.toolchain.cmake
onnxruntime_java.cmake Add linux and macos arm64 java aritifacts (#10981) 2022-03-25 16:23:17 -07:00
onnxruntime_java_unittests.cmake [Java] Adds support for DNNL, OpenVINO, TensorRT shared providers and refactors the CUDA shared provider loader (#8013) 2021-07-20 22:33:15 -07:00
onnxruntime_kernel_explorer.cmake [ROCm] Add GemmFastGelu CK implementation (#13759) 2023-01-05 17:53:30 +08:00
onnxruntime_language_interop_ops.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_mlas.cmake Supporting Intel AMX instructions in quantized GEMM (#14042) 2023-01-10 12:16:27 -08:00
onnxruntime_nodejs.cmake Add Node.js binding support to packaging pipeline (#9577) 2021-11-05 15:29:40 -07:00
onnxruntime_objectivec.cmake Remove SafeInt dependency from Objective-C API. (#13698) 2022-11-18 17:06:12 -08:00
onnxruntime_opschema_lib.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_optimizer.cmake Optimize computation orders (#13672) 2022-12-22 15:12:52 +08:00
onnxruntime_providers.cmake Add batched and strided batched gemm as TunableOp (#13841) 2023-01-07 19:11:40 +08:00
onnxruntime_pyop.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_python.cmake Refactor training build options (#13964) 2023-01-03 13:28:16 -08:00
onnxruntime_rocm_hipify.cmake rename CrossAttention to MultiHeadAttention (#14201) 2023-01-10 10:18:39 -08:00
onnxruntime_session.cmake Refactor training build options (#13964) 2023-01-03 13:28:16 -08:00
onnxruntime_snpe_provider.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
onnxruntime_training.cmake Refactor training build options (#13964) 2023-01-03 13:28:16 -08:00
onnxruntime_unittests.cmake Supporting Intel AMX instructions in quantized GEMM (#14042) 2023-01-10 12:16:27 -08:00
onnxruntime_util.cmake Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
onnxruntime_webassembly.cmake Fix usage of enable_training_ops and reduce ifdef complexity for training builds (#13888) 2022-12-14 08:32:46 -08:00
precompiled_header.cmake Fix Windows Store build (#8753) 2021-08-23 11:19:03 -07:00
Sdl.ruleset Update Sdl.ruleset to remove C26812 from the rules (#12695) 2022-09-01 20:05:20 -07:00
set_winapi_family_desktop.h
target_delayload.cmake Remove Windows Store specific code 2022-03-17 23:38:14 -07:00
uwp_stubs.h Fix Windows Store build (#8753) 2021-08-23 11:19:03 -07:00
wcos_rules_override.cmake
winml.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00
winml_cppwinrt.cmake Fix Windows Store build (#8753) 2021-08-23 11:19:03 -07:00
winml_sdk_helpers.cmake
winml_unittests.cmake Use target name for flatbuffers (#13991) 2022-12-20 11:44:02 -08:00