liqun Fu cc26b2dac2 Mlas Gemm 4bit avx2, avx512, and avx512vnni kernels (#20163) ... ### Description ``` Avx2: Int8 NS(Prompt) MLAS(Prompt) MLAS(Prompt)Gain/Loss NS(TokenGen) MLAS(TokenGen) MLAS(TokenGen)Gain/Loss Blklen16: 90.96 25.15 -72% 7.65 11.71 53% Blklen32: 90.73 48.55 -46% 7.86 14.28 81% Blklen64: 89.49 68.84 -23% 8.30 15.78 90% Blklen128: 87.38 78.37 -10% 7.90 16.05 103% Blklen256: 89.45 82.36 -7% 8.30 16.56 99% Fp32 NS(Prompt) MLAS(Prompt) MLAS(Prompt)Gain/Loss NS(TokenGen) MLAS(TokenGen) MLAS(TokenGen)Gain/Loss Blklen16: 91.36 105.18 15% 7.57 9.52 25% Blklen32: 89.30 105.99 18% 7.65 9.68 26% Blklen64: 89.53 101.41 13% 7.97 9.84 23% Blklen128: 85.23 99.71 16% 7.86 10.39 32% Blklen256: 88.46 97.94 10% 8.32 10.23 22% Avx512vnni: Int8 NS(Prompt) MLAS(Prompt) MLAS(Prompt)Gain/Loss NS(TokenGen) MLAS(TokenGen) MLAS(TokenGen)Gain/Loss Blklen16: 132.18 21.56 -83% 10.34 11.48 11% Blklen32: 168.28 43.69 -74% 11.85 14.73 24% Blklen64: 201.81 60.29 -70% 12.36 15.47 25% Blklen128: 194.92 57.04 -71% 13.03 14.67 12% Blklen256: 218.76 70.20 -68% 13.33 16.31 22% Fp32 NS(Prompt) MLAS(Prompt) MLAS(Prompt)Gain/Loss NS(TokenGen) MLAS(TokenGen) MLAS(TokenGen)Gain/Loss Blklen16: 102.81 92.74 -9% 8.41 9.18 9% Blklen32: 109.49 97.08 -11% 8.83 11.51 30% Blklen64: 104.13 101.57 -2% 9.32 12.00 28% Blklen128: 108.45 103.69 -4% 9.58 12.45 29% Blklen256: 109.43 106.43 -2% 9.19 12.2 32% ``` --------- Signed-off-by: Liqun Fu <liqfu@microsoft.com> Signed-off-by: liqunfu <liqun.fu@microsoft.com> Co-authored-by: edgchen1 <18449977+edgchen1@users.noreply.github.com>		2024-04-25 21:30:50 -07:00
..
c_cxx	Remove extraneous javascript includes (#17558)	2023-09-14 20:43:24 -07:00
execution_providers/images
images
python	Bump ruff to 0.3.2 and black to 24 (#19878)	2024-03-13 10:00:32 -07:00
ABI_Dev_Notes.md	Fix a typo in ABI_Dev_Notes.md (#17832)	2023-10-09 07:51:34 -07:00
Android_testing.md
C_API_Guidelines.md
cmake_guideline.md
Coding_Conventions_and_Standards.md	[docs] Specify Objective-C max line length. (#16503)	2023-06-28 16:58:23 -07:00
ContribOperators.md	Mlas Gemm 4bit avx2, avx512, and avx512vnni kernels (#20163)	2024-04-25 21:30:50 -07:00
FAQ.md	[Technical docs] Fixed a couple of old links in FAQ.md (#17415)	2023-09-26 13:38:24 -07:00
How_To_Update_ONNX_Dev_Notes.md	Integration with ONNX 1.16.0 (#19745)	2024-04-12 09:46:49 -07:00
Memory_Optimizer.md	Prompt layer-wise recompute when applicable (#20126)	2024-04-10 11:50:28 +08:00
Model_Test.md
NotesOnThreading.md
ONNX_Runtime_Server_Usage.md
onnxruntime_dependencies.dot
onnxruntime_dependencies.png
onnxruntime_extensions.md	Remove the extensions submodule (#17097)	2023-08-14 10:16:33 -07:00
OperatorKernels.md	add bf16 support for few ops (#20385)	2024-04-25 11:28:34 -07:00
ORT_Format_Update_in_1.13.md
ORT_Use_Trtion_Kernel.md	[ROCm] Add ROCm Triton TunableOp for GroupNorm (#16196)	2023-07-11 13:55:30 +08:00
ORTMobilePackageOperatorTypeSupport.md
ORTModule_Convergence_Notes.md	Fix and enable few ORTModule Unit Tests (#19847)	2024-03-12 10:49:19 +08:00
ORTModule_ModuleWithLoss_Wrapper.md	add steps to write modulewithloss wrapper (#16486)	2023-07-11 09:07:35 +08:00
ORTModule_PythonOp_Notes.md	Add document for PythonOp (#17888)	2023-10-12 08:36:22 +08:00
ORTModule_Training_Guidelines.md	Introducing ORTPipelineModule - DeepSpeed Parallel Pipeline Support. (#20287)	2024-04-18 11:30:15 -07:00
PR_Guidelines.md
Privacy.md
Python_Dev_Notes.md
Reduced_Operator_Kernel_build.md
ReleaseManagement.md
Roadmap.md
Server.md
TVM_EP.md	Fix: update hyperlinks to the Jupyter notebooks (#16145)	2023-08-21 09:53:05 -07:00
Versioning.md
WinML_principles.md