Chen Fu 3c10f027de 4b quantization for weights of LLMs (#16833) ... ### Description Blockwise 4b quantization for LLMs. 1. Introduce 4b block-wise quantization for linear layer weights. 2. Implements matrix multiplication kernel for fp32 x int4 3. Implements special operator MatMulFpQ4 4. Implements quantization tool, that convert MatMul operator to MatMulFpQ4, when the right hand side is 2D const tensor. ### Motivation and Context Compress and accelerate LLMs |Benchmark | Time(ns)| |-------------|----------| |Q4GEMM/Q4Sym/M:1/N:4096/K:4096/Threads:8| 218054| |Q4GEMM/Q4Sym/M:1024/N:4096/K:4096/Threads:8| 35830155| |Q4GEMM/Q4Sym/M:2048/N:4096/K:4096/Threads:8| 73479790| |Q4GEMM/Q4Zp8/M:1/N:4096/K:4096/Threads:8| 270152| |Q4GEMM/Q4Zp8/M:1024/N:4096/K:4096/Threads:8| 35826721| |Q4GEMM/Q4Zp8/M:2048/N:4096/K:4096/Threads:8| 73021200| |Q4GEMM/Q4Sym128/M:1/N:4096/K:4096/Threads:8| 213832| |Q4GEMM/Q4Sym128/M:1024/N:4096/K:4096/Threads:8| 36749874| |Q4GEMM/Q4Sym128/M:2048/N:4096/K:4096/Threads:8| 72618120| |Benchmark | Time(ns)| |-------------|----------| |SGEMM/LLM/M:1/N:4096/K:4096/Threads:8| 522610| |SGEMM/LLM/M:1024/N:4096/K:4096/Threads:8| 39237689| |SGEMM/LLM/M:2048/N:4096/K:4096/Threads:8| 75983467| --------- Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>		2023-08-07 12:23:55 -07:00
..
c_cxx	Training Documentation (#15612)	2023-04-25 11:44:12 -07:00
execution_providers/images
images
python	Disable PERF* rules in ruff to allow better readability (#16834)	2023-07-25 15:38:22 -07:00
ABI_Dev_Notes.md	skip windows GPU check if changes only in doc (#13248)	2022-10-11 13:51:44 +08:00
Android_testing.md
C_API_Guidelines.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
cmake_guideline.md	fix some typo in docs (#13212)	2022-10-07 15:58:18 -07:00
Coding_Conventions_and_Standards.md	[docs] Specify Objective-C max line length. (#16503)	2023-06-28 16:58:23 -07:00
ContribOperators.md	4b quantization for weights of LLMs (#16833)	2023-08-07 12:23:55 -07:00
FAQ.md	Fix typo enviroment => environment (#13195)	2022-10-03 17:02:26 -07:00
How_To_Update_ONNX_Dev_Notes.md	Remove exclusions for ONNX model tests that now pass. (#14337)	2023-01-24 08:04:27 +10:00
Memory_Optimizer.md	Add guidelines for ORTModule (#13553)	2022-11-04 19:42:10 +08:00
Model_Test.md
NotesOnThreading.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
ONNX_Runtime_Server_Usage.md
onnxruntime_dependencies.dot
onnxruntime_dependencies.png
onnxruntime_extensions.md	Fix broken and outdated links in documentation (#14092)	2023-02-23 10:48:04 -08:00
OperatorKernels.md	4b quantization for weights of LLMs (#16833)	2023-08-07 12:23:55 -07:00
ORT_Format_Update_in_1.13.md	Update ORT format v5 change docs to cover limited backwards compatibility in 1.14. (#14413)	2023-01-25 08:23:12 -08:00
ORT_Use_Trtion_Kernel.md	[ROCm] Add ROCm Triton TunableOp for GroupNorm (#16196)	2023-07-11 13:55:30 +08:00
ORTMobilePackageOperatorTypeSupport.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
ORTModule_Convergence_Notes.md	Enhance StatisticsSubscriber (#16098)	2023-06-12 18:32:08 +08:00
ORTModule_ModuleWithLoss_Wrapper.md	add steps to write modulewithloss wrapper (#16486)	2023-07-11 09:07:35 +08:00
ORTModule_Training_Guidelines.md	ORT Cache (#16744)	2023-07-27 09:00:43 -07:00
PR_Guidelines.md
Privacy.md
Python_Dev_Notes.md
Reduced_Operator_Kernel_build.md	replace 'master' branch ref to 'main' for onnx repo (#12678)	2022-08-30 13:41:42 -07:00
ReleaseManagement.md
Roadmap.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
Server.md
TVM_EP.md	stop support to VS 2019 (#16892)	2023-07-28 13:09:35 +08:00
Versioning.md	replace 'master' branch ref to 'main' for onnx repo (#12678)	2022-08-30 13:41:42 -07:00
WinML_principles.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00