saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-29 20:14:01 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/tools/ci_build
History
RandySheriffH 4dfb89b3ad
Implement mutex-free spin lock for task queue (#14834) 
Implemented "lock-free" spinlock to save CPU usage on context switching.
The change has been tested on queene service of Ads team, the lock-free
version of ort (40 threads) saves CPU usage on gen8 (128 logical
processors on 8 numa nodes) windows by nearly half, from 65% to 35%.

For 32 cores, the curve is flat:

Anubis, 32 vCPU, windows, hugging face models,
95 percentile E2E latency in ms:

model | mutex(ms) | mutex-free
--- | --- | ---
 alvert_base_v2 | 34.21 | 34.09
 bert_large_uncased | 116.27| 117.84
 bart_base | 72.06 | 71.99
 distilgpt2 | 25.43 | 25.02
 vit_base_patch16_224 | 37.33 | 37.76

Anubis, 32 vCPU win, Linux, 1st party models,
95 percentile E2E latency in ms:

model | mutex(ms) | mutex-free
--- | --- | ---
deepthink_v2 | 24.35 | 22.95
bing_feeds |  36.96 | 36.48
deep_writes |  14.46 | 14.32
keypoints |  9.34 | 7.69
model11 |  1.71 | 1.66
model12 |  1.82 | 1.44
model2 |  4.21 | 3.95
model6 |  1.08 | 1.05
agiencoder |  0.99 | 0.93
geminet_transformer |  5.32 | 5.24

---------

Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-19 10:12:10 -07:00
..
github Implement mutex-free spin lock for task queue (#14834) 2023-05-19 10:12:10 -07:00
__init__.py
amd_hipify.py Adopt linrtunner as the linting tool - take 2 (#15085) 2023-03-24 15:29:03 -07:00
build.py Implement mutex-free spin lock for task queue (#14834) 2023-05-19 10:12:10 -07:00
clean_docker_image_cache.py Adopt linrtunner as the linting tool - take 2 (#15085) 2023-03-24 15:29:03 -07:00
compile_triton.py integrate triton into ort (#15862) 2023-05-17 09:35:28 +08:00
coverage.py Format all python files under onnxruntime with black and isort (#11324) 2022-04-26 09:35:16 -07:00
gen_def.py Basic CSharp packaging support for ROCm EP (#15535) 2023-05-16 07:27:38 +08:00
get_docker_image.py Bump ruff in CI (#15533) 2023-04-17 10:11:44 -07:00
logger.py Format all python files under onnxruntime with black and isort (#11324) 2022-04-26 09:35:16 -07:00
op_registration_utils.py Support additional op domains in op reduction script. (#15424) 2023-04-11 08:57:51 -07:00
op_registration_validator.py Update kernel registration validation to allow kernel registrations to appear in arbitrary order. (#15705) 2023-04-27 18:49:31 -07:00
patch_manylinux.py detach patch manylinux from get_docker_image (#14958) 2023-03-09 15:40:58 +08:00
policheck_exclusions.xml Exculde hipify option from policheck (#13431) 2022-10-25 16:35:16 +08:00
reduce_op_kernels.py Bump ruff in CI (#15533) 2023-04-17 10:11:44 -07:00
replace_urls_in_deps.py Adopt linrtunner as the linting tool - take 2 (#15085) 2023-03-24 15:29:03 -07:00
requirements.txt upgrade protobuf to 3.20.2 and onnx to 1.13 (#14279) 2023-01-31 12:55:09 -08:00
update_tsaoptions.py Adopt linrtunner as the linting tool - take 2 (#15085) 2023-03-24 15:29:03 -07:00
upload_python_package_to_azure_storage.py Adopt linrtunner as the linting tool - take 2 (#15085) 2023-03-24 15:29:03 -07:00