mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-28 20:11:22 +00:00
55c19d6ab5
Enable option qnn_context_priority to set QNN context priority, options: "low", "normal", "normal_high", "high". ### Description Enable option qnn_context_priority to set QNN context priority, options: "low", "normal", "normal_high", "high". This feature guarantees the model inference with higher priority. Tested with onnxruntime_perf_test tool using same model. 1. Run the model on the NPU with single instance, the latency is 300ms. 2. Run the same model on NPU with 2 instance at same time. Case 1: both with same priority (high ) -- latency is 600ms Case 2: 1 with low priority -- latency is 30,000ms 1 with high priority -- latency is 300ms Case 3: 1 with normal priority -- latency is 15,000ms 1 with high priority -- latency is 300ms |
||
|---|---|---|
| .. | ||
| posix | ||
| windows | ||
| command_args_parser.cc | ||
| command_args_parser.h | ||
| main.cc | ||
| ort_test_session.cc | ||
| ort_test_session.h | ||
| performance_runner.cc | ||
| performance_runner.h | ||
| README.md | ||
| ReadMe.txt | ||
| test_configuration.h | ||
| test_session.h | ||
| tf_test_session.h | ||
| TFModelInfo.cc | ||
| TFModelInfo.h | ||
| utils.h | ||
ONNXRuntime Performance Test
This tool provides the performance results using the ONNX Runtime with the specific execution provider to run the inference for a given model using the sample input test data. This tool can provide a reliable measurement for the inference latency usign ONNX Runtime on the device. The options to use with the tool are listed below:
onnxruntime_perf_test [options...] model_path result_file
Options:
-A: Disable memory arena.
-M: Disable memory pattern.
-P: Use parallel executor instead of sequential executor.
-c: [parallel runs]: Specifies the (max) number of runs to invoke simultaneously. Default:1.
-e: [cpu|cuda|mkldnn|tensorrt|openvino|acl|vitisai]: Specifies the execution provider 'cpu','cuda','dnnn','tensorrt', 'openvino', 'acl' and 'vitisai'. Default is 'cpu'.
-m: [test_mode]: Specifies the test mode. Value coulde be 'duration' or 'times'. Provide 'duration' to run the test for a fix duration, and 'times' to repeated for a certain times. Default:'duration'.
-o: [optimization level]: Default is 1. Valid values are 0 (disable), 1 (basic), 2 (extended), 99 (all). Please see __onnxruntime_c_api.h__ (enum GraphOptimizationLevel) for the full list of all optimization levels.
-u: [path to save optimized model]: Default is empty so no optimized model would be saved.
-p: [profile_file]: Specifies the profile name to enable profiling and dump the profile data to the file.
-r: [repeated_times]: Specifies the repeated times if running in 'times' test mode.Default:1000.
-s: Show statistics result, like P75, P90.
-t: [seconds_to_run]: Specifies the seconds to run for 'duration' mode. Default:600.
-v: Show verbose information.
-x: [intra_op_num_threads]: Sets the number of threads used to parallelize the execution within nodes. A value of 0 means the test will auto-select a default. Must >=0.
-y: [inter_op_num_threads]: Sets the number of threads used to parallelize the execution of the graph (across nodes), A value of 0 means the test will auto-select a default. Must >=0.
-h: help.
Model path and input data dependency: Performance test uses the same input structure as onnx_test_runner tool. It requrires the directory trees as below:
--ModelName
--test_data_set_0
--input0.pb
--test_data_set_2
--input0.pb
--model.onnx
The path of model.onnx needs to be provided as <model_path> argument.
Sample output from the tool will look something like this:
Total time cost:58.8053
Total iterations:1000
Average time cost:58.8053 ms
Total run time:58.8102 s
Min Latency is 0.0559777sec
Max Latency is 0.0623472sec
P50 Latency is 0.0587108sec
P90 Latency is 0.0599845sec
P95 Latency is 0.0605676sec
P99 Latency is 0.0619517sec
P999 Latency is 0.0623472se