mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-29 20:14:01 +00:00
Make Perf Test Configurable (#7836)
- Allow anyone to kick off a perf test here. Customize: branch, eps, model selection, cuda version. - Only run shape inference when required. - Kill errored out memory processes. - Remove warmup run. - Clean up script. - Standalone_TRT is it's own "EP" vs as an additional run with TRT EP
This commit is contained in:
Olivia Jain
GitHub
parent
aa68157c3d
commit
b2247ece25
12 changed files with 365 additions and 457 deletions
|
|
@ -1,7 +1,21 @@
|
|||
# TensorRT EP Performance Test Script
|
||||
This script mainly focus on benchmarking ORT TensorRT EP performance compared with CUDA EP and standalone TensorRT. The metrics includes TensorRT EP performance gain, percentage of model operators and execution time that run on TensorRT EP.
|
||||
|
||||
## Usage
|
||||
## Usage with Azure Pipelines
|
||||
|
||||
### Linux GPU TensorRT Perf CI Pipeline
|
||||
- [x] **Build ORT** Build ORT from source. Specify _branch_ variable if not master.
|
||||
- [x] **Post to Dashboard** Post to ONNX Runtime EP Dashboard (No Docker).
|
||||
- [ ] **Run in Docker (CUDA 11.0)** Check to run in CUDA 11.0 vs 10.2 (default).
|
||||
- [ ] **Configure EPs** Choose which EPs to run against. Specify _epList_ variable.
|
||||
- **ModelGroups**: Select which model groups to run. (i.e. selected-models, specify _selected-models_ variable)
|
||||
|
||||
#### Variables (under Advanced Options)
|
||||
- **branch**: (*default: master*) Specified branch to run against.
|
||||
- **epList**: List of EPs to run separated by spaces [from available options](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/tensorrt/perf/benchmark.py#L26) _i.e. "CPUExecutionProvider TensorrtExecutionProvider"_
|
||||
- **selected-models**: Specified path to model json file or model folder if selected-models in ModelGroups.
|
||||
|
||||
## Usage Locally
|
||||
You can use following command to test whether models can be run using TensorRT and run benchmark:
|
||||
```
|
||||
./perf.sh
|
||||
|
|
@ -16,6 +30,7 @@ python3 benchmark.py -r benchmark -i random -t 100
|
|||
```
|
||||
Please note that benchmark_wrapper.py creates one process to execute benchmark.py for every model and every ep, therefore, when process runs into segmentation fault and is forced to exit, the wrapper can catch the error.
|
||||
However, benchmark.py creates only one process to run all the model inferences on all eps, once the process triggers segmentation fault, the whole process is forced to exit and can't successfully capture the error and testing results.
|
||||
|
||||
### Options
|
||||
- **-r, --running_mode**: (*defaul: benchmark*) There are two types of running mode, *validate* and *benchmark*. For validation, this test script records any runtime error as well as validates the accuracy of prediction result using *np.testing.assert_almost_equal()* and exposes result that doesn't meet accuracy requirement. For benchmark, it simply runs model inference assuming model is correct and get the performance metrics. (Note: If you run validation first and then benchmark, test script knows which model has issue and will skip benchmarking of this particular model.)
|
||||
- **-m, --model_source**: (*default: model_list.json*) There are two ways to specify list of models to test. (1) Explicitly specify model list file which contains model information. (2) Specify directory which has following layout:
|
||||
|
|
@ -36,10 +51,12 @@ However, benchmark.py creates only one process to run all the model inferences o
|
|||
```
|
||||
- **-i, --input_data**: (*default: random*) Where is the input data coming from. The value are *zoo* or *random*. The input data can be from ONNX model zoo or it can be randomly generated by test script.
|
||||
- **-t, --test_times**: (*default: 1*) Number of inference run when in 'benchmark' running mode.
|
||||
- **-w, --workspace**: (*default: ./*) Root directory of perf dir. Tensorrt_home should be discoverable from this path.
|
||||
- **-o, --perf_result_path**: (*default: result*) Directory for perf result..
|
||||
- **--fp16**: (*default: True*) Enable TensorRT/CUDA FP16 and include the performance of this floating point optimization.
|
||||
- **--trtexec**: Path of standalone TensorRT executable, for example: trtexec.
|
||||
- **--track_memory**: Track memory usage of CUDA and TensorRT execution providers
|
||||
- **--track_memory**: Track memory usage of CUDA and TensorRT execution providers.
|
||||
- **--ep_list**: Optional argument. List of eps to run against, surround with quotes and separate with spaces.
|
||||
|
||||
### Validation Configuration
|
||||
- **--percent_mismatch**: The allowed percentage of values to be incorrect when comparing given outputs to ORT outputs.
|
||||
|
|
@ -62,84 +79,20 @@ The output of running validation:
|
|||
Total time for running/profiling all models: 0:20:30.761618
|
||||
['bert-squad', 'faster-rcnn', 'mask-rcnn', 'ssd', 'tiny-yolov2', 'resnet152v1']
|
||||
|
||||
Total models: 6
|
||||
Fail models: 2
|
||||
Models FAIL/SUCCESS: 2/4
|
||||
|
||||
============================================
|
||||
========== Failing Models/EPs ==============
|
||||
============================================
|
||||
{'faster-rcnn': ['CUDAExecutionProvider_fp16'], 'mask-rcnn': ['CUDAExecutionProvider_fp16']}
|
||||
|
||||
========================================
|
||||
========== TRT detail metrics ==========
|
||||
========== Models/EPs metrics ==========
|
||||
========================================
|
||||
{ 'BERT-Squad': { 'ratio_of_execution_time_in_trt': 0.9980344366695495,
|
||||
'ratio_of_ops_in_trt': 0.9989451476793249,
|
||||
'total_execution_time': 12719,
|
||||
'total_ops': 948,
|
||||
'total_ops_in_trt': 947,
|
||||
'total_trt_execution_time': 12694},
|
||||
'BERT-Squad (FP16)': { 'ratio_of_execution_time_in_trt': 0.9948146725561744,
|
||||
'ratio_of_ops_in_trt': 0.9989451476793249,
|
||||
'total_execution_time': 5207,
|
||||
'total_ops': 948,
|
||||
'total_ops_in_trt': 947,
|
||||
'total_trt_execution_time': 5180},
|
||||
'FasterRCNN-10': { 'ratio_of_execution_time_in_trt': 0.881433685003768,
|
||||
'ratio_of_ops_in_trt': 0.8637346791636625,
|
||||
'total_execution_time': 106160,
|
||||
'total_ops': 2774,
|
||||
'total_ops_in_trt': 2396,
|
||||
'total_trt_execution_time': 93573},
|
||||
'FasterRCNN-10 (FP16)': { 'ratio_of_execution_time_in_trt': 0.8391227836682785,
|
||||
'total_execution_time': 67623,
|
||||
'total_trt_execution_time': 56744},
|
||||
'MaskRCNN-10': { 'ratio_of_execution_time_in_trt': 0.9084868640292711,
|
||||
'ratio_of_ops_in_trt': 0.8557567917205692,
|
||||
'total_execution_time': 147039,
|
||||
'total_ops': 3092,
|
||||
'total_ops_in_trt': 2646,
|
||||
'total_trt_execution_time': 133583},
|
||||
'MaskRCNN-10 (FP16)': { 'ratio_of_execution_time_in_trt': 0.8537288833951381,
|
||||
'total_execution_time': 87372,
|
||||
'total_trt_execution_time': 74592},
|
||||
'Resnet-152-v1': { 'ratio_of_execution_time_in_trt': 1.0,
|
||||
'ratio_of_ops_in_trt': 1.0,
|
||||
'total_execution_time': 12330,
|
||||
'total_ops': 360,
|
||||
'total_ops_in_trt': 360,
|
||||
'total_trt_execution_time': 12330},
|
||||
'Resnet-152-v1 (FP16)': { 'ratio_of_execution_time_in_trt': 1.0,
|
||||
'ratio_of_ops_in_trt': 1.0,
|
||||
'total_execution_time': 3201,
|
||||
'total_ops': 360,
|
||||
'total_ops_in_trt': 360,
|
||||
'total_trt_execution_time': 3201},
|
||||
'SSD': { 'ratio_of_execution_time_in_trt': 0.6751571867232051,
|
||||
'ratio_of_ops_in_trt': 0.9905660377358491,
|
||||
'total_execution_time': 102585,
|
||||
'total_ops': 212,
|
||||
'total_ops_in_trt': 210,
|
||||
'total_trt_execution_time': 69261},
|
||||
'SSD (FP16)': { 'ratio_of_execution_time_in_trt': 0.38334507797420264,
|
||||
'ratio_of_ops_in_trt': 0.9905660377358491,
|
||||
'total_execution_time': 32639,
|
||||
'total_ops': 212,
|
||||
'total_ops_in_trt': 210,
|
||||
'total_trt_execution_time': 12512},
|
||||
'tiny_yolov2': { 'ratio_of_execution_time_in_trt': 1.0,
|
||||
'ratio_of_ops_in_trt': 1.0,
|
||||
'total_execution_time': 3003,
|
||||
'total_ops': 33,
|
||||
'total_ops_in_trt': 33,
|
||||
'total_trt_execution_time': 3003},
|
||||
'tiny_yolov2 (FP16)': { 'ratio_of_execution_time_in_trt': 1.0,
|
||||
'ratio_of_ops_in_trt': 1.0,
|
||||
'total_execution_time': 864,
|
||||
'total_ops': 33,
|
||||
'total_ops_in_trt': 33,
|
||||
'total_trt_execution_time': 864}}
|
||||
'total_trt_execution_time': 12694}}
|
||||
|
||||
```
|
||||
|
||||
|
|
@ -147,40 +100,33 @@ The output of running benchmark:
|
|||
```
|
||||
|
||||
=========================================
|
||||
=========== CUDA/TRT latency ===========
|
||||
=========== Models/EPs latency ===========
|
||||
=========================================
|
||||
{ 'BERT-Squad': { 'CUDAExecutionProvider': '28.88',
|
||||
'CUDAExecutionProvider_fp16': '18.08',
|
||||
'TensorrtExecutionProvider': '15.55',
|
||||
'TensorrtExecutionProvider_fp16': '5.00',
|
||||
'Tensorrt_fp16_gain(%)': '72.35 %',
|
||||
'Tensorrt_gain(%)': '46.16 %'},
|
||||
'FasterRCNN-10': { 'CUDAExecutionProvider': '161.40',
|
||||
'TensorrtExecutionProvider': '109.24',
|
||||
'TensorrtExecutionProvider_fp16': '66.68',
|
||||
'Tensorrt_gain(%)': '32.32 %'},
|
||||
'MaskRCNN-10': { 'CUDAExecutionProvider': '221.93',
|
||||
'TensorrtExecutionProvider': '154.04',
|
||||
'TensorrtExecutionProvider_fp16': '83.78',
|
||||
'Tensorrt_gain(%)': '30.59 %'},
|
||||
'Resnet-152-v1': { 'CUDAExecutionProvider': '22.55',
|
||||
'CUDAExecutionProvider_fp16': '24.59',
|
||||
'TensorrtExecutionProvider': '9.82',
|
||||
'TensorrtExecutionProvider_fp16': '3.22',
|
||||
'Tensorrt_fp16_gain(%)': '86.91 %',
|
||||
'Tensorrt_gain(%)': '56.45 %'},
|
||||
'SSD': { 'CUDAExecutionProvider': '176.23',
|
||||
'CUDAExecutionProvider_fp16': '82.34',
|
||||
'TensorrtExecutionProvider': '109.34',
|
||||
'TensorrtExecutionProvider_fp16': '40.73',
|
||||
'Tensorrt_fp16_gain(%)': '50.53 %',
|
||||
'Tensorrt_gain(%)': '37.96 %'},
|
||||
'tiny_yolov2': { 'CUDAExecutionProvider': '6.99',
|
||||
'CUDAExecutionProvider_fp16': '5.50',
|
||||
'TensorrtExecutionProvider': '3.15',
|
||||
'TensorrtExecutionProvider_fp16': '1.39',
|
||||
'Tensorrt_fp16_gain(%)': '74.73 %',
|
||||
'Tensorrt_gain(%)': '54.94 %'}}
|
||||
{ 'ResNet101-DUC-7': { 'CPUExecutionProvider': { 'average_latency_ms': '10664.04',
|
||||
'latency_90_percentile': '10923.44'},
|
||||
'CUDAExecutionProvider': { 'average_latency_ms': '406.66',
|
||||
'latency_90_percentile': '412.55',
|
||||
'memory': 13259},
|
||||
'CUDAExecutionProvider_fp16': { 'average_latency_ms': '129.93',
|
||||
'latency_90_percentile': '130.80',
|
||||
'memory': 13003},
|
||||
'Standalone_TRT': { 'average_latency_ms': '365.334 ',
|
||||
'latency_90_percentile': '369.046 ',
|
||||
'memory': 1443},
|
||||
'Standalone_TRT_fp16': { 'average_latency_ms': '105.71 ',
|
||||
'latency_90_percentile': '107.01 ',
|
||||
'memory': 1071},
|
||||
'TRT_CUDA_fp16_gain(%)': '51.54 %',
|
||||
'TRT_CUDA_gain(%)': '14.07 %',
|
||||
'TRT_Standalone_fp16_gain(%)': '40.44 %',
|
||||
'TRT_Standalone_gain(%)': '4.35 %',
|
||||
'TensorrtExecutionProvider': { 'average_latency_ms': '349.43',
|
||||
'latency_90_percentile': '352.53',
|
||||
'memory': 1941},
|
||||
'TensorrtExecutionProvider_fp16': { 'average_latency_ms': '62.96',
|
||||
'latency_90_percentile': '63.96',
|
||||
'memory': 1257}},
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
|
@ -210,27 +156,28 @@ python comparison_script.py -p "prev" -c "current" -o "output.csv"
|
|||
|
||||
### Building ORT Env
|
||||
build_images.sh: This script should be run before running run_perf_docker.sh to make sure the docker images are up to date.
|
||||
- **-o, --ort_dockerfile_path**: Path to ORT Docker File.
|
||||
- **-p, --perf_dockerfile_path**: Path to EP Perf Docker File.
|
||||
- **-b, --branch**: ORT branch name you are perf testing on.
|
||||
- **-i, --image**: What the perf docker image will be named.
|
||||
- **-b, --branch**: ORT branch name you are perf testing on.
|
||||
|
||||
ort_build_latest.py: This script should be run before running run_perf_machine.sh or benchmark.py to make sure the latest ORT wheel file is being used.
|
||||
- **-o, --ort_master_path**: ORT master repo.
|
||||
- **-t, --tensorrt_home**: TensorRT home directory.
|
||||
- **-c, --cuda_home**: CUDA home directory.
|
||||
- **-b, --branch**: (*default: master*) ORT branch name you are perf testing on.
|
||||
|
||||
### Running Perf Script
|
||||
run_perf_docker.sh: Runs the perf script in docker environment.
|
||||
- **-d, --docker_image**: Name of perf docker image.
|
||||
- **-o, --option**: Name of which models you want to run {onnx-zoo-models, many-models, partner-models, selected-models}
|
||||
- **-m, --model_path**: Path to models either json or folder.
|
||||
- **-o, --option**: Name of which models you want to run {i.e. selected-models}
|
||||
- **-p, --perf_dir**: Path to perf directory.
|
||||
- **-m, --model_path**: Model path relative to workspace (/). If option is selected-models, include path to models either json or folder.
|
||||
|
||||
run_perf_machine.sh: Runs the perf script in docker environment.
|
||||
- **-o, --option**: Name of which models you want to run {onnx-zoo-models, many-models, partner-models, selected-models}
|
||||
- **-m, --model_path**: Path to models either json or folder.
|
||||
run_perf_machine.sh: Runs the perf script directly.
|
||||
- **-o, --option**: Name of which models you want to run {i.e. selected-models}
|
||||
- **-m, --model_path**: Model path relative to workspace (~/). If option is selected-models, include path to models either json or folder.
|
||||
|
||||
## Dependencies
|
||||
- When inferencing model using CUDA float16, this script following script to convert nodes in model graph from float32 to float16. It also modifies the converting script a little bit to better cover more model graph conversion.
|
||||
https://github.com/microsoft/onnxconverter-common/blob/master/onnxconverter_common/float16.py
|
||||
|
||||
- For dynamic input shape models, the script runs symbolic shape inference on the model. https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/symbolic_shape_infer.py
|
||||
|
|
|
|||
|
|
@ -54,6 +54,11 @@ LATENCY_FILE = ".latency_map"
|
|||
METRICS_FILE = ".metrics_map"
|
||||
MEMORY_FILE = './temp_memory.csv'
|
||||
|
||||
def split_and_sort_output(string_list):
|
||||
string_list = string_list.split("\n")
|
||||
string_list.sort()
|
||||
return string_list
|
||||
|
||||
def run_trt_standalone(trtexec, model_path, ort_inputs, all_inputs_shape, fp16):
|
||||
logger.info("running standalone trt")
|
||||
model_path = "--onnx=" + model_path
|
||||
|
|
@ -75,12 +80,11 @@ def run_trt_standalone(trtexec, model_path, ort_inputs, all_inputs_shape, fp16):
|
|||
logger.info(shapes_arg)
|
||||
|
||||
result = {}
|
||||
|
||||
if fp16:
|
||||
out = get_output([trtexec, model_path, "--fp16", "--percentile=90", "--explicitBatch", shapes_arg])
|
||||
else:
|
||||
out = get_output([trtexec, model_path, "--percentile=90", "--explicitBatch", shapes_arg])
|
||||
|
||||
command = [trtexec, model_path, "--percentile=90", "--explicitBatch", shapes_arg]
|
||||
if fp16:
|
||||
command.extend(["--fp16"])
|
||||
out = get_output(command)
|
||||
|
||||
tmp = out.split("\n")
|
||||
target_list = []
|
||||
for t in tmp:
|
||||
|
|
@ -103,12 +107,7 @@ def run_trt_standalone(trtexec, model_path, ort_inputs, all_inputs_shape, fp16):
|
|||
logger.info(result)
|
||||
return result
|
||||
|
||||
def get_trtexec_path():
|
||||
trtexec_options = get_output(["find", "/", "-name", "trtexec"])
|
||||
trtexec_path = re.search(r'.*/workspace/.*/bin/trtexec', trtexec_options).group()
|
||||
return trtexec_path
|
||||
|
||||
def get_latency_result(runtimes, batch_size, mem_mb=None):
|
||||
def get_latency_result(runtimes, batch_size):
|
||||
latency_ms = sum(runtimes) / float(len(runtimes)) * 1000.0
|
||||
latency_variance = numpy.var(runtimes, dtype=numpy.float64) * 1000.0
|
||||
throughput = batch_size * (1000.0 / latency_ms)
|
||||
|
|
@ -122,8 +121,6 @@ def get_latency_result(runtimes, batch_size, mem_mb=None):
|
|||
"average_latency_ms": "{:.2f}".format(latency_ms),
|
||||
"QPS": "{:.2f}".format(throughput),
|
||||
}
|
||||
if mem_mb:
|
||||
result.update({"memory":mem_mb})
|
||||
return result
|
||||
|
||||
|
||||
|
|
@ -178,7 +175,6 @@ def get_trtexec_pid(df, python_pid):
|
|||
def get_max_memory(trtexec):
|
||||
df = pd.read_csv(MEMORY_FILE)
|
||||
pid = df['pid'].iloc[0]
|
||||
|
||||
if trtexec:
|
||||
pid = get_trtexec_pid(df, pid)
|
||||
|
||||
|
|
@ -187,14 +183,20 @@ def get_max_memory(trtexec):
|
|||
return max_mem
|
||||
|
||||
def start_memory_tracking():
|
||||
logger.info("starting memory tracking process")
|
||||
p = subprocess.Popen(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv", "-l", "1", "-f", MEMORY_FILE])
|
||||
return p
|
||||
|
||||
def end_memory_tracking(p, trtexec):
|
||||
def end_memory_tracking(p, trtexec, success):
|
||||
logger.info("terminating memory tracking process")
|
||||
p.terminate()
|
||||
p.wait()
|
||||
mem_usage = get_max_memory(trtexec)
|
||||
os.remove(MEMORY_FILE)
|
||||
p.kill()
|
||||
mem_usage = None
|
||||
if success:
|
||||
mem_usage = get_max_memory(trtexec)
|
||||
if os.path.exists(MEMORY_FILE):
|
||||
os.remove(MEMORY_FILE)
|
||||
return mem_usage
|
||||
|
||||
def inference_ort(args, name, session, ep, ort_inputs, result_template, repeat_times, batch_size):
|
||||
|
|
@ -214,30 +216,20 @@ def inference_ort(args, name, session, ep, ort_inputs, result_template, repeat_t
|
|||
logger.info(sess_outputs)
|
||||
|
||||
try:
|
||||
if args.track_memory and track_ep_memory(ep):
|
||||
|
||||
p = start_memory_tracking()
|
||||
runtime = timeit.repeat(lambda: session.run(sess_outputs, sess_inputs), number=1, repeat=repeat_times)
|
||||
mem_usage = end_memory_tracking(p, False)
|
||||
else:
|
||||
runtime = timeit.repeat(lambda: session.run(sess_outputs, sess_inputs), number=1, repeat=repeat_times)
|
||||
|
||||
runtimes += runtime
|
||||
|
||||
runtime = timeit.repeat(lambda: session.run(sess_outputs, sess_inputs), number=1, repeat=repeat_times)
|
||||
runtimes += runtime[1:] # remove warmup
|
||||
|
||||
except Exception as e:
|
||||
logger.error(e)
|
||||
return None
|
||||
|
||||
logger.info(runtimes)
|
||||
runtimes[:] = runtimes[1:]
|
||||
logger.info(runtimes)
|
||||
|
||||
result = {}
|
||||
result.update(result_template)
|
||||
result.update({"io_binding": False})
|
||||
latency_result = get_latency_result(runtimes, batch_size, mem_usage)
|
||||
latency_result = get_latency_result(runtimes, batch_size)
|
||||
result.update(latency_result)
|
||||
logger.info(result)
|
||||
return result
|
||||
|
||||
def inference_ort_and_get_prediction(name, session, ort_inputs):
|
||||
|
|
@ -289,11 +281,8 @@ def get_acl_version():
|
|||
#######################################################################################################################################
|
||||
def load_onnx_model_zoo_test_data(path, all_inputs_shape, data_type="fp32"):
|
||||
logger.info("Parsing test data in {} ...".format(path))
|
||||
p1 = subprocess.Popen(["find", path, "-name", "test_data*", "-type", "d"], stdout=subprocess.PIPE)
|
||||
p2 = subprocess.Popen(["sort"], stdin=p1.stdout, stdout=subprocess.PIPE)
|
||||
stdout, sterr = p2.communicate()
|
||||
stdout = stdout.decode("ascii").strip()
|
||||
test_data_set_dir = stdout.split("\n")
|
||||
output = get_output(["find", path, "-name", "test_data*", "-type", "d"])
|
||||
test_data_set_dir = split_and_sort_output(output)
|
||||
logger.info(test_data_set_dir)
|
||||
|
||||
inputs = []
|
||||
|
|
@ -310,11 +299,8 @@ def load_onnx_model_zoo_test_data(path, all_inputs_shape, data_type="fp32"):
|
|||
os.chdir(test_data_dir)
|
||||
|
||||
# load inputs
|
||||
p1 = subprocess.Popen(["find", ".", "-name", "input*"], stdout=subprocess.PIPE)
|
||||
p2 = subprocess.Popen(["sort"], stdin=p1.stdout, stdout=subprocess.PIPE)
|
||||
stdout, sterr = p2.communicate()
|
||||
stdout = stdout.decode("ascii").strip()
|
||||
input_data = stdout.split("\n")
|
||||
output = get_output(["find", ".", "-name", "input*"])
|
||||
input_data = split_and_sort_output(output)
|
||||
logger.info(input_data)
|
||||
|
||||
input_data_pb = []
|
||||
|
|
@ -322,14 +308,10 @@ def load_onnx_model_zoo_test_data(path, all_inputs_shape, data_type="fp32"):
|
|||
tensor = onnx.TensorProto()
|
||||
with open(data, 'rb') as f:
|
||||
tensor.ParseFromString(f.read())
|
||||
|
||||
tensor_to_array = numpy_helper.to_array(tensor)
|
||||
|
||||
if data_type == "fp16" and tensor_to_array.dtype == np.dtype(np.float32):
|
||||
tensor_to_array = tensor_to_array.astype(np.float16)
|
||||
input_data_pb.append(tensor_to_array)
|
||||
|
||||
# print(np.array(input_data_pb[-1]).shape)
|
||||
if not shape_flag:
|
||||
all_inputs_shape.append(input_data_pb[-1].shape)
|
||||
logger.info(all_inputs_shape[-1])
|
||||
|
|
@ -337,14 +319,11 @@ def load_onnx_model_zoo_test_data(path, all_inputs_shape, data_type="fp32"):
|
|||
logger.info('Loaded {} inputs successfully.'.format(len(inputs)))
|
||||
|
||||
# load outputs
|
||||
p1 = subprocess.Popen(["find", ".", "-name", "output*"], stdout=subprocess.PIPE)
|
||||
p2 = subprocess.Popen(["sort"], stdin=p1.stdout, stdout=subprocess.PIPE)
|
||||
stdout, sterr = p2.communicate()
|
||||
stdout = stdout.decode("ascii").strip()
|
||||
output_data = stdout.split("\n")
|
||||
logger.info(output_data)
|
||||
output = get_output(["find", ".", "-name", "output*"])
|
||||
output_data = split_and_sort_output(output)
|
||||
|
||||
if len(output_data) > 0 and output_data[0] != '':
|
||||
logger.info(output_data)
|
||||
output_data_pb = []
|
||||
for data in output_data:
|
||||
tensor = onnx.TensorProto()
|
||||
|
|
@ -469,7 +448,7 @@ def remove_profiling_files(path):
|
|||
for f in files:
|
||||
if "custom_test_data" in f:
|
||||
continue
|
||||
subprocess.Popen(["sudo","rm","-rf", f], stdout=subprocess.PIPE)
|
||||
subprocess.Popen(["rm","-rf", f], stdout=subprocess.PIPE)
|
||||
|
||||
|
||||
def update_fail_report(fail_results, model, ep, e_type, e):
|
||||
|
|
@ -654,9 +633,11 @@ def get_cuda_version():
|
|||
version = re.search(r'CUDA Version: \d\d\.\d', nvidia_strings).group(0)
|
||||
return version
|
||||
|
||||
def get_trt_version():
|
||||
nvidia_strings = get_output(["dpkg", "-l"])
|
||||
version = re.search(r'nvinfer.*\d\.\d\.\d\-\d', nvidia_strings).group(0)
|
||||
def get_trt_version(workspace):
|
||||
libnvinfer = get_output(["find", workspace, "-name", "libnvinfer.so.*"])
|
||||
nvinfer = re.search(r'.*libnvinfer.so.*', libnvinfer).group(0)
|
||||
trt_strings = get_output(["nm", "-D", nvinfer])
|
||||
version = re.search(r'tensorrt_version.*', trt_strings).group(0)
|
||||
return version
|
||||
|
||||
def get_linux_distro():
|
||||
|
|
@ -694,10 +675,10 @@ def get_gpu_info():
|
|||
infos = re.findall('NVIDIA.*', info)
|
||||
return infos
|
||||
|
||||
def get_system_info():
|
||||
def get_system_info(workspace):
|
||||
info = {}
|
||||
info["cuda"] = get_cuda_version()
|
||||
info["trt"] = get_trt_version()
|
||||
info["trt"] = get_trt_version(workspace)
|
||||
info["linux_distro"] = get_linux_distro()
|
||||
info["cpu_info"] = get_cpu_info()
|
||||
info["gpu_info"] = get_gpu_info()
|
||||
|
|
@ -706,10 +687,8 @@ def get_system_info():
|
|||
return info
|
||||
|
||||
def find_model_path(path):
|
||||
p1 = subprocess.Popen(["find", path, "-name", "*.onnx"], stdout=subprocess.PIPE)
|
||||
stdout, sterr = p1.communicate()
|
||||
stdout = stdout.decode("ascii").strip()
|
||||
model_path = stdout.split("\n")
|
||||
output = get_output(["find", "-L", path, "-name", "*.onnx"])
|
||||
model_path = split_and_sort_output(output)
|
||||
logger.info(model_path)
|
||||
|
||||
if model_path == ['']:
|
||||
|
|
@ -729,23 +708,16 @@ def find_model_path(path):
|
|||
return target_model_path[0]
|
||||
|
||||
def find_model_directory(path):
|
||||
p1 = subprocess.Popen(["find", path, "-maxdepth", "1", "-mindepth", "1", "-name", "*", "-type", "d"], stdout=subprocess.PIPE)
|
||||
stdout, sterr = p1.communicate()
|
||||
stdout = stdout.decode("ascii").strip()
|
||||
model_dir = stdout.split("\n")
|
||||
# print(model_dir)
|
||||
|
||||
output = get_output(["find", "-L", path, "-maxdepth", "1", "-mindepth", "1", "-name", "*", "-type", "d"])
|
||||
model_dir = split_and_sort_output(output)
|
||||
if model_dir == ['']:
|
||||
return None
|
||||
|
||||
return model_dir
|
||||
|
||||
def find_test_data_directory(path):
|
||||
p1 = subprocess.Popen(["find", path, "-maxdepth", "1", "-name", "test_data*", "-type", "d"], stdout=subprocess.PIPE)
|
||||
p2 = subprocess.Popen(["sort"], stdin=p1.stdout, stdout=subprocess.PIPE)
|
||||
stdout, sterr = p2.communicate()
|
||||
stdout = stdout.decode("ascii").strip()
|
||||
test_data_dir = stdout.split("\n")
|
||||
output = get_output(["find", "-L", path, "-maxdepth", "1", "-name", "test_data*", "-type", "d"])
|
||||
test_data_dir = split_and_sort_output(output)
|
||||
logger.info(test_data_dir)
|
||||
|
||||
if test_data_dir == ['']:
|
||||
|
|
@ -783,7 +755,7 @@ def parse_models_info_from_directory(path, models):
|
|||
def parse_models_info_from_file(root_dir, path, models):
|
||||
|
||||
# default working directory
|
||||
root_working_directory = root_dir
|
||||
root_working_directory = root_dir + 'perf/'
|
||||
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
|
|
@ -806,7 +778,7 @@ def parse_models_info_from_file(root_dir, path, models):
|
|||
if os.path.isabs(row['working_directory']):
|
||||
model['working_directory'] = row['working_directory']
|
||||
else:
|
||||
model['working_directory'] = os.path.join(root_working_directory + row['working_directory'])
|
||||
model['working_directory'] = os.path.join(root_working_directory, row['working_directory'])
|
||||
else:
|
||||
logger.error('Model path must be provided in models_info.json')
|
||||
raise
|
||||
|
|
@ -831,15 +803,16 @@ def parse_models_info_from_file(root_dir, path, models):
|
|||
|
||||
|
||||
def convert_model_from_float_to_float16(model_path):
|
||||
# from onnxmltools.utils.float16_converter import convert_float_to_float16
|
||||
from onnxmltools.utils import load_model, save_model
|
||||
from float16 import convert_float_to_float16
|
||||
|
||||
onnx_model = load_model(model_path)
|
||||
new_onnx_model = convert_float_to_float16(onnx_model)
|
||||
save_model(new_onnx_model, 'new_fp16_model_by_trt_perf.onnx')
|
||||
new_model_path = os.path.join(os.getcwd(), "new_fp16_model_by_trt_perf.onnx")
|
||||
if not os.path.exists(new_model_path):
|
||||
onnx_model = load_model(model_path)
|
||||
new_onnx_model = convert_float_to_float16(onnx_model)
|
||||
save_model(new_onnx_model, 'new_fp16_model_by_trt_perf.onnx')
|
||||
|
||||
return os.path.join(os.getcwd(), "new_fp16_model_by_trt_perf.onnx")
|
||||
return new_model_path
|
||||
|
||||
def get_test_data(fp16, test_data_dir, all_inputs_shape):
|
||||
inputs = []
|
||||
|
|
@ -854,25 +827,25 @@ def get_test_data(fp16, test_data_dir, all_inputs_shape):
|
|||
return inputs, ref_outputs
|
||||
|
||||
def create_session(model_path, providers, session_options):
|
||||
|
||||
logger.info(model_path)
|
||||
try:
|
||||
session = onnxruntime.InferenceSession(model_path, providers=providers, sess_options=session_options)
|
||||
|
||||
return session
|
||||
except:
|
||||
logger.info("Use symbolic_shape_infer.py")
|
||||
except Exception as e:
|
||||
if "shape inference" in str(e):
|
||||
logger.info("Using model from symbolic_shape_infer.py")
|
||||
|
||||
new_model_path = model_path[:].replace(".onnx", "_new_by_trt_perf.onnx")
|
||||
exec = os.environ["SYMBOLIC_SHAPE_INFER"]
|
||||
|
||||
new_model_path = model_path[:].replace(".onnx", "_new_by_trt_perf.onnx")
|
||||
exec = os.environ["SYMBOLIC_SHAPE_INFER"]
|
||||
logger.info(exec)
|
||||
|
||||
if not os.path.exists(new_model_path):
|
||||
subprocess.run("python3 " + exec +" --input " + model_path + " --output " + new_model_path + " --auto_merge", shell=True, check=True)
|
||||
session = onnxruntime.InferenceSession(new_model_path, providers=providers, sess_options=session_options)
|
||||
|
||||
return session
|
||||
if not os.path.exists(new_model_path):
|
||||
p = subprocess.run("python3 " + exec + " --input " + model_path + " --output " + new_model_path + " --auto_merge", shell=True, check=True, )
|
||||
logger.info(p)
|
||||
|
||||
session = onnxruntime.InferenceSession(new_model_path, providers=providers, sess_options=session_options)
|
||||
return session
|
||||
else:
|
||||
raise Exception(e)
|
||||
|
||||
def run_onnxruntime(args, models):
|
||||
|
||||
|
|
@ -923,23 +896,25 @@ def run_onnxruntime(args, models):
|
|||
# iterate ep
|
||||
#######################
|
||||
for ep in ep_list:
|
||||
|
||||
if skip_ep(name, ep, model_to_fail_ep):
|
||||
continue
|
||||
|
||||
ep_ = ep_to_provider_list[ep][0]
|
||||
if (ep_ not in onnxruntime.get_available_providers()):
|
||||
logger.error("No {} support".format(ep_))
|
||||
continue
|
||||
|
||||
if standalone_trt not in ep:
|
||||
ep_ = ep_to_provider_list[ep][0]
|
||||
if (ep_ not in onnxruntime.get_available_providers()):
|
||||
logger.error("No {} support".format(ep_))
|
||||
continue
|
||||
|
||||
model_path = model_info["model_path"]
|
||||
test_data_dir = model_info["test_data_path"]
|
||||
|
||||
if ep == cuda_fp16:
|
||||
logger.info("[Initialize] model = {}, ep = {} ,FP16 = True ...".format(name, ep))
|
||||
fp16 = True
|
||||
os.environ["ORT_TENSORRT_FP16_ENABLE"] = "1"
|
||||
|
||||
fp16 = False
|
||||
os.environ["ORT_TENSORRT_FP16_ENABLE"] = "1" if "fp16" in ep else "0"
|
||||
logger.info("[Initialize] model = {}, ep = {} ...".format(name, ep))
|
||||
|
||||
# use float16.py for cuda fp16 only
|
||||
if "cuda_fp16" in ep:
|
||||
|
||||
# handle model
|
||||
if "model_path_fp16" in model_info:
|
||||
model_path = model_info["model_path_fp16"]
|
||||
|
|
@ -947,7 +922,7 @@ def run_onnxruntime(args, models):
|
|||
else:
|
||||
try:
|
||||
model_path = convert_model_from_float_to_float16(model_path)
|
||||
|
||||
fp16 = True
|
||||
except Exception as e:
|
||||
logger.error(e)
|
||||
update_fail_model_map(model_to_fail_ep, name, ep, 'script error', e)
|
||||
|
|
@ -956,23 +931,9 @@ def run_onnxruntime(args, models):
|
|||
# handle test data
|
||||
if "test_data_path_fp16" in model_info:
|
||||
test_data_dir = model_info["test_data_path_fp16"]
|
||||
inputs, ref_outputs = get_test_data(False, test_data_dir, all_inputs_shape)
|
||||
else:
|
||||
inputs, ref_outputs = get_test_data(True, test_data_dir, all_inputs_shape)
|
||||
fp16 = False
|
||||
|
||||
elif ep == trt_fp16:
|
||||
logger.info("[Initialize] model = {}, ep = {} ,FP16 = True ...".format(name, ep))
|
||||
fp16 = True
|
||||
os.environ["ORT_TENSORRT_FP16_ENABLE"] = "1"
|
||||
|
||||
inputs, ref_outputs = get_test_data(False, test_data_dir, all_inputs_shape)
|
||||
else:
|
||||
logger.info("[Initialize] model = {}, ep = {} ,FP16 = False ...".format(name, ep))
|
||||
fp16 = False
|
||||
os.environ["ORT_TENSORRT_FP16_ENABLE"] = "0"
|
||||
|
||||
inputs, ref_outputs = get_test_data(False, test_data_dir, all_inputs_shape)
|
||||
|
||||
inputs, ref_outputs = get_test_data(fp16, test_data_dir, all_inputs_shape)
|
||||
|
||||
# generate random input data
|
||||
if args.input_data == "random":
|
||||
|
|
@ -984,82 +945,94 @@ def run_onnxruntime(args, models):
|
|||
if args.running_mode == 'benchmark':
|
||||
logger.info("\n----------------------------- benchmark -------------------------------------")
|
||||
|
||||
# resolve providers to create session
|
||||
if standalone_trt in ep:
|
||||
providers = ep_to_provider_list[trt]
|
||||
else:
|
||||
providers = ep_to_provider_list[ep]
|
||||
|
||||
options = onnxruntime.SessionOptions()
|
||||
options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
|
||||
|
||||
|
||||
# create onnxruntime inference session
|
||||
try:
|
||||
sess = create_session(model_path, ep_to_provider_list[ep], options)
|
||||
sess = create_session(model_path, providers, options)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(e)
|
||||
update_fail_model_map(model_to_fail_ep, name, ep, 'runtime error', e)
|
||||
continue
|
||||
|
||||
logger.info("start to inference {} with {} ...".format(name, ep))
|
||||
logger.info(sess.get_providers())
|
||||
|
||||
if sess:
|
||||
logger.info("Model inputs nodes:")
|
||||
for input_meta in sess.get_inputs():
|
||||
logger.info(input_meta)
|
||||
logger.info("Model outputs nodes:")
|
||||
for output_meta in sess.get_outputs():
|
||||
logger.info(output_meta)
|
||||
|
||||
batch_size = 1
|
||||
result_template = {
|
||||
"engine": "onnxruntime",
|
||||
"version": onnxruntime.__version__,
|
||||
"device": ep,
|
||||
"fp16": fp16,
|
||||
"io_binding": False,
|
||||
"model_name": name,
|
||||
"inputs": len(sess.get_inputs()),
|
||||
"batch_size": batch_size,
|
||||
"sequence_length": 1,
|
||||
"datetime": str(datetime.now()),}
|
||||
|
||||
if trt in ep and args.trtexec:
|
||||
|
||||
# get standalone TensorRT perf
|
||||
|
||||
# memory tracking variables
|
||||
p = None # keep track of process to kill upon error
|
||||
mem_usage = None
|
||||
|
||||
# get standalone TensorRT perf
|
||||
if standalone_trt in ep and args.trtexec:
|
||||
trtexec = True
|
||||
try:
|
||||
ep = standalone_trt_fp16 if fp16 else standalone_trt
|
||||
|
||||
if args.track_memory:
|
||||
p = start_memory_tracking()
|
||||
result = run_trt_standalone(args.trtexec, model_path, sess.get_inputs(), all_inputs_shape, fp16)
|
||||
mem_usage = end_memory_tracking(p, True)
|
||||
if result and mem_usage:
|
||||
result["memory"] = mem_usage
|
||||
|
||||
mem_usage = end_memory_tracking(p, trtexec, True)
|
||||
else:
|
||||
result = run_trt_standalone(args.trtexec, model_path, sess.get_inputs(), all_inputs_shape, fp16)
|
||||
except Exception as e:
|
||||
logger.error(e)
|
||||
if args.track_memory:
|
||||
end_memory_tracking(p, trtexec, False)
|
||||
update_fail_model_map(model_to_fail_ep, name, ep, 'runtime error', e)
|
||||
continue
|
||||
|
||||
else:
|
||||
result = inference_ort(args, name, sess, ep, inputs, result_template, args.test_times, batch_size)
|
||||
|
||||
if result:
|
||||
# inference with onnxruntime ep
|
||||
else:
|
||||
logger.info("start to inference {} with {} ...".format(name, ep))
|
||||
logger.info(sess.get_providers())
|
||||
|
||||
if sess:
|
||||
logger.info("Model inputs nodes:")
|
||||
for input_meta in sess.get_inputs():
|
||||
logger.info(input_meta)
|
||||
logger.info("Model outputs nodes:")
|
||||
for output_meta in sess.get_outputs():
|
||||
logger.info(output_meta)
|
||||
|
||||
batch_size = 1
|
||||
result_template = {
|
||||
"engine": "onnxruntime",
|
||||
"version": onnxruntime.__version__,
|
||||
"device": ep,
|
||||
"fp16": fp16,
|
||||
"io_binding": False,
|
||||
"model_name": name,
|
||||
"inputs": len(sess.get_inputs()),
|
||||
"batch_size": batch_size,
|
||||
"sequence_length": 1,
|
||||
"datetime": str(datetime.now()),}
|
||||
|
||||
if args.track_memory and track_ep_memory(ep):
|
||||
trtexec = False
|
||||
p = start_memory_tracking()
|
||||
result = inference_ort(args, name, sess, ep, inputs, result_template, args.test_times, batch_size)
|
||||
success = True if result else False
|
||||
mem_usage = end_memory_tracking(p, trtexec, success)
|
||||
else:
|
||||
result = inference_ort(args, name, sess, ep, inputs, result_template, args.test_times, batch_size)
|
||||
if result:
|
||||
latency_result[ep] = {}
|
||||
latency_result[ep]["average_latency_ms"] = result["average_latency_ms"]
|
||||
latency_result[ep]["latency_90_percentile"] = result["latency_90_percentile"]
|
||||
if "memory" in result:
|
||||
mem_usage = result.pop("memory")
|
||||
if mem_usage:
|
||||
latency_result[ep]["memory"] = mem_usage
|
||||
|
||||
if not args.trtexec: # skip standalone
|
||||
success_results.append(result)
|
||||
|
||||
model_to_latency[name] = copy.deepcopy(latency_result)
|
||||
|
||||
logger.info("---------------------------- benchmark [end] ----------------------------------\n")
|
||||
|
||||
|
||||
|
||||
elif args.running_mode == 'validate':
|
||||
logger.info("\n----------------------------- validate -------------------------------------")
|
||||
|
||||
|
|
@ -1072,7 +1045,6 @@ def run_onnxruntime(args, models):
|
|||
# create onnxruntime inference session
|
||||
try:
|
||||
sess = create_session(model_path, ep_to_provider_list[ep], options)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(e)
|
||||
update_fail_model_map(model_to_fail_ep, name, ep, 'runtime error', e)
|
||||
|
|
@ -1179,8 +1151,6 @@ def output_details(results, csv_filename):
|
|||
for result in results:
|
||||
csv_writer.writerow(result)
|
||||
|
||||
logger.info(f"Detail results are saved to csv file: {csv_filename}")
|
||||
|
||||
def output_fail(model_to_fail_ep, csv_filename):
|
||||
|
||||
with open(csv_filename, mode="w", newline='') as csv_file:
|
||||
|
|
@ -1197,8 +1167,6 @@ def output_fail(model_to_fail_ep, csv_filename):
|
|||
result["error type"] = ep_info["error_type"]
|
||||
result["error message"] = ep_info["error_message"]
|
||||
csv_writer.writerow(result)
|
||||
|
||||
logger.info(f"Failing results are saved to csv file: {csv_filename}")
|
||||
|
||||
def read_success_from_file(success_file):
|
||||
success_results = []
|
||||
|
|
@ -1549,8 +1517,6 @@ def parse_arguments():
|
|||
|
||||
parser.add_argument("-c", "--comparison", required=False, default="cuda_trt", choices=["cuda_trt", "acl"], help="EPs to compare: CPU vs. CUDA vs. TRT or CPU vs. ACL")
|
||||
|
||||
parser.add_argument("-d", "--working_dir", required=False, default="./", help="Perf folder path")
|
||||
|
||||
parser.add_argument("-m", "--model_source", required=False, default="model_list.json", help="Model source: (1) model list file (2) model directory.")
|
||||
|
||||
parser.add_argument("-r", "--running_mode", required=False, default="benchmark", choices=["validate", "benchmark"], help="Testing mode.")
|
||||
|
|
@ -1559,9 +1525,13 @@ def parse_arguments():
|
|||
|
||||
parser.add_argument("-o", "--perf_result_path", required=False, default="result", help="Directory for perf result.")
|
||||
|
||||
parser.add_argument("-w", "--workspace", required=False, default="/", help="Workspace to find tensorrt and perf script (with models if parsing with model file)")
|
||||
|
||||
parser.add_argument("--track_memory", required=False, default=True, help="Track CUDA and TRT Memory Usage")
|
||||
|
||||
parser.add_argument("--ep", required=False, default=None, help="Specify ORT Execution Provider.")
|
||||
|
||||
parser.add_argument("--ep_list", nargs="+", required=False, default=None, help="Specify ORT Execution Providers list.")
|
||||
|
||||
parser.add_argument("--fp16", required=False, default=True, action="store_true", help="Inlcude Float16 into benchmarking.")
|
||||
|
||||
|
|
@ -1599,7 +1569,7 @@ def setup_logger(verbose):
|
|||
def parse_models_helper(args, models):
|
||||
if ".json" in args.model_source:
|
||||
logger.info("Parsing model information from file ...")
|
||||
parse_models_info_from_file(args.working_dir, args.model_source, models)
|
||||
parse_models_info_from_file(args.workspace, args.model_source, models)
|
||||
else:
|
||||
logger.info("Parsing model information from directory ...")
|
||||
parse_models_info_from_directory(args.model_source, models)
|
||||
|
|
@ -1608,16 +1578,13 @@ def main():
|
|||
args = parse_arguments()
|
||||
setup_logger(False)
|
||||
pp = pprint.PrettyPrinter(indent=4)
|
||||
|
||||
|
||||
logger.info("\n\nStart perf run ...\n")
|
||||
|
||||
models = {}
|
||||
parse_models_helper(args, models)
|
||||
|
||||
if not os.path.exists("symbolic_shape_infer.py"):
|
||||
p1 = subprocess.Popen(["sudo", "wget", "https://raw.githubusercontent.com/microsoft/onnxruntime/master/onnxruntime/python/tools/symbolic_shape_infer.py"])
|
||||
p1.wait()
|
||||
os.environ["SYMBOLIC_SHAPE_INFER"] = os.path.join(os.getcwd(), "symbolic_shape_infer.py")
|
||||
os.environ["SYMBOLIC_SHAPE_INFER"] = os.path.join(os.getcwd(), "../../symbolic_shape_infer.py")
|
||||
|
||||
perf_start_time = datetime.now()
|
||||
success_results, model_to_latency, model_to_fail_ep, model_to_metrics = run_onnxruntime(args, models)
|
||||
|
|
@ -1659,7 +1626,7 @@ def main():
|
|||
logger.info("=========== Models/EPs latency ===========")
|
||||
logger.info("==========================================")
|
||||
add_improvement_information(model_to_latency)
|
||||
pp.pprint(model_to_latency)
|
||||
pretty_print(pp, model_to_latency)
|
||||
write_map_to_file(model_to_latency, LATENCY_FILE)
|
||||
if args.write_test_result:
|
||||
csv_filename = args.benchmark_latency_csv if args.benchmark_latency_csv else f"benchmark_latency_{time_stamp}.csv"
|
||||
|
|
@ -1675,7 +1642,7 @@ def main():
|
|||
logger.info("\n=========================================")
|
||||
logger.info("========== Models/EPs metrics ==========")
|
||||
logger.info("=========================================")
|
||||
pp.pprint(model_to_metrics)
|
||||
pretty_print(pp, model_to_metrics)
|
||||
write_map_to_file(model_to_metrics, METRICS_FILE)
|
||||
|
||||
if args.write_test_result:
|
||||
|
|
@ -1683,16 +1650,5 @@ def main():
|
|||
csv_filename = os.path.join(path, csv_filename)
|
||||
output_metrics(model_to_metrics, csv_filename)
|
||||
|
||||
if False:
|
||||
logger.info("\n===========================================")
|
||||
logger.info("=========== System information ===========")
|
||||
logger.info("===========================================")
|
||||
info = get_system_info()
|
||||
pp.pprint(info)
|
||||
csv_filename = args.benchmark_fail_csv if args.benchmark_fail_csv else f"system_info_csv{time_stamp}.csv"
|
||||
csv_filename = os.path.join(path, csv_filename)
|
||||
output_system_info(info, csv_filename)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
|
|||
|
|
@ -21,11 +21,26 @@ def get_ep_list(comparison):
|
|||
ep_list = [cpu, cuda, trt, standalone_trt, cuda_fp16, trt_fp16, standalone_trt_fp16]
|
||||
return ep_list
|
||||
|
||||
def resolve_trtexec_path(workspace):
|
||||
trtexec_options = get_output(["find", workspace, "-name", "trtexec"])
|
||||
trtexec_path = re.search(r'.*/bin/trtexec', trtexec_options).group(0)
|
||||
logger.info("using trtexec {}".format(trtexec_path))
|
||||
return trtexec_path
|
||||
|
||||
def main():
|
||||
args = parse_arguments()
|
||||
setup_logger(False)
|
||||
pp = pprint.PrettyPrinter(indent=4)
|
||||
|
||||
# create ep list to iterate through
|
||||
if args.ep_list:
|
||||
ep_list = args.ep_list
|
||||
else:
|
||||
ep_list = get_ep_list(args.comparison)
|
||||
|
||||
if standalone_trt in ep_list or standalone_trt_fp16 in ep_list:
|
||||
trtexec = resolve_trtexec_path(args.workspace)
|
||||
|
||||
models = {}
|
||||
parse_models_helper(args, models)
|
||||
|
||||
|
|
@ -47,11 +62,6 @@ def main():
|
|||
model_list_file = os.path.join(os.getcwd(), model +'.json')
|
||||
write_model_info_to_file([model_info], model_list_file)
|
||||
|
||||
if args.ep:
|
||||
ep_list = [args.ep]
|
||||
else:
|
||||
ep_list = get_ep_list(args.comparison)
|
||||
|
||||
for ep in ep_list:
|
||||
|
||||
command = ["python3",
|
||||
|
|
@ -59,35 +69,32 @@ def main():
|
|||
"-r", args.running_mode,
|
||||
"-m", model_list_file,
|
||||
"-o", args.perf_result_path,
|
||||
"--ep", ep,
|
||||
"--write_test_result", "false"]
|
||||
|
||||
if "Standalone" in ep:
|
||||
if args.running_mode == "validate":
|
||||
continue
|
||||
else:
|
||||
trtexec_path = get_trtexec_path()
|
||||
command.extend(["--trtexec", trtexec_path])
|
||||
ep = trt_fp16 if "fp16" in ep else trt
|
||||
command.extend(["--trtexec", trtexec])
|
||||
|
||||
command.extend(["--ep", ep])
|
||||
|
||||
if args.running_mode == "validate":
|
||||
command.extend(["--benchmark_fail_csv", benchmark_fail_csv,
|
||||
"--benchmark_metrics_csv", benchmark_metrics_csv])
|
||||
command.extend(["--benchmark_metrics_csv", benchmark_metrics_csv])
|
||||
|
||||
elif args.running_mode == "benchmark":
|
||||
command.extend(["-t", str(args.test_times),
|
||||
"-o", args.perf_result_path,
|
||||
"--write_test_result", "false",
|
||||
"--benchmark_fail_csv", benchmark_fail_csv,
|
||||
"--benchmark_latency_csv", benchmark_latency_csv,
|
||||
"--benchmark_success_csv", benchmark_success_csv])
|
||||
|
||||
|
||||
p = subprocess.run(command)
|
||||
logger.info(p)
|
||||
|
||||
if p.returncode != 0:
|
||||
error_type = "runtime error"
|
||||
error_message = "perf script exited with returncode = " + str(p.returncode)
|
||||
error_message = "Benchmark script exited with returncode = " + str(p.returncode)
|
||||
logger.error(error_message)
|
||||
update_fail_model_map(model_to_fail_ep, model, ep, error_type, error_message)
|
||||
write_map_to_file(model_to_fail_ep, FAIL_MODEL_FILE)
|
||||
|
|
@ -101,16 +108,6 @@ def main():
|
|||
Path(path).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if args.running_mode == "validate":
|
||||
logger.info("\n=========================================================")
|
||||
logger.info("========== Failing Models/EPs (accumulated) ==============")
|
||||
logger.info("==========================================================")
|
||||
|
||||
if os.path.exists(FAIL_MODEL_FILE) or len(model_to_fail_ep) > 1:
|
||||
model_to_fail_ep = read_map_from_file(FAIL_MODEL_FILE)
|
||||
output_fail(model_to_fail_ep, os.path.join(path, benchmark_fail_csv))
|
||||
logger.info("\nSaved model fail results to {}".format(benchmark_fail_csv))
|
||||
logger.info(model_to_fail_ep)
|
||||
|
||||
logger.info("\n=========================================")
|
||||
logger.info("=========== Models/EPs metrics ==========")
|
||||
logger.info("=========================================")
|
||||
|
|
@ -121,6 +118,16 @@ def main():
|
|||
logger.info("\nSaved model metrics results to {}".format(benchmark_metrics_csv))
|
||||
|
||||
elif args.running_mode == "benchmark":
|
||||
logger.info("\n=========================================================")
|
||||
logger.info("========== Failing Models/EPs (accumulated) ==============")
|
||||
logger.info("==========================================================")
|
||||
|
||||
if os.path.exists(FAIL_MODEL_FILE) or len(model_to_fail_ep) > 1:
|
||||
model_to_fail_ep = read_map_from_file(FAIL_MODEL_FILE)
|
||||
output_fail(model_to_fail_ep, os.path.join(path, benchmark_fail_csv))
|
||||
logger.info(model_to_fail_ep)
|
||||
logger.info("\nSaved model failing results to {}".format(benchmark_fail_csv))
|
||||
|
||||
logger.info("\n=======================================================")
|
||||
logger.info("=========== Models/EPs Status (accumulated) ===========")
|
||||
logger.info("=======================================================")
|
||||
|
|
@ -156,8 +163,9 @@ def main():
|
|||
logger.info("\n===========================================")
|
||||
logger.info("=========== System information ===========")
|
||||
logger.info("===========================================")
|
||||
info = get_system_info()
|
||||
info = get_system_info(args.workspace)
|
||||
pretty_print(pp, info)
|
||||
logger.info("\n")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ RUN apt-get update &&\
|
|||
RUN unattended-upgrade
|
||||
|
||||
WORKDIR /code
|
||||
ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:/code/cmake-3.14.3-Linux-x86_64/bin:/opt/miniconda/bin:${PATH}
|
||||
ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:/code/cmake-3.18.3-Linux-x86_64/bin:/opt/miniconda/bin:${PATH}
|
||||
ENV LD_LIBRARY_PATH /opt/miniconda/lib:$LD_LIBRARY_PATH
|
||||
|
||||
# Prepare onnxruntime repository & build onnxruntime with TensorRT
|
||||
|
|
@ -28,4 +28,4 @@ RUN git clone --single-branch --branch ${ONNXRUNTIME_BRANCH} --recursive ${ONNXR
|
|||
/bin/sh ./build.sh --parallel --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu/ --use_tensorrt --tensorrt_home /workspace/tensorrt --config Release --build_wheel --update --build --cmake_extra_defines ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) &&\
|
||||
pip install /code/onnxruntime/build/Linux/Release/dist/*.whl &&\
|
||||
cd .. &&\
|
||||
rm -rf onnxruntime cmake-3.14.3-Linux-x86_64
|
||||
rm -rf onnxruntime cmake-3.18.3-Linux-x86_64
|
||||
|
|
|
|||
|
|
@ -13,9 +13,9 @@ apt-get update && apt-get install -y --no-install-recommends \
|
|||
protobuf-compiler \
|
||||
pciutils
|
||||
|
||||
pip install pandas coloredlogs numpy flake8 onnx Cython onnxmltools sympy packaging psutil
|
||||
pip install pandas coloredlogs numpy flake8 onnx Cython onnxmltools sympy packaging psutil mysql-connector-python SQLAlchemy
|
||||
|
||||
# Dependencies: cmake
|
||||
wget --quiet https://github.com/Kitware/CMake/releases/download/v3.14.3/cmake-3.14.3-Linux-x86_64.tar.gz
|
||||
tar zxf cmake-3.14.3-Linux-x86_64.tar.gz
|
||||
rm -rf cmake-3.14.3-Linux-x86_64.tar.gz
|
||||
wget --quiet https://github.com/Kitware/CMake/releases/download/v3.18.3/cmake-3.18.3-Linux-x86_64.tar.gz
|
||||
tar zxf cmake-3.18.3-Linux-x86_64.tar.gz
|
||||
rm -rf cmake-3.18.3-Linux-x86_64.tar.gz
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@ import os
|
|||
import subprocess
|
||||
import argparse
|
||||
import tarfile
|
||||
from perf_utils import get_latest_commit_hash
|
||||
|
||||
def parse_arguments():
|
||||
parser = argparse.ArgumentParser()
|
||||
|
|
@ -10,7 +9,7 @@ def parse_arguments():
|
|||
parser.add_argument("-o", "--ort_master_path", required=True, help="ORT master repo")
|
||||
parser.add_argument("-t", "--tensorrt_home", required=True, help="TensorRT home directory")
|
||||
parser.add_argument("-c", "--cuda_home", required=True, help="CUDA home directory")
|
||||
parser.add_argument("-v", "--commit_hash", required = False, help="Github commit to test perf off of")
|
||||
parser.add_argument("-b", "--branch", required=False, default="master", help="Github branch to test perf off of")
|
||||
parser.add_argument("-s", "--save", required=False, help="Directory to archive wheel file")
|
||||
parser.add_argument("-a", "--use_archived", required=False, help="Archived wheel file")
|
||||
args = parser.parse_args()
|
||||
|
|
@ -19,30 +18,27 @@ def parse_arguments():
|
|||
def archive_wheel_file(save_path, ort_wheel_file):
|
||||
if not os.path.exists(save_path):
|
||||
os.mkdir(save_path)
|
||||
p1 = subprocess.Popen(["cp", ort_wheel_file, save_path])
|
||||
p1.wait()
|
||||
subprocess.run(["cp", ort_wheel_file, save_path], check=True)
|
||||
|
||||
def install_new_ort_wheel(ort_master_path):
|
||||
ort_wheel_path = os.path.join(ort_master_path, "build", "Linux", "Release", "dist")
|
||||
p1 = subprocess.Popen(["find", ort_wheel_path, "-name", "*.whl"], stdout=subprocess.PIPE)
|
||||
stdout, sterr = p1.communicate()
|
||||
stdout = stdout.decode("utf-8").strip()
|
||||
p1 = subprocess.run(["find", ort_wheel_path, "-name", "*.whl"], stdout=subprocess.PIPE, check=True)
|
||||
stdout = p1.stdout.decode("utf-8").strip()
|
||||
ort_wheel = stdout.split("\n")[0]
|
||||
p1 = subprocess.Popen(["pip3", "install", "-I", ort_wheel])
|
||||
p1.wait()
|
||||
subprocess.run(["python3", "-m", "pip", "install", "--force-reinstall", ort_wheel], check=True)
|
||||
return ort_wheel
|
||||
|
||||
def main():
|
||||
args = parse_arguments()
|
||||
|
||||
cmake_tar = "cmake-3.17.4-Linux-x86_64.tar.gz"
|
||||
cmake_tar = "cmake-3.18.4-Linux-x86_64.tar.gz"
|
||||
if not os.path.exists(cmake_tar):
|
||||
p = subprocess.run(["wget", "-c", "https://cmake.org/files/v3.17/" + cmake_tar], check=True)
|
||||
p = subprocess.run(["wget", "-c", "https://cmake.org/files/v3.18/" + cmake_tar], check=True)
|
||||
tar = tarfile.open(cmake_tar)
|
||||
tar.extractall()
|
||||
tar.close()
|
||||
|
||||
os.environ["PATH"] = os.path.join(os.path.abspath("cmake-3.17.4-Linux-x86_64"), "bin") + ":" + os.environ["PATH"]
|
||||
os.environ["PATH"] = os.path.join(os.path.abspath("cmake-3.18.4-Linux-x86_64"), "bin") + ":" + os.environ["PATH"]
|
||||
os.environ["CUDACXX"] = os.path.join(args.cuda_home, "bin", "nvcc")
|
||||
|
||||
ort_master_path = args.ort_master_path
|
||||
|
|
@ -51,26 +47,18 @@ def main():
|
|||
|
||||
if args.use_archived:
|
||||
ort_wheel_file = args.use_archived
|
||||
p1 = subprocess.Popen(["pip3", "install", "-I", ort_wheel_file])
|
||||
p1.wait()
|
||||
subprocess.run(["python3", "-m", "pip", "install", "--force-reinstall", ort_wheel_file], check=True)
|
||||
|
||||
else:
|
||||
if args.commit_hash:
|
||||
commit = args.commit_hash
|
||||
p1 = subprocess.Popen(["git", "checkout", commit])
|
||||
else:
|
||||
commit = get_latest_commit_hash()
|
||||
p1 = subprocess.Popen(["git", "pull", "origin", "master"])
|
||||
p1.wait()
|
||||
|
||||
p1 = subprocess.Popen(["./build.sh", "--config", "Release", "--use_tensorrt", "--tensorrt_home", args.tensorrt_home, "--cuda_home", args.cuda_home, "--cudnn", "/usr/lib/x86_64-linux-gnu", "--build_wheel", "--skip_tests", "--parallel"])
|
||||
p1.wait()
|
||||
subprocess.run(["git", "fetch"], check=True)
|
||||
subprocess.run(["git", "checkout", args.branch], check=True)
|
||||
subprocess.run(["git", "pull", "origin", args.branch], check=True)
|
||||
subprocess.run(["./build.sh", "--config", "Release", "--use_tensorrt", "--tensorrt_home", args.tensorrt_home, "--cuda_home", args.cuda_home, "--cudnn", "/usr/lib/x86_64-linux-gnu", "--build_wheel", "--skip_tests", "--parallel"], check=True)
|
||||
|
||||
ort_wheel_file = install_new_ort_wheel(ort_master_path)
|
||||
|
||||
if args.save:
|
||||
save_path = os.path.join(args.save, commit)
|
||||
archive_wheel_file(save_path, ort_wheel_file)
|
||||
archive_wheel_file(args.save, ort_wheel_file)
|
||||
|
||||
os.chdir(pwd)
|
||||
|
||||
|
|
@ -1,14 +1,30 @@
|
|||
#!/bin/bash
|
||||
|
||||
while getopts d:o:m: parameter
|
||||
while getopts d:o:m:w:e: parameter
|
||||
do case "${parameter}"
|
||||
in
|
||||
in
|
||||
d) PERF_DIR=${OPTARG};;
|
||||
o) OPTION=${OPTARG};;
|
||||
m) MODEL_PATH=${OPTARG};;
|
||||
w) WORKSPACE=${OPTARG};;
|
||||
e) EP_LIST=${OPTARG};;
|
||||
esac
|
||||
done
|
||||
|
||||
# add ep list
|
||||
RUN_EPS=""
|
||||
if [ ! -z "$EP_LIST" ]
|
||||
then
|
||||
RUN_EPS="--ep_list $EP_LIST"
|
||||
fi
|
||||
|
||||
# change dir if docker
|
||||
if [ ! -z $PERF_DIR ]
|
||||
then
|
||||
echo 'changing to '$PERF_DIR
|
||||
cd $PERF_DIR
|
||||
fi
|
||||
|
||||
# metadata
|
||||
FAIL_MODEL_FILE=".fail_model_map"
|
||||
LATENCY_FILE=".latency_map"
|
||||
|
|
@ -16,8 +32,6 @@ METRICS_FILE=".metrics_map"
|
|||
PROFILE="*onnxruntime_profile*"
|
||||
|
||||
# files to download info
|
||||
SYMBOLIC_SHAPE_INFER="symbolic_shape_infer.py"
|
||||
SYMBOLIC_SHAPE_INFER_LINK="https://raw.githubusercontent.com/microsoft/onnxruntime/master/onnxruntime/python/tools/symbolic_shape_infer.py"
|
||||
FLOAT_16="float16.py"
|
||||
FLOAT_16_LINK="https://raw.githubusercontent.com/microsoft/onnxconverter-common/master/onnxconverter_common/float16.py"
|
||||
|
||||
|
|
@ -25,23 +39,20 @@ cleanup_files() {
|
|||
rm -f $FAIL_MODEL_FILE
|
||||
rm -f $LATENCY_FILE
|
||||
rm -f $METRICS_FILE
|
||||
rm -f $SYMBOLIC_SHAPE_INFER
|
||||
rm -f $FLOAT_16
|
||||
rm -rf result/$OPTION
|
||||
find -name $PROFILE -delete
|
||||
}
|
||||
|
||||
download_files() {
|
||||
wget --no-check-certificate -c $SYMBOLIC_SHAPE_INFER_LINK
|
||||
wget --no-check-certificate -c $FLOAT_16_LINK
|
||||
}
|
||||
|
||||
setup() {
|
||||
cd $PERF_DIR
|
||||
cleanup_files
|
||||
download_files
|
||||
}
|
||||
|
||||
setup
|
||||
python3 benchmark_wrapper.py -r validate -m $MODEL_PATH -o result/$OPTION
|
||||
python3 benchmark_wrapper.py -r benchmark -i random -t 10 -m $MODEL_PATH -o result/$OPTION
|
||||
python3 benchmark_wrapper.py -r validate -m $MODEL_PATH -o result/$OPTION -w $WORKSPACE $RUN_EPS
|
||||
python3 benchmark_wrapper.py -r benchmark -t 10 -m $MODEL_PATH -o result/$OPTION -w $WORKSPACE $RUN_EPS
|
||||
|
|
|
|||
|
|
@ -24,10 +24,6 @@ def pretty_print(pp, json_object):
|
|||
pp.pprint(json_object)
|
||||
sys.stdout.flush()
|
||||
|
||||
def get_latest_commit_hash():
|
||||
commit = get_output(["git", "rev-parse", "--short", "HEAD"])
|
||||
return commit
|
||||
|
||||
def parse_single_file(f):
|
||||
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ def insert_latency(commit_hash, report_url, latency):
|
|||
|
||||
# delete old records
|
||||
delete_query = ('DELETE FROM onnxruntime.ep_latency_over_time '
|
||||
'WHERE UploadTime < DATE_SUB(Now(), INTERVAL 30 DAY);'
|
||||
'WHERE UploadTime < DATE_SUB(Now(), INTERVAL 100 DAY);'
|
||||
)
|
||||
|
||||
cursor.execute(delete_query)
|
||||
|
|
@ -144,9 +144,9 @@ def main():
|
|||
cert = get_database_cert()
|
||||
ssl_args = {'ssl_ca': cert}
|
||||
connection_string = sql_connector + \
|
||||
user + \
|
||||
user + ':' + \
|
||||
password + \
|
||||
host + \
|
||||
'@' + host + '/' + \
|
||||
database
|
||||
engine = create_engine(connection_string, connect_args=ssl_args)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,46 +1,21 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Parse Arguments
|
||||
while getopts d:o:m:p: parameter
|
||||
while getopts d:o:m:p:e: parameter
|
||||
do case "${parameter}"
|
||||
in
|
||||
d) DOCKER_IMAGE=${OPTARG};;
|
||||
o) OPTION=${OPTARG};;
|
||||
m) MODEL_PATH=${OPTARG};;
|
||||
p) PERF_DIR=${OPTARG};;
|
||||
e) EP_LIST=${OPTARG};;
|
||||
esac
|
||||
done
|
||||
|
||||
# Variables
|
||||
DOCKER_PERF_DIR=/usr/share/perf/
|
||||
PERF_SCRIPT=$DOCKER_PERF_DIR'perf.sh'
|
||||
VOLUME=$PERF_DIR:$DOCKER_PERF_DIR
|
||||
ONNX_ZOO_VOLUME=' -v /home/hcsuser/perf/models:/usr/share/perf/models'
|
||||
MANY_MODELS_VOLUME=' -v /home/hcsuser/mount/many-models:/usr/share/mount/many-models'
|
||||
PARTNER_VOLUME=' -v /home/hcsuser/perf/partner:/usr/share/perf/partner'
|
||||
DOCKER_PERF_DIR='/perf/'
|
||||
HOME_PERF_DIR='/home/hcsuser/perf/'
|
||||
WORKSPACE='/'
|
||||
MODEL_PATH=$WORKSPACE$MODEL_PATH
|
||||
|
||||
# Add Remaining Variables
|
||||
if [ $OPTION == "onnx-zoo-models" ]
|
||||
then
|
||||
MODEL_PATH='model_list.json'
|
||||
VOLUME=$VOLUME$ONNX_ZOO_VOLUME
|
||||
fi
|
||||
|
||||
if [ $OPTION == "many-models" ]
|
||||
then
|
||||
MODEL_PATH=/usr/share/mount/many-models
|
||||
VOLUME=$VOLUME$MANY_MODELS_VOLUME
|
||||
fi
|
||||
|
||||
if [ $OPTION == "partner-models" ]
|
||||
then
|
||||
MODEL_PATH='partner/partner_model_list.json'
|
||||
VOLUME=$VOLUME$PARTNER_VOLUME
|
||||
fi
|
||||
|
||||
if [ $OPTION == "selected-models" ]
|
||||
then
|
||||
VOLUME=$VOLUME$ONNX_ZOO_VOLUME$MANY_MODELS_VOLUME$PARTNER_VOLUME' -v /home/hcsuser/perf/subset_jsons/:/usr/share/perf/subset_jsons'
|
||||
fi
|
||||
|
||||
sudo docker run --gpus all -v $VOLUME $DOCKER_IMAGE /bin/bash $PERF_SCRIPT -d $DOCKER_PERF_DIR -o $OPTION -m $MODEL_PATH
|
||||
docker run --gpus all -v $PERF_DIR:$DOCKER_PERF_DIR -v $HOME_PERF_DIR$OPTION:$DOCKER_PERF_DIR$OPTION $DOCKER_IMAGE /bin/bash $DOCKER_PERF_DIR'perf.sh' -d $DOCKER_PERF_DIR -o $OPTION -m $MODEL_PATH -w $WORKSPACE -e "$EP_LIST"
|
||||
|
|
|
|||
|
|
@ -1,31 +1,16 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Parse Arguments
|
||||
while getopts o:m: parameter
|
||||
while getopts o:m:e: parameter
|
||||
do case "${parameter}"
|
||||
in
|
||||
o) OPTION=${OPTARG};;
|
||||
m) MODEL_PATH=${OPTARG};;
|
||||
e) EP_LIST=${OPTARG};;
|
||||
esac
|
||||
done
|
||||
|
||||
# Variables
|
||||
PERF_DIR=/home/hcsuser/perf/
|
||||
WORKSPACE=/home/hcsuser/
|
||||
|
||||
# Select models to be tested or run selected-models
|
||||
if [ $OPTION == "onnx-zoo-models" ]
|
||||
then
|
||||
MODEL_PATH='model_list.json'
|
||||
fi
|
||||
|
||||
if [ $OPTION == "many-models" ]
|
||||
then
|
||||
MODEL_PATH=/home/hcsuser/mount/many-models
|
||||
fi
|
||||
|
||||
if [ $OPTION == "partner-models" ]
|
||||
then
|
||||
MODEL_PATH='partner_model_list.json'
|
||||
fi
|
||||
|
||||
./perf.sh -d $PERF_DIR -o $OPTION -m $MODEL_PATH
|
||||
./perf.sh -o $OPTION -m $WORKSPACE$MODEL_PATH -w $WORKSPACE -e "$EP_LIST"
|
||||
|
|
|
|||
|
|
@ -1,43 +1,85 @@
|
|||
jobs:
|
||||
|
||||
parameters:
|
||||
|
||||
- name: BuildORT
|
||||
displayName: Build ORT
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
- name: PostToDashboard
|
||||
displayName: Post to Dashboard
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
- name: RunDocker
|
||||
displayName: Run in Docker (CUDA 11.0)
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
- name: ConfigureEPs
|
||||
displayName: Configure EPs (set epList variable - separate by spaces)
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
- name: ModelGroups
|
||||
type: object
|
||||
default:
|
||||
- "onnx-zoo-models"
|
||||
- "many-models"
|
||||
- "partner-models"
|
||||
|
||||
jobs:
|
||||
- job: Linux_CI_GPU_TENSORRT_PERF
|
||||
pool: Linux-GPU-TensorRT-Perf
|
||||
variables:
|
||||
ALLOW_RELEASED_ONNX_OPSET_ONLY: '1'
|
||||
branch: 'master'
|
||||
timeoutInMinutes: 3000
|
||||
steps:
|
||||
|
||||
- script: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build/build_image.sh -p $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build/Dockerfile.tensorrt-perf -b $(branch) -i ort-$(branch)'
|
||||
displayName: 'Build latest ORT Images'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build'
|
||||
variables:
|
||||
- name: environment
|
||||
${{ if eq(parameters.RunDocker, true) }}:
|
||||
value: docker.sh -d ort-$(branch) -p $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf
|
||||
${{ if ne(parameters.RunDocker, true) }}:
|
||||
value: machine.sh
|
||||
- name: with_arguments
|
||||
value: $(environment) -e "$(epList)"
|
||||
- name: run_conda
|
||||
value: eval "$(command conda 'shell.bash' 'hook' 2> /dev/null)" && conda activate perf3.8
|
||||
timeoutInMinutes: 3000
|
||||
|
||||
steps:
|
||||
|
||||
- script: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/run_perf_docker.sh -d ort-$(branch) -o "onnx-zoo-models" -p $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf'
|
||||
displayName: 'Onnx Zoo Models Perf'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
- ${{ if eq(parameters.BuildORT, true) }}:
|
||||
|
||||
- script: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/run_perf_docker.sh -d ort-$(branch) -o "many-models" -p $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf'
|
||||
displayName: 'Many Models Perf'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
|
||||
- script: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/run_perf_docker.sh -d ort-$(branch) -o "partner-models" -p $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf'
|
||||
displayName: 'Partner Models Perf'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
|
||||
- script: 'mkdir $(Build.SourcesDirectory)/Artifact && cp -r $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/result/ $(Build.SourcesDirectory)/Artifact'
|
||||
displayName: 'Prepare Artifacts'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
|
||||
- task: PublishBuildArtifacts@1
|
||||
inputs:
|
||||
pathtoPublish: '$(Build.SourcesDirectory)/Artifact'
|
||||
artifactName: 'result'
|
||||
- ${{ if eq(parameters.RunDocker, true) }}:
|
||||
- script: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build/build_image.sh -p $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build/Dockerfile.tensorrt-perf -b $(branch) -i ort-$(branch)'
|
||||
displayName: 'Build latest ORT Images'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build'
|
||||
|
||||
- ${{ if eq(parameters.RunDocker, false) }}:
|
||||
- script: '$(run_conda) && python3 $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build/ort_build_latest.py -b $(branch) -c /usr/local/cuda -o ~/repos/onnxruntime/ -t ~/TensorRT-7.2.2.3'
|
||||
displayName: 'Build latest ORT'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/build'
|
||||
|
||||
- ${{ each option in parameters.ModelGroups }}:
|
||||
- script: '$(run_conda) && $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/run_perf_$(with_arguments) -o ${{option}} -m $(${{option}})'
|
||||
displayName: '${{option}} perf'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
env:
|
||||
LD_LIBRARY_PATH : "/home/hcsuser/TensorRT-7.2.2.3/lib"
|
||||
|
||||
- ${{ if not(eq(length(parameters.ModelGroups), 0)) }}:
|
||||
- script: 'mkdir $(Build.SourcesDirectory)/Artifact && cp -r $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/result/ $(Build.SourcesDirectory)/Artifact'
|
||||
displayName: 'Prepare Artifacts'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
- task: PublishBuildArtifacts@1
|
||||
inputs:
|
||||
pathtoPublish: '$(Build.SourcesDirectory)/Artifact'
|
||||
artifactName: 'result'
|
||||
|
||||
|
||||
- script: 'python3 $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/post.py -r $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/result -c $(Build.SourceVersion) -u "https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=$(Build.BuildId)" '
|
||||
displayName: 'Post to Dashboard'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
env:
|
||||
DASHBOARD_MYSQL_ORT_PASSWORD: $(dashboard-mysql-ort-password)
|
||||
- ${{ if eq(parameters.PostToDashboard, true) }}:
|
||||
- script: '$(run_conda) && python3 $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/post.py -r $(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/result -c $(Build.SourceVersion) -u "https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=$(Build.BuildId)" '
|
||||
displayName: 'Post to Dashboard'
|
||||
workingDirectory: '$(Build.SourcesDirectory)/onnxruntime/python/tools/tensorrt/perf/'
|
||||
env:
|
||||
DASHBOARD_MYSQL_ORT_PASSWORD: $(dashboard-mysql-ort-password)
|
||||
|
||||
- script: sudo rm -rf $(Agent.BuildDirectory)
|
||||
displayName: Clean build files (POSIX)
|
||||
|
|
|
|||
Loading…
Reference in a new issue