saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-27 20:02:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/onnxruntime/test/run_benchmark.readme.md
Edward Chen 2b3071119a
Add onnxruntime/test/run_benchmark.py helper script. (#19234) 
### Description
Add onnxruntime/test/run_benchmark.py helper script to repeat benchmark
runs until a target coefficient of variance is reached. It works with
[Google Benchmark](https://github.com/google/benchmark) programs like
`onnxruntime_mlas_benchmark`.

### Motivation and Context
Sometimes there is variability in benchmark run results. This automates
the repeated running needed to get results that are stable enough.
2024-04-05 07:02:01 -07:00

975 B
Raw Blame History

run_benchmark.py

run_benchmark.py is a helper script that runs a Google Benchmark program repeatedly until the measurements are within the desired coefficient of variation and then outputs the measurements.

It can be useful for obtaining measurements that are stable enough when repeated invocations of a benchmark program show some measurement variance across runs.

Note that the script runs the benchmark program with specific options and parses specifically formatted output, so it is only expected to work with Google Benchmark programs.

Example usage

To run a benchmark program and get measurements for benchmark test(s) with a particular name:

python run_benchmark.py --program <path to benchmark program, e.g., onnxruntime_mlas_benchmark> --pattern <benchmark test name pattern>

For more detailed usage information, run it with the --help option.