* Initialize tensorrt perf script
* Add bert-squad dependencies
* Modified code to make ort inference with CUDA/Tensorrt
* Add get CUDA/TRT version
* uncomment bert-squad
* Add BERT-SQUAD inputs.json
* Add FastRCNN
* Make preprocess/validation in to common functions
* Add MaskRCNN and SSD and consolidate the code
* Add dependencies for MaskRCNN
* following modifications are made:
- create common fetch function to get inputs/outputs of model from ONNX model zoo.
- create common validation function to compare inference outputs with reference outputs from ONNX model zoo.
- move run/repeat time to argument list. (still working on other arguments, like fp16 or fp32, latency percentile).
- generate table in csv file to show the latency comparison (TRT vs CUDA) side by side.
* Add approache to analyze profling file and also update model related
settings
* Add models
* Add most of models from ONNX model zoo
* Add model input name and print all the model names at the end of run
* Add system info
* Add TRT fp16 support
* Refine the code
* Handle TRT fall back and modify the way to get input data
* Refine code
* Modify code
* Add more precise approach to measure inference
* Add io-binding
* Add YoLoV4
* Refine the code
* Refine the code
* Add models
* Add yolov4 notebook for jetson device
* Update notebook
* Update notebook
* Add CVS models
* Add missing model
* Add support of float16
* Add new way to get trt version
* Add "validate" and "benchmark" mode
* Add randomly generated input
* Refine perf script
* Refine the code.
* Add README
* Refine the code
* Update README.md
* Refine code
* Update README.md
* Remove all the model related python and instead using model_list.json as
models configuration.
Refine the benchmark.py
* Refine the code
Co-authored-by: Chi Lo <lochi@microsoft.com>
* Added config flags for VPU Fast Recompile
* clean-up ifdefs
* Add VPU Fast compile config option
Adds an option that enables Fast compilation of models to VPU
hardware specific format.
* Add config option to choose specific device id for inference
Inference of all subgraphs will be scheduled only on this device
even if other devices of the same type are available.
* Add Python API to list available device IDs
* code cleanup
* Add second C/C++ API with settings string parameter
Adds an additional C/C++ API that allows passing multiple
key-value pairs for settings as a single string. Multiple
settings are delimited by '\n' while the key and value
within a setting are delimited by '|'.
* Append 'Ex' to the extended C/C++ API
* Use set_providers Py API to set config options.
Uses Session.set_providers Python API to set EP runtime config
options as key/val pairs
Deprecated older module function definitions for config settings.
Updates documentation.
* avoid globals for py config options where possible
Co-authored-by: intel <you@example.com>
* change the version check of ort format save/load
* Address PR comments, update the unit test ort models
* Update some variable names to code convention
* Move IsOrtModelVersionSupported inside of #if defined(ENABLE_ORT_FORMAT_LOAD)
(1) Save gpt2 test data during test generation.
(2) Use torch fp32 model as baseline when onnx model is fp16.
(3) Refine logic to compose onnx model path
* add option, feature to orttrainer and test
* address comments
* minor fixes
* further address comments
* minor changes
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* fix hash conflict
* Add verbose for engine deserialization and destroy old engine memory if new engine is generated
* update parser
* Update tensorrt_execution_provider.cc
* use a better hash algorithm
* Update tensorrt_execution_provider.cc
* Remove SparseTensor support from minimal build.
Currently the only valid usage of a SparseTensor is as an attribute of a Constant node. That would have been lifted to a dense tensor initializer when loading the onnx model, so would not exist when saving the ORT format model. Due to that there can be no SparseTensors in an ORT format model.
Co-authored-by: gwang <wanggy@outlook.com>
* Fix places where MinSizeRel wasn't having relevant flags added in the same way as Release and RelWithDebInfo
Enable LTO for minimal build. Cleanups onnx_minimal.cmake to remove some things handled when LTO is enabled in CMakeLists.txt
* Only enable LTO for MSVC in a minimal build
* Nuget store packaging
* Move DNNL workaround to EP
* Fix warning as error
* Disable store tests
* Skip store tests
* msbuild target
* Cross compile protoc in Store
* Disable DML in store
* Move store builds to CPU queue
* Copy uap10 to final nuget
* Fix pip8 error
* Remove extra dml copies
* Fix argparse
* pep8
* Forward IsStoreBuild
* Apply is_store_build to duplicate generate_nuspec
* runtimes
* Refactor uap10
* Store .NET
* uap
* PR feedback
