mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-29 20:14:01 +00:00
587e891cae
Implement CloudEP for hybrid inferencing. The PR introduces zero new API, customers could configure session and run options to do inferencing with Azure [triton endpoint.](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-with-triton?tabs=azure-cli%2Cendpoint) Sample configuration in python be like: ``` sess_opt.add_session_config_entry('cloud.endpoint_type', 'triton'); sess_opt.add_session_config_entry('cloud.uri', 'https://cloud.com'); sess_opt.add_session_config_entry('cloud.model_name', 'detection2'); sess_opt.add_session_config_entry('cloud.model_version', '7'); // optional, default 1 sess_opt.add_session_config_entry('cloud.verbose', '1'); // optional, default '0', meaning no verbose ... run_opt.add_run_config_entry('use_cloud', '1') # 0 for local inferencing, 1 for cloud endpoint. run_opt.add_run_config_entry('cloud.auth_key', '...') ... sess.run(None, {'input':input_}, run_opt) ``` Co-authored-by: Randy Shuai <rashuai@microsoft.com> |
||
|---|---|---|
| .. | ||
| android-dump-logs-from-steps.yml | ||
| android-java-api-aar-test.yml | ||
| android-java-api-aar.yml | ||
| c-api-artifacts-package-and-publish-steps-posix.yml | ||
| c-api-artifacts-package-and-publish-steps-windows.yml | ||
| c-api-cpu.yml | ||
| c-api-linux-cpu.yml | ||
| clean-agent-build-directory-step.yml | ||
| compliance.yml | ||
| component-governance-component-detection-steps.yml | ||
| download-deps.yml | ||
| esrp_nuget.yml | ||
| explicitly-defined-final-tasks.yml | ||
| get-docker-image-steps.yml | ||
| java-api-artifacts-package-and-publish-steps-posix.yml | ||
| java-api-artifacts-package-and-publish-steps-windows.yml | ||
| linux-ci.yml | ||
| linux-set-variables-and-download.yml | ||
| mac-ci.yml | ||
| mac-esrp-dll.yml | ||
| mac-packaging.yml | ||
| nodejs-artifacts-package-and-publish-steps-posix.yml | ||
| nodejs-artifacts-package-and-publish-steps-windows.yml | ||
| orttraining-linux-gpu-ortmodule-test-ci-pipeline.yml | ||
| publish-nuget.yml | ||
| py-linux-gpu.yml | ||
| py-linux-ubuntu.yml | ||
| py-linux.yml | ||
| py-packaging-linux-test.yml | ||
| py-packaging-selectable-stage.yml | ||
| py-packaging-stage.yml | ||
| py-packaging-training-cuda-stage.yml | ||
| py-win-gpu.yml | ||
| react-native-ci.yml | ||
| rocm.yml | ||
| run-docker-build-steps.yml | ||
| set-nightly-build-option-variable-step.yml | ||
| set-python-manylinux-variables-step.yml | ||
| set-test-data-variables-step.yml | ||
| set-version-number-variables-step.yml | ||
| telemetry-steps.yml | ||
| use-android-ndk.yml | ||
| validate-package.yml | ||
| web-browserstack-ci.yml | ||
| web-ci.yml | ||
| win-ci-vs-2019.yml | ||
| win-ci.yml | ||
| win-esrp-dll.yml | ||
| win-wasm-ci.yml | ||
| win-web-ci.yml | ||
| win-web-multi-browsers.yml | ||
| windows-build-and-test-steps.yml | ||
| windows-build-tools-setup-steps.yml | ||
| with-container-registry-steps.yml | ||