mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-05-16 21:00:14 +00:00
587e891cae
Implement CloudEP for hybrid inferencing. The PR introduces zero new API, customers could configure session and run options to do inferencing with Azure [triton endpoint.](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-with-triton?tabs=azure-cli%2Cendpoint) Sample configuration in python be like: ``` sess_opt.add_session_config_entry('cloud.endpoint_type', 'triton'); sess_opt.add_session_config_entry('cloud.uri', 'https://cloud.com'); sess_opt.add_session_config_entry('cloud.model_name', 'detection2'); sess_opt.add_session_config_entry('cloud.model_version', '7'); // optional, default 1 sess_opt.add_session_config_entry('cloud.verbose', '1'); // optional, default '0', meaning no verbose ... run_opt.add_run_config_entry('use_cloud', '1') # 0 for local inferencing, 1 for cloud endpoint. run_opt.add_run_config_entry('cloud.auth_key', '...') ... sess.run(None, {'input':input_}, run_opt) ``` Co-authored-by: Randy Shuai <rashuai@microsoft.com>
28 lines
972 B
Bash
Executable file
28 lines
972 B
Bash
Executable file
#!/bin/bash
|
|
set -e -x
|
|
while getopts "x:i:" parameter_Option
|
|
do case "${parameter_Option}"
|
|
in
|
|
i) DOCKER_IMAGE=${OPTARG};;
|
|
x) BUILD_EXTR_PAR=${OPTARG};;
|
|
esac
|
|
done
|
|
|
|
mkdir -p $HOME/.onnx
|
|
docker run --rm \
|
|
--volume /data/onnx:/data/onnx:ro \
|
|
--volume $BUILD_SOURCESDIRECTORY:/onnxruntime_src \
|
|
--volume $BUILD_BINARIESDIRECTORY:/build \
|
|
--volume /data/models:/build/models:ro \
|
|
--volume $HOME/.onnx:/home/onnxruntimedev/.onnx \
|
|
-u root \
|
|
-w /onnxruntime_src \
|
|
-e NIGHTLY_BUILD \
|
|
-e BUILD_BUILDNUMBER \
|
|
$DOCKER_IMAGE tools/ci_build/github/linux/build_linux_arm64_python_package.sh $BUILD_EXTR_PAR
|
|
|
|
sudo rm -rf $BUILD_BINARIESDIRECTORY/Release/onnxruntime $BUILD_BINARIESDIRECTORY/Release/pybind11 \
|
|
$BUILD_BINARIESDIRECTORY/Release/models $BUILD_BINARIESDIRECTORY/Release/_deps \
|
|
$BUILD_BINARIESDIRECTORY/Release/CMakeFiles
|
|
cd $BUILD_BINARIESDIRECTORY/Release
|
|
find -executable -type f > $BUILD_BINARIESDIRECTORY/Release/perms.txt
|