saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-19 19:00:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/onnxruntime/test/shared_lib/test_inference.cc

3785 lines
159 KiB
C++
Raw Normal View History

Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
#include <memory>
#include <vector>
#include <iostream>
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
#include <fstream>
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
#include <sstream>
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
#include <atomic>
Missing header for `std::once_flag` and `std::call_once`. (#5010)
2020-09-02 07:46:59 +00:00
#include <mutex>
Support more C# APIs (#5608)
2020-10-31 02:19:50 +00:00
#include <algorithm>
Custom thread creation and join hooks (#9426)
2021-11-13 03:10:31 +00:00
#include <thread>
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
Enable Address Sanitizer in CI (#19073) ### Description 1. Add two build jobs for enabling Address Sanitizer in CI. One for Windows CPU, One for Linux CPU. 2. Set default compiler flags/linker flags in build.py for normal Windows/Linux/MacOS build. This can help control compiler flags in a more centralized way. 3. All Windows binaries in our official packages will be built with "/PROFILE" flag. Symbols of onnxruntime.dll can be found at [Microsoft public symbol server](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/microsoft-public-symbols). Limitations: 1. On Linux Address Sanitizer ignores RPATH settings in ELF binaries. Therefore once Address Sanitizer is enabled, before running tests we need to manually set LD_LIBRARY_PATH properly otherwise libonnxruntime.so may not be able to find custom ops and shared EPs. 4. On Linux we also need to set LD_PRELOAD before running some tests(if the main executable, like python, is not built with address sanitizer. On Windows we do not need to. 5. On Windows before running python tests we should manually copy address sanitizer DLL to the onnxruntime/capi directory, because python 3.8 and above has enabled "Safe DLL Search Mode" that wouldn't use the information provided by PATH env. 6. On Linux Address Sanitizer found a lot of memory leaks from our python binding code. Therefore right now we cannot enable Address Sanitizer when building ONNX Runtime with python binding. 7. Address Sanitizer itself uses a lot of memory address space and delays memory deallocations, which is easy to cause OOM issues in 32-bit applications. We cannot run all the tests in onnxruntime_test_all in 32-bit mode with Address Sanitizer due to this reason. However, we still can run individual tests in such a way. We just cannot run all of them in one process. ### Motivation and Context To catch memory issues.
2024-01-12 15:24:40 +00:00
#include <absl/base/config.h>
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
#include "gtest/gtest.h"
#include "gmock/gmock.h"
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
#include "core/common/common.h"
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
#include "core/common/narrow.h"
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
#include "core/graph/constants.h"
#include "core/session/onnxruntime_c_api.h"
#include "core/session/onnxruntime_cxx_api.h"
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
#include "core/session/onnxruntime_lite_custom_op.h"
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
#include "core/session/onnxruntime_session_options_config_keys.h"
Add ability for memory arenas to "shrink" periodically (#7284)
2021-05-08 14:53:21 +00:00
#include "core/session/onnxruntime_run_options_config_keys.h"
Custom thread creation and join hooks (#9426)
2021-11-13 03:10:31 +00:00
#include "core/util/thread_utils.h"
Add GetVersionSting API for C++, C# and Python (#14873) ### Description Added APIs. ### Motivation and Context Addresses https://github.com/microsoft/onnxruntime/issues/14584 Cc: @Craigacp cp
2023-03-03 01:11:07 +00:00
#include "onnxruntime_config.h"
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
#include "providers.h"
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
#include "test_allocator.h"
#include "test_fixture.h"
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
#include "utils.h"
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
#include "custom_op_utils.h"
Switch GSL to MS GSL 4.0.0 (#13416)
2022-10-29 11:15:20 +00:00
#include "core/common/gsl.h"
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
Re-enable custom op shared library test for debug builds (#5475)
2020-10-20 00:14:31 +00:00
#ifdef _WIN32
#include <Windows.h>
#else
#include <dlfcn.h>
#endif
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
#ifdef USE_CUDA
#include <cuda_runtime.h>
#endif
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
// Once we use C++17 this could be replaced with std::size
template <typename T, size_t N>
constexpr size_t countof(T (&)[N]) { return N; }
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
extern std::unique_ptr<Ort::Env> ort_env;
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
template <typename OutT, typename InT = float, typename InputT = Input>
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
void RunSession(OrtAllocator* allocator, Ort::Session& session_object,
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
const std::vector<InputT>& inputs,
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
const char* output_name,
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
const std::vector<int64_t>& dims_y,
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
const std::vector<OutT>& values_y,
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
Ort::Value* output_tensor) {
std::vector<Ort::Value> ort_inputs;
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
std::vector<const char*> input_names;
fix trivial size_t warnings (#843)
2019-04-16 21:37:50 +00:00
for (size_t i = 0; i < inputs.size(); i++) {
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
input_names.emplace_back(inputs[i].name);
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
ort_inputs.emplace_back(
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
Ort::Value::CreateTensor<InT>(allocator->Info(allocator), const_cast<InT*>(inputs[i].values.data()),
inputs[i].values.size(), inputs[i].dims.data(), inputs[i].dims.size()));
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
}
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
std::vector<Ort::Value> ort_outputs;
if (output_tensor)
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
session_object.Run(Ort::RunOptions{nullptr}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
&output_name, output_tensor, 1);
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
else {
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
ort_outputs = session_object.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
&output_name, 1);
Fix some warnings on Windows (#2560) 1. Enable warning "4503" # Decorated name length exceeded. 2. Enable warning "4146" # unary minus operator applied to unsigned type. 3. Enable float64 support for the Softmax operator 4. Enable compliance checks for Windows x86 32bits build 5. Use TryBatchParallelFor to replace some fallback code in mlas pooling.cc 6. Fix Android CI pipeline.
2020-01-22 23:59:11 +00:00
ASSERT_EQ(ort_outputs.size(), 1u);
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
output_tensor = &ort_outputs[0];
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
}
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
auto type_info = output_tensor->GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), dims_y);
size_t total_len = type_info.GetElementCount();
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
ASSERT_EQ(values_y.size(), total_len);
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
OutT* f = output_tensor->GetTensorMutableData<OutT>();
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
for (size_t i = 0; i != total_len; ++i) {
Fix an error in test/shared_lib/test_inference.cc (#16090) ### Description Fix an error in test/shared_lib/test_inference.cc. It should use ASSERT_NEAR to test float values. ### Motivation and Context Our OpenVino pipeline is failing because of this.
2023-05-25 05:59:28 +00:00
if constexpr (std::is_same<OutT, float>::value || std::is_same<OutT, double>::value) {
ASSERT_NEAR(values_y[i], f[i], 1e-3);
} else {
ASSERT_EQ(values_y[i], f[i]);
}
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
}
}
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
template <typename OutT, typename InT = float, typename InputT = Input>
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
static void TestInference(Ort::Env& env, const std::basic_string<ORTCHAR_T>& model_uri,
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
const std::vector<InputT>& inputs,
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
const char* output_name,
const std::vector<int64_t>& expected_dims_y,
const std::vector<OutT>& expected_values_y,
int provider_type,
OrtCustomOpDomain* custom_op_domain_ptr,
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
const ORTCHAR_T* custom_op_library_filename,
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
bool test_session_creation_only = false,
Changes to support standalone custom ops in a minimal build. (#14497) ### Description <!-- Describe your changes. --> Changes to support standalone custom ops in a minimal build. Also incorporates changes from #14492 (needed to test builds prior to that being checked in). We first need to save the schema info from the operators used by the standalone op invoker in the ORT format model. Add mechanism for that. Merge the kernel lookup logic so the same is used in full and minimal build. NOTE: the version matching is now consistent with all other kernel lookups, and the call to CreateOp MUST use the exact version for the operator. Previously matching wasn't as strict, but this can lead to the incorrect kernel being chosen. Add tests. NOTE: There is currently no way to detect the ops/types/opsets used inside these custom ops as they don't exist until we create kernels, which is after model loading completes (which is the point the ORT format model is saved). Due to that they have to be manually added to the configuration used to do the reduced ops build. That shouldn't be too hard for the custom op author to add given the custom op implementation is specifying the op, opset and type constraints (i.e. they have the info and it's just a case of capturing/formatting it correctly). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable usage of the standalone op invoker by custom ops in a minimal build. --------- Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-03-01 01:22:54 +00:00
void* cuda_compute_stream = nullptr,
Ort::SessionOptions* predefined_session_options = nullptr) {
Ort::SessionOptions default_session_options;
Ort::SessionOptions& session_options = predefined_session_options ? *predefined_session_options
: default_session_options;
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
if (provider_type == 1) {
#ifdef USE_CUDA
std::cout << "Running simple inference with cuda provider" << std::endl;
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
auto cuda_options = CreateDefaultOrtCudaProviderOptionsWithCustomStream(cuda_compute_stream);
session_options.AppendExecutionProvider_CUDA(cuda_options);
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
#else
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
ORT_UNUSED_PARAMETER(cuda_compute_stream);
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
return;
#endif
} else if (provider_type == 2) {
Renaming MKL-DNN as DNNL (#2515) * DNNL: Moving Files to rename file names * DNNL name change * azure pipeline updated * disable ceil/dialation and enable Opset10 * disable ceil/dialation tests in Python * mlperf_ssd_resnet34_1200 disabled
2019-12-03 15:34:23 +00:00
#ifdef USE_DNNL
[oneDNN] Improved thread handling (#13618) * Added the OrtDnnlProviderOptions structure to expose configuration options to the user * The number of threads can be defined by the user with the -i flag on the perftest * Number of threads can also be configured via the OMP_NUM_THREADS environment variable * The number of threads defined in the OrtDnnlProviderOptions is prioritized over the environment variable ### Description Avoids thread oversubscription caused by OpenMP allocating the maximum number of threads possible for oneDNN EP. Added support for the OrtDnnlProviderOptions, this will allow for more EP customization capabilities, and allows for user defined number of threads. ### Motivation and Context - Improves performances and allows for user to fine tune the number of threads
2023-01-31 22:37:13 +00:00
OrtDnnlProviderOptions dnnl_options;
dnnl_options.use_arena = 1;
dnnl_options.threadpool_args = nullptr;
session_options.AppendExecutionProvider_Dnnl(dnnl_options);
EP context for custom op (#16454) Implement infrastructures to allow EP resources surfaced to custom ops. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-08-16 20:03:40 +00:00
#else
return;
#endif
} else if (provider_type == 3) {
#ifdef USE_ROCM
OrtROCMProviderOptions rocm_options;
session_options.AppendExecutionProvider_ROCM(rocm_options);
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
#else
return;
#endif
} else {
std::cout << "Running simple inference with default provider" << std::endl;
}
Ryanunderhill/custom op (#550) * Prototype version that demonstrates it can work * Switched to OrtValue and removed the OrtCustomOpTensor code. * Support multiple outputs and reading of attributes * Add custom domain handling to custom ops * Update documentation * more wording changes
2019-03-07 03:09:55 +00:00
if (custom_op_domain_ptr) {
Ryanunderhill/api interface (#1855) * Convert ABI to a versioned interface. * Convert ORT_THROW_ON_ERROR to inline function to fix link errors.
2019-09-20 20:39:11 +00:00
session_options.Add(custom_op_domain_ptr);
Ryanunderhill/custom op (#550) * Prototype version that demonstrates it can work * Switched to OrtValue and removed the OrtCustomOpTensor code. * Support multiple outputs and reading of attributes * Add custom domain handling to custom ops * Update documentation * more wording changes
2019-03-07 03:09:55 +00:00
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
if (custom_op_library_filename) {
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
session_options.RegisterCustomOpsLibrary(custom_op_library_filename);
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
}
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
// if session creation passes, model loads fine
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
Ort::Session session(env, model_uri.c_str(), session_options);
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
// caller wants to test running the model (not just loading the model)
if (!test_session_creation_only) {
// Now run
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto default_allocator = std::make_unique<MockedOrtAllocator>();
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// without preallocated output tensor
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
RunSession<OutT, InT, InputT>(default_allocator.get(),
session,
inputs,
output_name,
expected_dims_y,
expected_values_y,
nullptr);
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// with preallocated output tensor
Introduce output type/shape validation (#17301) ### Description Validate outputs type and shapes. Make sure sparse initializers are taken into account. ### Motivation and Context ORT currently does not validate output types or shapes. Further, neither inputs or outputs take into account sparse initializers that are converted from dense. It is currently possible to pre-allocate a wrong type/shape buffer for output. Cc: @Craigacp
2023-09-05 22:25:12 +00:00
Ort::Value value_y = Ort::Value::CreateTensor<OutT>(default_allocator.get(),
expected_dims_y.data(), expected_dims_y.size());
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// test it twice
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
for (int i = 0; i != 2; ++i)
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
RunSession<OutT, InT, InputT>(default_allocator.get(),
session,
inputs,
output_name,
expected_dims_y,
expected_values_y,
&value_y);
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
}
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
}
Cleanup naming of test input to use .onnx for models. (#1337) * Cleanup naming of test input to use .onnx for models. * Remove file deleted on master
2019-07-04 03:10:29 +00:00
static constexpr PATH_TYPE MODEL_URI = TSTR("testdata/mul_1.onnx");
Add ability for pre-packed weights of shared initializers to be shared across sessions (#7421)
2021-05-15 03:44:42 +00:00
static constexpr PATH_TYPE MATMUL_MODEL_URI = TSTR("testdata/matmul_1.onnx");
[js/react_native] Use a mobile ORT instead of a full ORT (#8042) * Change full ort to mobile ort * Update Android example to load mobile ort * Change the format of test models to ort * update ios to use mobile ort * revise README * use onnxruntime-mobile-c CocoaPods in a npm package
2021-06-15 20:36:05 +00:00
#ifndef ORT_NO_RTTI
Fix Linux/Mac error message on input type mismatch (#6256)
2021-01-06 06:21:24 +00:00
static constexpr PATH_TYPE SEQUENCE_MODEL_URI = TSTR("testdata/sequence_length.onnx");
[js/react_native] Use a mobile ORT instead of a full ORT (#8042) * Change full ort to mobile ort * Update Android example to load mobile ort * Change the format of test models to ort * update ios to use mobile ort * revise README * use onnxruntime-mobile-c CocoaPods in a npm package
2021-06-15 20:36:05 +00:00
#endif
Fix Reduced ops pipeline (#10861)
2022-03-15 03:42:14 +00:00
#if !defined(REDUCED_OPS_BUILD) && defined(USE_CUDA)
Fix bug in MemcpyToHost (#10816)
2022-03-10 15:02:27 +00:00
static constexpr PATH_TYPE SEQUENCE_MODEL_URI_2 = TSTR("testdata/optional_sequence_tensor.onnx");
#endif
Cleanup naming of test input to use .onnx for models. (#1337) * Cleanup naming of test input to use .onnx for models. * Remove file deleted on master
2019-07-04 03:10:29 +00:00
static constexpr PATH_TYPE CUSTOM_OP_MODEL_URI = TSTR("testdata/foo_1.onnx");
Custom op shape inference API (#17737) Add c/cxx API to allow custom ops do shape inference. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-10-13 19:57:42 +00:00
static constexpr PATH_TYPE CUSTOM_OP_LIBRARY_ATTR_TESTER_URI = TSTR("testdata/custom_op_library/attr_tester.onnx");
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
static constexpr PATH_TYPE CUSTOM_OP_LIBRARY_TEST_MODEL_URI = TSTR("testdata/custom_op_library/custom_op_test.onnx");
Patch lite custom op API (#17605) A few enhancements: - Support compute returning status; - Support variadic; --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-09-26 21:02:18 +00:00
static constexpr PATH_TYPE CUSTOM_OP_LIBRARY_COPY_TENSOR_ARRAY_2 = TSTR("testdata/custom_op_library/copy_2_inputs_2_outputs.onnx");
static constexpr PATH_TYPE CUSTOM_OP_LIBRARY_COPY_TENSOR_ARRAY_3 = TSTR("testdata/custom_op_library/copy_3_inputs_3_outputs.onnx");
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
#if !defined(DISABLE_FLOAT8_TYPES)
static constexpr PATH_TYPE CUSTOM_OP_LIBRARY_TEST_MODEL_FLOAT8_URI = TSTR("testdata/custom_op_library/custom_op_test_float8.onnx");
#endif
Custom Op runtime wrapper (#13427) ### Description Adds the below C APIs to support custom ops that wrap an entire model to be inferenced with an external runtime. The current SNPE EP is an example of an EP that could be ported to use a custom op wrapper. Ex: The custom op stores the serialized SNPE DLC binary as a string attribute. The SNPE model is built when the kernel is created. The model is inferenced with SNPE APIs on call to the kernel's compute method. #### C APIs | API | Description | Why | | --- | --- | --- | | `KernelInfo_GetInputCount` | Gets number of inputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputCount` | Gets number of outputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputName` | Gets an input's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputName` | Gets an output's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputTypeInfo` | Gets the type/shape information for an input. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputTypeInfo` | Gets the type/shape information for an output. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfoGetAttribute_tensor` | Get a OrtValue tensor stored as an attribute in the graph node | Extract serialized models, weights, etc. | | `GetSessionConfigEntry` | Get a session configuration value | Need to be able to get session-time configurations from within custom op | | `HasSessionConfigEntry` | Check if session configuration entry exists. | Need to be able to get session-time configurations from within custom op | #### Why so many KernelInfo APIs?<sup>1</sup> Similar APIs currently exist for `OrtKernelContext`, but not `OrtKernelInfo`. Note that `OrtKernelContext` is passed to the custom op on call to its kernel's compute() function. However, `OrtKernelInfo` is available on kernel creation, which occurs when the session is created. Having these APIs available from `OrtKernelInfo` allows an operator to trade-off computation time for session-creation time, and vice versa. Operators that must build expensive state may prefer to do it during session creation time instead of compute-time. SNPE is an example of an EP that needs to be able to query `KernelInfo` for the name, type, and shape of inputs and outputs in order to build the model from the serialized DLC data. This is an expensive operation. Other providers (e.g., OpenVINO) are able to query i/o info from the serialized model, so they do not strictly need these APIs. However, the APIs can still be used to validate the expected I/O characteristics. Additionally, several of our CPU contrib ops currently use the same internal version of these KernelInfo APIs (Ex: [qlinear_softmax](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/quantization/qlinear_softmax.cc#L71)). If custom ops are also meant to be a test bed for future ops, then all custom ops (not just runtime wrappers) would benefit from the addition of these public KernelInfo APIs (IMO). #### Example of usage in a custom OP From `onnxruntime/test/testdata/custom_op_openvino_wrapper_library/openvino_wrapper.h` ```c++ struct CustomOpOpenVINO : Ort::CustomOpBase<CustomOpOpenVINO, KernelOpenVINO> { explicit CustomOpOpenVINO(Ort::ConstSessionOptions session_options); CustomOpOpenVINO(const CustomOpOpenVINO&) = delete; CustomOpOpenVINO& operator=(const CustomOpOpenVINO&) = delete; void* CreateKernel(const OrtApi& api, const OrtKernelInfo* info) const; constexpr const char* GetName() const noexcept { return "OpenVINO_Wrapper"; } constexpr const char* GetExecutionProviderType() const noexcept { return "CPUExecutionProvider"; } // IMPORTANT: In order to wrap a generic runtime-specific model, the custom operator // must have a non-homogeneous variadic input and output. constexpr size_t GetInputTypeCount() const noexcept { return 1; } constexpr size_t GetOutputTypeCount() const noexcept { return 1; } constexpr ONNXTensorElementDataType GetInputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr ONNXTensorElementDataType GetOutputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr OrtCustomOpInputOutputCharacteristic GetInputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr OrtCustomOpInputOutputCharacteristic GetOutputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr bool GetVariadicInputHomogeneity() const noexcept { return false; // heterogenous } constexpr bool GetVariadicOutputHomogeneity() const noexcept { return false; // heterogeneous } std::vector<std::string> GetSessionConfigKeys() const { return {"device_type"}; } private: std::unordered_map<std::string, std::string> session_configs_; }; ``` #### How to create a session: ```c++ Ort::Env env; Ort::SessionOptions session_opts; Ort::CustomOpConfigs custom_op_configs; // Create local session config entries for the custom op. custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU"); // Register custom op library and pass in the custom op configs (optional). session_opts.RegisterCustomOpsLibrary(lib_name, custom_op_configs); Ort::Session session(env, model_path.data(), session_opts); ``` ### Motivation and Context Allows creation of simple "wrapper" EPs outside of the main ORT code base.
2023-01-18 17:09:32 +00:00
#if defined(USE_OPENVINO) && (!defined(ORT_MINIMAL_BUILD) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS))
static constexpr PATH_TYPE CUSTOM_OP_OPENVINO_WRAPPER_LIB_TEST_MODEL_URI = TSTR(
"testdata/custom_op_openvino_wrapper_library/custom_op_mnist_ov_wrapper.onnx");
#endif
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
static constexpr PATH_TYPE OVERRIDABLE_INITIALIZER_MODEL_URI = TSTR("testdata/overridable_initializer.onnx");
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
static constexpr PATH_TYPE NAMED_AND_ANON_DIM_PARAM_URI = TSTR("testdata/capi_symbolic_dims.onnx");
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
static constexpr PATH_TYPE MODEL_WITH_CUSTOM_MODEL_METADATA = TSTR("testdata/model_with_valid_ort_config_json.onnx");
Enable varied input type for custom op (#6066) * allow custom op taking varied types * refactor test case * add test model * refactor test case * enable copy elision * update test case * fix issue in ToString function
2020-12-09 23:10:42 +00:00
static constexpr PATH_TYPE VARIED_INPUT_CUSTOM_OP_MODEL_URI = TSTR("testdata/VariedInputCustomOp.onnx");
Support multiple dynamic inputs in custom ops (#6666)
2021-02-17 04:54:30 +00:00
static constexpr PATH_TYPE VARIED_INPUT_CUSTOM_OP_MODEL_URI_2 = TSTR("testdata/foo_3.onnx");
Support optional inputs/outputs in custom op development (#6727)
2021-03-03 13:59:23 +00:00
static constexpr PATH_TYPE OPTIONAL_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI = TSTR("testdata/foo_bar_1.onnx");
static constexpr PATH_TYPE OPTIONAL_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI_2 = TSTR("testdata/foo_bar_2.onnx");
Support for custom op variadic inputs/outputs (#13946) ### Description Adds support for variadic inputs and outputs to custom operators. ### Motivation and Context Needed for custom ops that wrap external runtimes/models and maybe TensorRT plugins.
2022-12-23 19:41:15 +00:00
static constexpr PATH_TYPE VARIADIC_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI = TSTR("testdata/custom_op_variadic_io.onnx");
static constexpr PATH_TYPE VARIADIC_UNDEF_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI = TSTR(
"testdata/custom_op_variadic_undef_io.onnx");
Support parsing an array of values stored as an attribute in a custom op (#6878)
2021-03-09 07:49:58 +00:00
static constexpr PATH_TYPE CUSTOM_OP_MODEL_WITH_ATTRIBUTES_URI = TSTR("testdata/foo_bar_3.onnx");
Single-schema-multi-kernel (#15184) The PR is to allow custom op of different input types to have same op name in a graph. The idea to go over all ops of same name and merge their input/output types into a type-inference function. With the enhancement, custom op node inside a graph can have same op-type given that the input/output types are different. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-04-27 20:39:59 +00:00
static constexpr PATH_TYPE CUSTOM_OP_SINGLE_SCHEMA_MULTI_KERNEL = TSTR("testdata/custom_op_single_schema_multi_kernel.onnx");
Make SparseTensor infrastructure optional (#8802) Add cmake parameter and #ifdefs to allow for disabling sparse tensor support. This comes with a significant binary size cost so we want to be able to exclude it in a minimal build.
2021-08-27 07:12:26 +00:00
#if !defined(DISABLE_SPARSE_TENSORS)
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
static constexpr PATH_TYPE SPARSE_OUTPUT_MODEL_URI = TSTR("testdata/sparse_initializer_as_output.onnx");
#ifndef DISABLE_CONTRIB_OPS
static constexpr PATH_TYPE SPARSE_INPUT_MATMUL_MODEL_URI = TSTR("testdata/sparse_to_dense_matmul.onnx");
#endif
Make SparseTensor infrastructure optional (#8802) Add cmake parameter and #ifdefs to allow for disabling sparse tensor support. This comes with a significant binary size cost so we want to be able to exclude it in a minimal build.
2021-08-27 07:12:26 +00:00
#endif // !defined(DISABLE_SPARSE_TENSORS)
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
Integrate onnxruntime-extensions into onnxruntime. (#8143) Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
2021-07-01 16:34:03 +00:00
#ifdef ENABLE_EXTENSION_CUSTOM_OPS
static constexpr PATH_TYPE ORT_CUSTOM_OPS_MODEL_URI = TSTR("testdata/custom_op_string_lower.onnx");
static constexpr PATH_TYPE ORT_CUSTOM_OPS_MODEL_URI_2 = TSTR("testdata/custom_op_negpos.onnx");
#endif
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
#ifdef ENABLE_LANGUAGE_INTEROP_OPS
Cleanup naming of test input to use .onnx for models. (#1337) * Cleanup naming of test input to use .onnx for models. * Remove file deleted on master
2019-07-04 03:10:29 +00:00
static constexpr PATH_TYPE PYOP_FLOAT_MODEL_URI = TSTR("testdata/pyop_1.onnx");
EnrichPyOpUT (#4681) * cancel night build on pyop * enrich PyOp UTs * init script only once * remove space * update models * Show usage of kwargs in doc
2020-08-05 21:11:56 +00:00
static constexpr PATH_TYPE PYOP_MULTI_MODEL_URI = TSTR("testdata/pyop_2.onnx");
static constexpr PATH_TYPE PYOP_KWARG_MODEL_URI = TSTR("testdata/pyop_3.onnx");
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
#endif
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
Improve reduced ops and types build (#9908) * Improve reduceops and types build * minor update * fix test error * fix minimal build break * minor update and add comments * Address CR comments
2021-12-07 21:02:05 +00:00
#ifndef REDUCED_OPS_BUILD
Cancel transpose optimizer for resize (#9870) * cancel transpose optimizer for resize * add UT * addressing comments * fix build err
2021-11-30 06:30:16 +00:00
static constexpr PATH_TYPE RESIZE_AND_CROP_MODEL_URI = TSTR("testdata/crop_and_resize.onnx");
Improve reduced ops and types build (#9908) * Improve reduceops and types build * minor update * fix test error * fix minimal build break * minor update and add comments * Address CR comments
2021-12-07 21:02:05 +00:00
#endif
Cancel transpose optimizer for resize (#9870) * cancel transpose optimizer for resize * add UT * addressing comments * fix build err
2021-11-30 06:30:16 +00:00
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
static constexpr PATH_TYPE SIMPLIFIED_SSD_MODEL_URI = TSTR("testdata/multi_stream_models/simplified_ssd.onnx");
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
class CApiTestWithProvider : public testing::Test, public ::testing::WithParamInterface<int> {
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
};
TEST_P(CApiTestWithProvider, simple) {
// simple inference test
// prepare inputs
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
std::vector<Input> inputs(1);
Input& input = inputs.back();
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {3, 2};
std::vector<float> expected_values_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, MODEL_URI, inputs, "Y", expected_dims_y, expected_values_y, GetParam(),
nullptr, nullptr);
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
}
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
TEST(CApiTest, dim_param) {
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
Ort::SessionOptions session_options;
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
Ort::Session session(*ort_env, NAMED_AND_ANON_DIM_PARAM_URI, session_options);
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
auto in0 = session.GetInputTypeInfo(0);
auto in0_ttsi = in0.GetTensorTypeAndShapeInfo();
auto num_input_dims = in0_ttsi.GetDimensionsCount();
Fix some warnings on Windows (#2560) 1. Enable warning "4503" # Decorated name length exceeded. 2. Enable warning "4146" # unary minus operator applied to unsigned type. 3. Enable float64 support for the Softmax operator 4. Enable compliance checks for Windows x86 32bits build 5. Use TryBatchParallelFor to replace some fallback code in mlas pooling.cc 6. Fix Android CI pipeline.
2020-01-22 23:59:11 +00:00
ASSERT_GE(num_input_dims, 1u);
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
// reading 1st dimension only so don't need to malloc int64_t* or const char** values for the Get*Dimensions calls
int64_t dim_value = 0;
const char* dim_param = nullptr;
Deprecate CustomApi and refactor public API for better safety and consistency (#13215) ### Description Deprecate CustomOpApi and refactor dependencies for exception safety and eliminate memory leaks. Refactor API classes for clear ownership and semantics. Introduce `InitProviderOrtApi()` ### Motivation and Context Make public API better and safer. Special note about `Ort::Unowned`. The class suffers from the following problems: 1. It is not able to hold const pointers to the underlying C objects. This forces users to `const_cast` and circumvent constness of the returned object. The user is now able to call mutating interfaces on the object which violates invariants and may be a thread-safety issue. It also enables to take ownership of the pointer and destroy it unintentionally (see examples below). 2. The objects that are unowned cannot be copied and that makes coding inconvenient and at times unsafe. 3. It directly inherits from the type it `unowns`. All of the above creates great conditions for inadvertent unowned object mutations and destructions. Consider the following examples of object slicing, one of them is from a real customer issue and the other one I accidentally coded myself (and I am supposed to know how this works). None of the below can be solved by aftermarket patches and can be hard to diagnose. #### Example 1 slicing of argument ```cpp void SlicingOnArgument(Ort::Value& value) { // This will take possession of the input and if the argument // is Ort::Unowned<Ort::Value> it would again double free the ptr // regardless if it was const or not since we cast it away. Ort::Value output_values[] = {std::move(value)}; } void main() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; // onowned is destroyed when the call returns. SlicingOnArgument(unowned); } ``` #### Example 2 slicing of return value ```cpp // The return will be sliced to Ort::Value that would own and relase (double free the ptr) Ort::Value SlicingOnReturn() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; return unowned; } ```
2022-10-06 21:57:37 +00:00
auto dims = in0_ttsi.GetShape();
if (!dims.empty()) dim_value = dims[0];
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
in0_ttsi.GetSymbolicDimensions(&dim_param, 1);
ASSERT_EQ(dim_value, -1) << "symbolic dimension should be -1";
ASSERT_EQ(strcmp(dim_param, "n"), 0) << "Expected 'n'. Got: " << dim_param;
auto out0 = session.GetOutputTypeInfo(0);
auto out0_ttsi = out0.GetTensorTypeAndShapeInfo();
auto num_output_dims = out0_ttsi.GetDimensionsCount();
Fix some warnings on Windows (#2560) 1. Enable warning "4503" # Decorated name length exceeded. 2. Enable warning "4146" # unary minus operator applied to unsigned type. 3. Enable float64 support for the Softmax operator 4. Enable compliance checks for Windows x86 32bits build 5. Use TryBatchParallelFor to replace some fallback code in mlas pooling.cc 6. Fix Android CI pipeline.
2020-01-22 23:59:11 +00:00
ASSERT_EQ(num_output_dims, 1u);
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
Deprecate CustomApi and refactor public API for better safety and consistency (#13215) ### Description Deprecate CustomOpApi and refactor dependencies for exception safety and eliminate memory leaks. Refactor API classes for clear ownership and semantics. Introduce `InitProviderOrtApi()` ### Motivation and Context Make public API better and safer. Special note about `Ort::Unowned`. The class suffers from the following problems: 1. It is not able to hold const pointers to the underlying C objects. This forces users to `const_cast` and circumvent constness of the returned object. The user is now able to call mutating interfaces on the object which violates invariants and may be a thread-safety issue. It also enables to take ownership of the pointer and destroy it unintentionally (see examples below). 2. The objects that are unowned cannot be copied and that makes coding inconvenient and at times unsafe. 3. It directly inherits from the type it `unowns`. All of the above creates great conditions for inadvertent unowned object mutations and destructions. Consider the following examples of object slicing, one of them is from a real customer issue and the other one I accidentally coded myself (and I am supposed to know how this works). None of the below can be solved by aftermarket patches and can be hard to diagnose. #### Example 1 slicing of argument ```cpp void SlicingOnArgument(Ort::Value& value) { // This will take possession of the input and if the argument // is Ort::Unowned<Ort::Value> it would again double free the ptr // regardless if it was const or not since we cast it away. Ort::Value output_values[] = {std::move(value)}; } void main() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; // onowned is destroyed when the call returns. SlicingOnArgument(unowned); } ``` #### Example 2 slicing of return value ```cpp // The return will be sliced to Ort::Value that would own and relase (double free the ptr) Ort::Value SlicingOnReturn() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; return unowned; } ```
2022-10-06 21:57:37 +00:00
dim_value = 0;
dims = out0_ttsi.GetShape();
if (!dims.empty()) dim_value = dims[0];
Add ability to register custom ops by specifying a function name (#14177) ### Description <!-- Describe your changes. --> Use dlsym/GetProcAddress to lookup a custom ops registration function by name and call it. This will be better on mobile platforms where the custom ops library is linked against, and there isn't necessarily a filesystem that a library path can be loaded from. Alternative is to wire up passing in the address of the function, but that has multiple complications which differ by platform. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable using ort and ort-ext packages on mobile platforms. Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-01-12 05:11:34 +00:00
Add ability to get symbolic dimension info for graph inputs and outputs. (#2051) * Add ability to get symbolic dimension info for graph inputs and outputs. WIP to get initial feedback. * Fix linxu build error. Update C# API and add unit test * Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
2019-10-12 22:46:28 +00:00
out0_ttsi.GetSymbolicDimensions(&dim_param, 1);
ASSERT_EQ(dim_value, -1) << "symbolic dimension should be -1";
ASSERT_EQ(strcmp(dim_param, ""), 0);
}
merge training kernels to master (#2999) * merge training kernels to master * merge training kernels to master * revert two files * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master * merge training kernels to master
2020-02-13 22:52:35 +00:00
INSTANTIATE_TEST_SUITE_P(CApiTestWithProviders,
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
CApiTestWithProvider,
::testing::Values(0, 1, 2, 3, 4));
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
Make SparseTensor infrastructure optional (#8802) Add cmake parameter and #ifdefs to allow for disabling sparse tensor support. This comes with a significant binary size cost so we want to be able to exclude it in a minimal build.
2021-08-27 07:12:26 +00:00
#if !defined(DISABLE_SPARSE_TENSORS)
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
TEST(CApiTest, SparseOutputModel) {
std::vector<int64_t> dense_shape{3, 3};
std::vector<float> values{1.764052391052246, 0.40015721321105957, 0.978738009929657};
std::vector<int64_t> values_shape{3};
std::vector<int64_t> coo_indices{2, 3, 5};
std::vector<int64_t> indices_shape{3};
std::vector<Ort::Value> ort_inputs;
std::vector<const char*> input_names;
const char* const output_names[] = {"values"};
Ort::Session session(*ort_env, SPARSE_OUTPUT_MODEL_URI, Ort::SessionOptions{});
auto ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
Expose an API to query the CUDA compute stream to launch a custom kernel (#9141)
2021-11-09 05:10:34 +00:00
output_names, 1);
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
ASSERT_EQ(ort_outputs.size(), 1U);
const auto& sparse_output = ort_outputs[0];
auto ti = sparse_output.GetTypeInfo();
ASSERT_EQ(ONNX_TYPE_SPARSETENSOR, ti.GetONNXType());
auto tensor_type_shape = ti.GetTensorTypeAndShapeInfo();
ASSERT_EQ(dense_shape, tensor_type_shape.GetShape());
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, tensor_type_shape.GetElementType());
ASSERT_EQ(ORT_SPARSE_COO, sparse_output.GetSparseFormat());
auto values_ts = sparse_output.GetSparseTensorValuesTypeAndShapeInfo();
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, values_ts.GetElementType());
ASSERT_EQ(values_shape, values_ts.GetShape());
const auto* values_fetch = sparse_output.GetSparseTensorValues<float>();
auto val_span = gsl::make_span(values_fetch, values.size());
Switch GSL to MS GSL 4.0.0 (#13416)
2022-10-29 11:15:20 +00:00
ASSERT_TRUE(std::equal(values.cbegin(), values.cend(), val_span.begin(), val_span.end()));
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
auto indices_ts = sparse_output.GetSparseTensorIndicesTypeShapeInfo(ORT_SPARSE_COO_INDICES);
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64, indices_ts.GetElementType());
ASSERT_EQ(indices_shape, indices_ts.GetShape());
size_t num_indices = 0;
const int64_t* indices = sparse_output.GetSparseTensorIndicesData<int64_t>(ORT_SPARSE_COO_INDICES, num_indices);
ASSERT_EQ(num_indices, static_cast<size_t>(indices_shape[0]));
auto ind_span = gsl::make_span(indices, num_indices);
Switch GSL to MS GSL 4.0.0 (#13416)
2022-10-29 11:15:20 +00:00
ASSERT_TRUE(std::equal(coo_indices.cbegin(), coo_indices.cend(), ind_span.begin(), ind_span.end()));
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
}
#ifndef DISABLE_CONTRIB_OPS
TEST(CApiTest, SparseInputModel) {
std::vector<int64_t> common_shape{9, 9}; // inputs and outputs same shape
std::vector<float> A_values{1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0,
10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0,
18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0,
26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0,
34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0,
42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0,
50.0, 51.0, 52.0, 53.0};
// 2 - D index
std::vector<int64_t> indices_shape{gsl::narrow<int64_t>(A_values.size()), 2};
std::vector<int64_t> A_indices{0, 1, 0, 2, 0, 6, 0, 7, 0, 8, 1, 0, 1,
1, 1, 2, 1, 6, 1, 7, 1, 8, 2, 0, 2, 1,
2, 2, 2, 6, 2, 7, 2, 8, 3, 3, 3, 4, 3,
5, 3, 6, 3, 7, 3, 8, 4, 3, 4, 4, 4, 5,
4, 6, 4, 7, 4, 8, 5, 3, 5, 4, 5, 5, 5,
6, 5, 7, 5, 8, 6, 0, 6, 1, 6, 2, 6, 3,
6, 4, 6, 5, 7, 0, 7, 1, 7, 2, 7, 3, 7,
4, 7, 5, 8, 0, 8, 1, 8, 2, 8, 3, 8, 4,
8, 5};
std::vector<float> B_data{0, 1, 2, 0, 0, 0, 3, 4, 5,
6, 7, 8, 0, 0, 0, 9, 10, 11,
12, 13, 14, 0, 0, 0, 15, 16, 17,
0, 0, 0, 18, 19, 20, 21, 22, 23,
0, 0, 0, 24, 25, 26, 27, 28, 29,
0, 0, 0, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 0, 0, 0,
42, 43, 44, 45, 46, 47, 0, 0, 0,
48, 49, 50, 51, 52, 53, 0, 0, 0};
Expose an API to query the CUDA compute stream to launch a custom kernel (#9141)
2021-11-09 05:10:34 +00:00
std::vector<float> Y_result{546, 561, 576, 552, 564, 576, 39, 42, 45,
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
1410, 1461, 1512, 1362, 1392, 1422, 201, 222, 243,
2274, 2361, 2448, 2172, 2220, 2268, 363, 402, 441,
2784, 2850, 2916, 4362, 4485, 4608, 1551, 1608, 1665,
3540, 3624, 3708, 5604, 5763, 5922, 2037, 2112, 2187,
4296, 4398, 4500, 6846, 7041, 7236, 2523, 2616, 2709,
678, 789, 900, 2892, 3012, 3132, 4263, 4494, 4725,
786, 915, 1044, 3324, 3462, 3600, 4911, 5178, 5445,
894, 1041, 1188, 3756, 3912, 4068, 5559, 5862, 6165};
Expose an API to query the CUDA compute stream to launch a custom kernel (#9141)
2021-11-09 05:10:34 +00:00
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
Ort::Value::Shape ort_dense_shape{common_shape.data(), common_shape.size()};
Ort::Value::Shape ort_values_shape{&indices_shape[0], 1U};
auto a_st = Ort::Value::CreateSparseTensor(info, A_values.data(), ort_dense_shape, ort_values_shape);
a_st.UseCooIndices(A_indices.data(), A_indices.size());
auto b_tensor = Ort::Value::CreateTensor(info, B_data.data(), B_data.size(), common_shape.data(), common_shape.size());
std::vector<Ort::Value> ort_inputs;
ort_inputs.push_back(std::move(a_st));
ort_inputs.push_back(std::move(b_tensor));
const char* input_names[] = {"sparse_A", "dense_B"};
const char* const output_names[] = {"dense_Y"};
Ort::Session session(*ort_env, SPARSE_INPUT_MATMUL_MODEL_URI, Ort::SessionOptions{});
auto ort_outputs = session.Run(Ort::RunOptions{}, input_names, ort_inputs.data(), ort_inputs.size(),
output_names, 1);
ASSERT_EQ(ort_outputs.size(), 1U);
const auto& dense_Y = ort_outputs[0];
ASSERT_TRUE(dense_Y.IsTensor());
auto result_ts = dense_Y.GetTensorTypeAndShapeInfo();
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, result_ts.GetElementType());
ASSERT_EQ(common_shape, result_ts.GetShape());
const auto* result_vals = dense_Y.GetTensorData<float>();
auto result_span = gsl::make_span(result_vals, Y_result.size());
Switch GSL to MS GSL 4.0.0 (#13416)
2022-10-29 11:15:20 +00:00
ASSERT_TRUE(std::equal(Y_result.cbegin(), Y_result.cend(), result_span.begin(), result_span.end()));
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
}
Expose an API to query the CUDA compute stream to launch a custom kernel (#9141)
2021-11-09 05:10:34 +00:00
#endif // DISABLE_CONTRIB_OPS
#endif // !defined(DISABLE_SPARSE_TENSORS)
Introduce C and C++ APIs for Sparse Tensors (#8621) Add IsSparseTensor Add CreateSparseTensor Add utilities and test fully sparse instantiation Fully sparse blocksparse Add test and docs for fully sparse tensor instantiation Rework creation API Use API Non string API Retrofit of existing String API Add tests Add documentation Address build issues (Winml pending) Add inference test Bump binary size Add ifdef DISABLE CONTRIB
2021-08-16 23:33:47 +00:00
Enable Address Sanitizer in CI (#19073) ### Description 1. Add two build jobs for enabling Address Sanitizer in CI. One for Windows CPU, One for Linux CPU. 2. Set default compiler flags/linker flags in build.py for normal Windows/Linux/MacOS build. This can help control compiler flags in a more centralized way. 3. All Windows binaries in our official packages will be built with "/PROFILE" flag. Symbols of onnxruntime.dll can be found at [Microsoft public symbol server](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/microsoft-public-symbols). Limitations: 1. On Linux Address Sanitizer ignores RPATH settings in ELF binaries. Therefore once Address Sanitizer is enabled, before running tests we need to manually set LD_LIBRARY_PATH properly otherwise libonnxruntime.so may not be able to find custom ops and shared EPs. 4. On Linux we also need to set LD_PRELOAD before running some tests(if the main executable, like python, is not built with address sanitizer. On Windows we do not need to. 5. On Windows before running python tests we should manually copy address sanitizer DLL to the onnxruntime/capi directory, because python 3.8 and above has enabled "Safe DLL Search Mode" that wouldn't use the information provided by PATH env. 6. On Linux Address Sanitizer found a lot of memory leaks from our python binding code. Therefore right now we cannot enable Address Sanitizer when building ONNX Runtime with python binding. 7. Address Sanitizer itself uses a lot of memory address space and delays memory deallocations, which is easy to cause OOM issues in 32-bit applications. We cannot run all the tests in onnxruntime_test_all in 32-bit mode with Address Sanitizer due to this reason. However, we still can run individual tests in such a way. We just cannot run all of them in one process. ### Motivation and Context To catch memory issues.
2024-01-12 15:24:40 +00:00
// Memory leak
#ifndef ABSL_HAVE_ADDRESS_SANITIZER
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
TEST(CApiTest, custom_op_handler) {
Ryanunderhill/custom op (#550) * Prototype version that demonstrates it can work * Switched to OrtValue and removed the OrtCustomOpTensor code. * Support multiple outputs and reading of attributes * Add custom domain handling to custom ops * Update documentation * more wording changes
2019-03-07 03:09:55 +00:00
std::cout << "Running custom op inference" << std::endl;
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
std::vector<Input> inputs(1);
Input& input = inputs[0];
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
Ryanunderhill/custom op (#550) * Prototype version that demonstrates it can work * Switched to OrtValue and removed the OrtCustomOpTensor code. * Support multiple outputs and reading of attributes * Add custom domain handling to custom ops * Update documentation * more wording changes
2019-03-07 03:09:55 +00:00
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {3, 2};
std::vector<float> expected_values_y = {2.0f, 4.0f, 6.0f, 8.0f, 10.0f, 12.0f};
Allow a custom op with the same name to be registered for several providers. (#3400)
2020-04-02 22:38:51 +00:00
#ifdef USE_CUDA
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
cudaStream_t compute_stream = nullptr;
cudaStreamCreateWithFlags(&compute_stream, cudaStreamNonBlocking);
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
MyCustomOp custom_op{onnxruntime::kCudaExecutionProvider};
Allow a custom op with the same name to be registered for several providers. (#3400)
2020-04-02 22:38:51 +00:00
#else
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
MyCustomOp custom_op{onnxruntime::kCpuExecutionProvider};
Allow a custom op with the same name to be registered for several providers. (#3400)
2020-04-02 22:38:51 +00:00
#endif
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
Ort::CustomOpDomain custom_op_domain("test");
More C++ API improvements and conversions (#998) * More C++ API improvements and conversions * Mark more constructors as explicit * Fix CSharp function name changes * Change more test cases to use C++ API
2019-05-13 20:56:54 +00:00
custom_op_domain.Add(&custom_op);
Ryanunderhill/custom op (#550) * Prototype version that demonstrates it can work * Switched to OrtValue and removed the OrtCustomOpTensor code. * Support multiple outputs and reading of attributes * Add custom domain handling to custom ops * Update documentation * more wording changes
2019-03-07 03:09:55 +00:00
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
#ifdef USE_CUDA
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, CUSTOM_OP_MODEL_URI, inputs, "Y", expected_dims_y, expected_values_y, 1,
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
custom_op_domain, nullptr, false, compute_stream);
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
cudaStreamDestroy(compute_stream);
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
#else
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, CUSTOM_OP_MODEL_URI, inputs, "Y", expected_dims_y, expected_values_y, 0,
custom_op_domain, nullptr);
Support custom ops targeting the CUDA EP (#3165) * Initial commit * Minor nit * Comment * Fix build * Fix build
2020-03-11 07:49:01 +00:00
#endif
Ryanunderhill/custom op (#550) * Prototype version that demonstrates it can work * Switched to OrtValue and removed the OrtCustomOpTensor code. * Support multiple outputs and reading of attributes * Add custom domain handling to custom ops * Update documentation * more wording changes
2019-03-07 03:09:55 +00:00
}
Enable Address Sanitizer in CI (#19073) ### Description 1. Add two build jobs for enabling Address Sanitizer in CI. One for Windows CPU, One for Linux CPU. 2. Set default compiler flags/linker flags in build.py for normal Windows/Linux/MacOS build. This can help control compiler flags in a more centralized way. 3. All Windows binaries in our official packages will be built with "/PROFILE" flag. Symbols of onnxruntime.dll can be found at [Microsoft public symbol server](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/microsoft-public-symbols). Limitations: 1. On Linux Address Sanitizer ignores RPATH settings in ELF binaries. Therefore once Address Sanitizer is enabled, before running tests we need to manually set LD_LIBRARY_PATH properly otherwise libonnxruntime.so may not be able to find custom ops and shared EPs. 4. On Linux we also need to set LD_PRELOAD before running some tests(if the main executable, like python, is not built with address sanitizer. On Windows we do not need to. 5. On Windows before running python tests we should manually copy address sanitizer DLL to the onnxruntime/capi directory, because python 3.8 and above has enabled "Safe DLL Search Mode" that wouldn't use the information provided by PATH env. 6. On Linux Address Sanitizer found a lot of memory leaks from our python binding code. Therefore right now we cannot enable Address Sanitizer when building ONNX Runtime with python binding. 7. Address Sanitizer itself uses a lot of memory address space and delays memory deallocations, which is easy to cause OOM issues in 32-bit applications. We cannot run all the tests in onnxruntime_test_all in 32-bit mode with Address Sanitizer due to this reason. However, we still can run individual tests in such a way. We just cannot run all of them in one process. ### Motivation and Context To catch memory issues.
2024-01-12 15:24:40 +00:00
#endif
Ryanunderhill/custom op (#550) * Prototype version that demonstrates it can work * Switched to OrtValue and removed the OrtCustomOpTensor code. * Support multiple outputs and reading of attributes * Add custom domain handling to custom ops * Update documentation * more wording changes
2019-03-07 03:09:55 +00:00
Allow custom ops to set input memory type (#10879)
2022-10-29 04:45:26 +00:00
#ifdef USE_CUDA
TEST(CApiTest, custom_op_set_input_memory_type) {
std::cout << "Running custom op inference" << std::endl;
std::vector<Input> inputs(1);
Input& input = inputs[0];
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {3, 2};
std::vector<float> expected_values_y = {2.0f, 4.0f, 6.0f, 8.0f, 10.0f, 12.0f};
cudaStream_t compute_stream = nullptr;
cudaStreamCreateWithFlags(&compute_stream, cudaStreamNonBlocking);
MyCustomOpSecondInputOnCpu custom_op{onnxruntime::kCudaExecutionProvider, compute_stream};
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
Ort::CustomOpDomain custom_op_domain("test");
Allow custom ops to set input memory type (#10879)
2022-10-29 04:45:26 +00:00
custom_op_domain.Add(&custom_op);
auto x_mem_type = custom_op.GetInputMemoryType(0);
auto y_mem_type = custom_op.GetInputMemoryType(1);
ASSERT_EQ(x_mem_type, OrtMemType::OrtMemTypeDefault);
ASSERT_EQ(y_mem_type, OrtMemType::OrtMemTypeCPUInput);
TestInference<float>(*ort_env, CUSTOM_OP_MODEL_URI, inputs, "Y", expected_dims_y, expected_values_y, 1,
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
custom_op_domain, nullptr, false, compute_stream);
Allow custom ops to set input memory type (#10879)
2022-10-29 04:45:26 +00:00
cudaStreamDestroy(compute_stream);
}
#endif
Changes to support standalone custom ops in a minimal build. (#14497) ### Description <!-- Describe your changes. --> Changes to support standalone custom ops in a minimal build. Also incorporates changes from #14492 (needed to test builds prior to that being checked in). We first need to save the schema info from the operators used by the standalone op invoker in the ORT format model. Add mechanism for that. Merge the kernel lookup logic so the same is used in full and minimal build. NOTE: the version matching is now consistent with all other kernel lookups, and the call to CreateOp MUST use the exact version for the operator. Previously matching wasn't as strict, but this can lead to the incorrect kernel being chosen. Add tests. NOTE: There is currently no way to detect the ops/types/opsets used inside these custom ops as they don't exist until we create kernels, which is after model loading completes (which is the point the ORT format model is saved). Due to that they have to be manually added to the configuration used to do the reduced ops build. That shouldn't be too hard for the custom op author to add given the custom op implementation is specifying the op, opset and type constraints (i.e. they have the info and it's just a case of capturing/formatting it correctly). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable usage of the standalone op invoker by custom ops in a minimal build. --------- Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-03-01 01:22:54 +00:00
#if !defined(ORT_MINIMAL_BUILD)
TEST(CApiTest, StandaloneOpHandler) {
APIs for custom op to invoke ort operator directly (#10713) * draft kernel creation * setup eager context * call into kernel in eager mode * redefine test case * refact eager context * add comment * remove header * rename argument * redefine API definition with types * list outputs as argument * switch to int to represent length * fix compile err * create attribute API * add test case for topk * remove bool from c api * add gru test case * remove var * fix compile warnings * rename status * fix compile err * exclude sparse tensor * fix comments * fix comments * fix build err * rename file and move location * format code * move file to session folder * fix comments Co-authored-by: Randy <Randy@randysmac.attlocal.net>
2022-05-03 21:16:30 +00:00
std::vector<Input> inputs(1);
Input& input = inputs[0];
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
std::vector<int64_t> expected_dims_y = {3, 2};
std::vector<float> expected_values_y = {2.0f, 4.0f, 6.0f, 8.0f, 10.0f, 12.0f};
Generalize native op creation (#11539) * create op from ep * read input count from context * create holder to host nodes * fix typo * cast type before comparison * throw error on API fail * silence warning from minimal build * switch to unique_ptr with deleter to host nodes * fix typo * fix build err for minimal * fix build err for minimal * add UT for conv * enable test on CUDA * add comment * fix typo * use gsl::span and string view for Node constructor * Added two APIs - CopyKernelInfo and ReleaseKernelInfo * pass gsl::span by value * switch to span<NodeArg* const> to allow for reference to const containers * fix typo * fix reduced build err * fix reduced build err * refactoring node construction logic * rename exceptions * add input and output count as arguments for op creation * refactor static member * use ORT_CATCH instead of catch * cancel try catch * add static value name map * format input definition and set err code * fix comments * fix typo
2022-06-28 04:12:15 +00:00
#ifdef USE_CUDA
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
StandaloneCustomOp standalone_op{onnxruntime::kCudaExecutionProvider};
Generalize native op creation (#11539) * create op from ep * read input count from context * create holder to host nodes * fix typo * cast type before comparison * throw error on API fail * silence warning from minimal build * switch to unique_ptr with deleter to host nodes * fix typo * fix build err for minimal * fix build err for minimal * add UT for conv * enable test on CUDA * add comment * fix typo * use gsl::span and string view for Node constructor * Added two APIs - CopyKernelInfo and ReleaseKernelInfo * pass gsl::span by value * switch to span<NodeArg* const> to allow for reference to const containers * fix typo * fix reduced build err * fix reduced build err * refactoring node construction logic * rename exceptions * add input and output count as arguments for op creation * refactor static member * use ORT_CATCH instead of catch * cancel try catch * add static value name map * format input definition and set err code * fix comments * fix typo
2022-06-28 04:12:15 +00:00
#else
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
StandaloneCustomOp standalone_op{onnxruntime::kCpuExecutionProvider};
Generalize native op creation (#11539) * create op from ep * read input count from context * create holder to host nodes * fix typo * cast type before comparison * throw error on API fail * silence warning from minimal build * switch to unique_ptr with deleter to host nodes * fix typo * fix build err for minimal * fix build err for minimal * add UT for conv * enable test on CUDA * add comment * fix typo * use gsl::span and string view for Node constructor * Added two APIs - CopyKernelInfo and ReleaseKernelInfo * pass gsl::span by value * switch to span<NodeArg* const> to allow for reference to const containers * fix typo * fix reduced build err * fix reduced build err * refactoring node construction logic * rename exceptions * add input and output count as arguments for op creation * refactor static member * use ORT_CATCH instead of catch * cancel try catch * add static value name map * format input definition and set err code * fix comments * fix typo
2022-06-28 04:12:15 +00:00
#endif
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
Ort::CustomOpDomain custom_op_domain("test");
Generalize native op creation (#11539) * create op from ep * read input count from context * create holder to host nodes * fix typo * cast type before comparison * throw error on API fail * silence warning from minimal build * switch to unique_ptr with deleter to host nodes * fix typo * fix build err for minimal * fix build err for minimal * add UT for conv * enable test on CUDA * add comment * fix typo * use gsl::span and string view for Node constructor * Added two APIs - CopyKernelInfo and ReleaseKernelInfo * pass gsl::span by value * switch to span<NodeArg* const> to allow for reference to const containers * fix typo * fix reduced build err * fix reduced build err * refactoring node construction logic * rename exceptions * add input and output count as arguments for op creation * refactor static member * use ORT_CATCH instead of catch * cancel try catch * add static value name map * format input definition and set err code * fix comments * fix typo
2022-06-28 04:12:15 +00:00
custom_op_domain.Add(&standalone_op);
APIs for custom op to invoke ort operator directly (#10713) * draft kernel creation * setup eager context * call into kernel in eager mode * redefine test case * refact eager context * add comment * remove header * rename argument * redefine API definition with types * list outputs as argument * switch to int to represent length * fix compile err * create attribute API * add test case for topk * remove bool from c api * add gru test case * remove var * fix compile warnings * rename status * fix compile err * exclude sparse tensor * fix comments * fix comments * fix build err * rename file and move location * format code * move file to session folder * fix comments Co-authored-by: Randy <Randy@randysmac.attlocal.net>
2022-05-03 21:16:30 +00:00
Generalize native op creation (#11539) * create op from ep * read input count from context * create holder to host nodes * fix typo * cast type before comparison * throw error on API fail * silence warning from minimal build * switch to unique_ptr with deleter to host nodes * fix typo * fix build err for minimal * fix build err for minimal * add UT for conv * enable test on CUDA * add comment * fix typo * use gsl::span and string view for Node constructor * Added two APIs - CopyKernelInfo and ReleaseKernelInfo * pass gsl::span by value * switch to span<NodeArg* const> to allow for reference to const containers * fix typo * fix reduced build err * fix reduced build err * refactoring node construction logic * rename exceptions * add input and output count as arguments for op creation * refactor static member * use ORT_CATCH instead of catch * cancel try catch * add static value name map * format input definition and set err code * fix comments * fix typo
2022-06-28 04:12:15 +00:00
#ifdef USE_CUDA
Changes to support standalone custom ops in a minimal build. (#14497) ### Description <!-- Describe your changes. --> Changes to support standalone custom ops in a minimal build. Also incorporates changes from #14492 (needed to test builds prior to that being checked in). We first need to save the schema info from the operators used by the standalone op invoker in the ORT format model. Add mechanism for that. Merge the kernel lookup logic so the same is used in full and minimal build. NOTE: the version matching is now consistent with all other kernel lookups, and the call to CreateOp MUST use the exact version for the operator. Previously matching wasn't as strict, but this can lead to the incorrect kernel being chosen. Add tests. NOTE: There is currently no way to detect the ops/types/opsets used inside these custom ops as they don't exist until we create kernels, which is after model loading completes (which is the point the ORT format model is saved). Due to that they have to be manually added to the configuration used to do the reduced ops build. That shouldn't be too hard for the custom op author to add given the custom op implementation is specifying the op, opset and type constraints (i.e. they have the info and it's just a case of capturing/formatting it correctly). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable usage of the standalone op invoker by custom ops in a minimal build. --------- Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-03-01 01:22:54 +00:00
TestInference<float>(*ort_env, CUSTOM_OP_MODEL_URI, inputs, "Y", expected_dims_y, expected_values_y, 1,
custom_op_domain, nullptr);
Generalize native op creation (#11539) * create op from ep * read input count from context * create holder to host nodes * fix typo * cast type before comparison * throw error on API fail * silence warning from minimal build * switch to unique_ptr with deleter to host nodes * fix typo * fix build err for minimal * fix build err for minimal * add UT for conv * enable test on CUDA * add comment * fix typo * use gsl::span and string view for Node constructor * Added two APIs - CopyKernelInfo and ReleaseKernelInfo * pass gsl::span by value * switch to span<NodeArg* const> to allow for reference to const containers * fix typo * fix reduced build err * fix reduced build err * refactoring node construction logic * rename exceptions * add input and output count as arguments for op creation * refactor static member * use ORT_CATCH instead of catch * cancel try catch * add static value name map * format input definition and set err code * fix comments * fix typo
2022-06-28 04:12:15 +00:00
#else
Changes to support standalone custom ops in a minimal build. (#14497) ### Description <!-- Describe your changes. --> Changes to support standalone custom ops in a minimal build. Also incorporates changes from #14492 (needed to test builds prior to that being checked in). We first need to save the schema info from the operators used by the standalone op invoker in the ORT format model. Add mechanism for that. Merge the kernel lookup logic so the same is used in full and minimal build. NOTE: the version matching is now consistent with all other kernel lookups, and the call to CreateOp MUST use the exact version for the operator. Previously matching wasn't as strict, but this can lead to the incorrect kernel being chosen. Add tests. NOTE: There is currently no way to detect the ops/types/opsets used inside these custom ops as they don't exist until we create kernels, which is after model loading completes (which is the point the ORT format model is saved). Due to that they have to be manually added to the configuration used to do the reduced ops build. That shouldn't be too hard for the custom op author to add given the custom op implementation is specifying the op, opset and type constraints (i.e. they have the info and it's just a case of capturing/formatting it correctly). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable usage of the standalone op invoker by custom ops in a minimal build. --------- Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-03-01 01:22:54 +00:00
Ort::SessionOptions session_options;
const std::basic_string<ORTCHAR_T> ort_file = ORT_TSTR("testdata/foo_1.onnx.test_output.ort");
session_options.SetOptimizedModelFilePath(ort_file.c_str());
TestInference<float>(*ort_env, CUSTOM_OP_MODEL_URI, inputs, "Y", expected_dims_y, expected_values_y, 0,
custom_op_domain, nullptr, false, nullptr, &session_options);
TestInference<float>(*ort_env, ort_file, inputs, "Y", expected_dims_y, expected_values_y, 0,
custom_op_domain, nullptr);
Generalize native op creation (#11539) * create op from ep * read input count from context * create holder to host nodes * fix typo * cast type before comparison * throw error on API fail * silence warning from minimal build * switch to unique_ptr with deleter to host nodes * fix typo * fix build err for minimal * fix build err for minimal * add UT for conv * enable test on CUDA * add comment * fix typo * use gsl::span and string view for Node constructor * Added two APIs - CopyKernelInfo and ReleaseKernelInfo * pass gsl::span by value * switch to span<NodeArg* const> to allow for reference to const containers * fix typo * fix reduced build err * fix reduced build err * refactoring node construction logic * rename exceptions * add input and output count as arguments for op creation * refactor static member * use ORT_CATCH instead of catch * cancel try catch * add static value name map * format input definition and set err code * fix comments * fix typo
2022-06-28 04:12:15 +00:00
#endif
APIs for custom op to invoke ort operator directly (#10713) * draft kernel creation * setup eager context * call into kernel in eager mode * redefine test case * refact eager context * add comment * remove header * rename argument * redefine API definition with types * list outputs as argument * switch to int to represent length * fix compile err * create attribute API * add test case for topk * remove bool from c api * add gru test case * remove var * fix compile warnings * rename status * fix compile err * exclude sparse tensor * fix comments * fix comments * fix build err * rename file and move location * format code * move file to session folder * fix comments Co-authored-by: Randy <Randy@randysmac.attlocal.net>
2022-05-03 21:16:30 +00:00
}
#endif
Integrate onnxruntime-extensions into onnxruntime. (#8143) Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
2021-07-01 16:34:03 +00:00
#ifdef ENABLE_EXTENSION_CUSTOM_OPS
// test enabled ort-customops negpos
TEST(CApiTest, test_enable_ort_customops_negpos) {
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
auto allocator = std::make_unique<MockedOrtAllocator>();
// Create Inputs
std::vector<Ort::Value> ort_inputs;
std::vector<float> input_data = {-1.1f, 2.2f, 4.4f, -5.5f};
std::vector<int64_t> input_dims = {2, 2};
ort_inputs.emplace_back(Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_data.data()), input_data.size(), input_dims.data(), input_dims.size()));
// Create Session with ORT CustomOps
Ort::SessionOptions session_options;
session_options.EnableOrtCustomOps();
Ort::Session session(*ort_env, ORT_CUSTOM_OPS_MODEL_URI_2, session_options);
// Create Input and Output Names
std::vector<const char*> input_names = {"X"};
const char* output_names[] = {"out0", "out1"};
// Run Session
std::vector<Ort::Value> ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(), output_names, countof(output_names));
// Validate Results
ASSERT_EQ(ort_outputs.size(), 2u);
std::vector<int64_t> out_dims = {2, 2};
std::vector<float> values_out0 = {-1.1f, 0.0f, 0.0f, -5.5f};
auto type_info = ort_outputs[0].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), out_dims);
size_t total_len = type_info.GetElementCount();
ASSERT_EQ(values_out0.size(), total_len);
float* f = ort_outputs[0].GetTensorMutableData<float>();
for (size_t i = 0; i != total_len; ++i) {
ASSERT_EQ(values_out0[i], f[i]);
}
}
// test enabled ort-customops stringlower
TEST(CApiTest, test_enable_ort_customops_stringlower) {
auto allocator = std::make_unique<MockedOrtAllocator>();
// Create Inputs
std::vector<Ort::Value> ort_inputs;
std::string input_data{"HI, This is ENGINEER from Microsoft."};
const char* const input_strings[] = {input_data.c_str()};
std::vector<int64_t> input_dims = {1, 1};
Ort::Value input_tensor = Ort::Value::CreateTensor(allocator.get(), input_dims.data(), input_dims.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
input_tensor.FillStringTensor(input_strings, 1U);
ort_inputs.push_back(std::move(input_tensor));
// Create Session with ORT CustomOps
Ort::SessionOptions session_options;
session_options.EnableOrtCustomOps();
Ort::Session session(*ort_env, ORT_CUSTOM_OPS_MODEL_URI, session_options);
// Create Input and Output Names
std::vector<const char*> input_names = {"input_1"};
const char* output_names[] = {"customout"};
// Run Session
std::vector<Ort::Value> ort_outputs = session.Run(Ort::RunOptions{nullptr}, input_names.data(), ort_inputs.data(), ort_inputs.size(), output_names, countof(output_names));
// Validate Results
ASSERT_EQ(ort_outputs.size(), 1u);
std::vector<int64_t> out_dims = {1, 1};
auto type_info = ort_outputs[0].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), out_dims);
ASSERT_EQ(type_info.GetElementType(), ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
std::string output_data{"hi, this is engineer from microsoft."};
auto expected_string = output_data.c_str();
size_t expected_string_len = strlen(expected_string);
auto data_length = ort_outputs[0].GetStringTensorDataLength();
ASSERT_EQ(expected_string_len, data_length);
std::string result(data_length, '\0');
std::vector<size_t> offsets(type_info.GetElementCount());
ort_outputs[0].GetStringTensorContent((void*)result.data(), data_length, offsets.data(), offsets.size());
ASSERT_STREQ(result.c_str(), expected_string);
}
#endif
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// test custom op which accepts float and double as inputs
Enable varied input type for custom op (#6066) * allow custom op taking varied types * refactor test case * add test model * refactor test case * enable copy elision * update test case * fix issue in ToString function
2020-12-09 23:10:42 +00:00
TEST(CApiTest, varied_input_custom_op_handler) {
std::vector<Input> inputs(2);
inputs[0].name = "X";
inputs[0].dims = {3};
inputs[0].values = {2.0f, 3.0f, 4.0f};
inputs[1].name = "Y";
inputs[1].dims = {3};
inputs[1].values = {5.0f, 6.0f, 7.0f};
std::vector<int64_t> expected_dims_z = {1};
std::vector<float> expected_values_z = {10.0f};
#ifdef USE_CUDA
Expose an API to query the CUDA compute stream to launch a custom kernel (#9141)
2021-11-09 05:10:34 +00:00
SliceCustomOp slice_custom_op{onnxruntime::kCudaExecutionProvider};
Enable varied input type for custom op (#6066) * allow custom op taking varied types * refactor test case * add test model * refactor test case * enable copy elision * update test case * fix issue in ToString function
2020-12-09 23:10:42 +00:00
#else
Expose an API to query the CUDA compute stream to launch a custom kernel (#9141)
2021-11-09 05:10:34 +00:00
SliceCustomOp slice_custom_op{onnxruntime::kCpuExecutionProvider};
Enable varied input type for custom op (#6066) * allow custom op taking varied types * refactor test case * add test model * refactor test case * enable copy elision * update test case * fix issue in ToString function
2020-12-09 23:10:42 +00:00
#endif
Ort::CustomOpDomain custom_op_domain("abc");
custom_op_domain.Add(&slice_custom_op);
#ifdef USE_CUDA
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, VARIED_INPUT_CUSTOM_OP_MODEL_URI, inputs, "Z",
Expose an API to query the CUDA compute stream to launch a custom kernel (#9141)
2021-11-09 05:10:34 +00:00
expected_dims_z, expected_values_z, 1, custom_op_domain, nullptr);
Enable varied input type for custom op (#6066) * allow custom op taking varied types * refactor test case * add test model * refactor test case * enable copy elision * update test case * fix issue in ToString function
2020-12-09 23:10:42 +00:00
#else
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, VARIED_INPUT_CUSTOM_OP_MODEL_URI, inputs, "Z",
expected_dims_z, expected_values_z, 0, custom_op_domain, nullptr);
Enable varied input type for custom op (#6066) * allow custom op taking varied types * refactor test case * add test model * refactor test case * enable copy elision * update test case * fix issue in ToString function
2020-12-09 23:10:42 +00:00
#endif
}
Support multiple dynamic inputs in custom ops (#6666)
2021-02-17 04:54:30 +00:00
TEST(CApiTest, multiple_varied_input_custom_op_handler) {
#ifdef USE_CUDA
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
cudaStream_t compute_stream = nullptr;
cudaStreamCreateWithFlags(&compute_stream, cudaStreamNonBlocking);
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
MyCustomOpMultipleDynamicInputs custom_op{onnxruntime::kCudaExecutionProvider};
Support multiple dynamic inputs in custom ops (#6666)
2021-02-17 04:54:30 +00:00
#else
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
MyCustomOpMultipleDynamicInputs custom_op{onnxruntime::kCpuExecutionProvider};
Support multiple dynamic inputs in custom ops (#6666)
2021-02-17 04:54:30 +00:00
#endif
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
Ort::CustomOpDomain custom_op_domain("test");
Support multiple dynamic inputs in custom ops (#6666)
2021-02-17 04:54:30 +00:00
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
#ifdef USE_CUDA
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
auto cuda_options = CreateDefaultOrtCudaProviderOptionsWithCustomStream(compute_stream);
session_options.AppendExecutionProvider_CUDA(cuda_options);
Support multiple dynamic inputs in custom ops (#6666)
2021-02-17 04:54:30 +00:00
#endif
session_options.Add(custom_op_domain);
Ort::Session session(*ort_env, VARIED_INPUT_CUSTOM_OP_MODEL_URI_2, session_options);
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
std::vector<const char*> input_names;
// input 0 (float type)
input_names.emplace_back("X");
std::vector<float> input_0_data = {1.f, 2.f, 3.f, 4.f, 5.f, 6.f};
std::vector<int64_t> input_0_dims = {3, 2};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_0_data.data()),
input_0_data.size(), input_0_dims.data(), input_0_dims.size()));
// input 1 (double type)
input_names.emplace_back("W");
std::vector<double> input_1_data = {2, 3, 4, 5, 6, 7};
std::vector<int64_t> input_1_dims = {3, 2};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<double>(info, const_cast<double*>(input_1_data.data()),
input_1_data.size(), input_1_dims.data(), input_1_dims.size()));
// Run
const char* output_name = "Y";
auto ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
&output_name, 1);
ASSERT_EQ(ort_outputs.size(), 1u);
// Validate results
std::vector<int64_t> y_dims = {3, 2};
std::vector<float> values_y = {3.f, 5.f, 7.f, 9.f, 11.f, 13.f};
auto type_info = ort_outputs[0].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), y_dims);
size_t total_len = type_info.GetElementCount();
ASSERT_EQ(values_y.size(), total_len);
float* f = ort_outputs[0].GetTensorMutableData<float>();
for (size_t i = 0; i != total_len; ++i) {
ASSERT_EQ(values_y[i], f[i]);
}
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
#ifdef USE_CUDA
cudaStreamDestroy(compute_stream);
#endif
Support multiple dynamic inputs in custom ops (#6666)
2021-02-17 04:54:30 +00:00
}
Support for custom op variadic inputs/outputs (#13946) ### Description Adds support for variadic inputs and outputs to custom operators. ### Motivation and Context Needed for custom ops that wrap external runtimes/models and maybe TensorRT plugins.
2022-12-23 19:41:15 +00:00
TEST(CApiTest, variadic_input_output_custom_op) {
// Create a custom op with 1 variadic input and 1 variadic output.
// The model passes in 3 string inputs and expects 3 int64_t outputs.
TemplatedCustomOp<MyCustomStringLengthsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true,
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
std::vector<Ort::Value> ort_inputs;
Ort::AllocatorWithDefaultOptions allocator;
std::vector<std::vector<int64_t>> expected_dims;
std::vector<std::vector<int64_t>> expected_lens;
std::vector<std::string> input_names;
std::vector<std::string> output_names;
// Create inputs.
AddInputForCustomStringLengthsKernel("hello", allocator, ort_inputs, input_names, output_names, expected_dims,
expected_lens);
AddInputForCustomStringLengthsKernel("", allocator, ort_inputs, input_names, output_names, expected_dims,
expected_lens);
AddInputForCustomStringLengthsKernel("123", allocator, ort_inputs, input_names, output_names, expected_dims,
expected_lens);
// Create arrays of c-strings for input and output names.
auto get_c_str = [](const std::string& str) { return str.c_str(); };
std::vector<const char*> input_name_cstrs(input_names.size());
std::transform(input_names.begin(), input_names.end(), input_name_cstrs.begin(), get_c_str);
std::vector<const char*> output_name_cstrs(output_names.size());
std::transform(output_names.begin(), output_names.end(), output_name_cstrs.begin(), get_c_str);
Ort::Session session(*ort_env, VARIADIC_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
auto ort_outputs = session.Run(Ort::RunOptions{}, input_name_cstrs.data(), ort_inputs.data(), ort_inputs.size(),
output_name_cstrs.data(), output_name_cstrs.size());
ASSERT_EQ(ort_outputs.size(), 3u);
// Validate outputs.
for (size_t i = 0; i < ort_outputs.size(); ++i) {
auto type_info = ort_outputs[i].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), expected_dims[i]);
ASSERT_EQ(type_info.GetElementCount(), 1u);
int64_t* lens_data = ort_outputs[i].GetTensorMutableData<int64_t>();
ASSERT_EQ(lens_data[0], expected_lens[i][0]);
}
}
TEST(CApiTest, mixed_variadic_input_output_custom_op) {
// Create a custom op with 2 inputs (required, variadic) and 2 outputs (required, variadic).
// The model passes in 3 string inputs and expects 3 int64_t outputs.
TemplatedCustomOp<MyCustomStringLengthsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING,
ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_REQUIRED,
OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true,
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64,
ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_REQUIRED,
OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
std::vector<Ort::Value> ort_inputs;
Ort::AllocatorWithDefaultOptions allocator;
std::vector<std::vector<int64_t>> expected_dims;
std::vector<std::vector<int64_t>> expected_lens;
std::vector<std::string> input_names;
std::vector<std::string> output_names;
// Create inputs.
AddInputForCustomStringLengthsKernel("mixed variadic", allocator, ort_inputs, input_names, output_names,
expected_dims, expected_lens);
AddInputForCustomStringLengthsKernel("", allocator, ort_inputs, input_names, output_names, expected_dims,
expected_lens);
AddInputForCustomStringLengthsKernel("abcd", allocator, ort_inputs, input_names, output_names, expected_dims,
expected_lens);
// Create arrays of c-strings for input and output names.
auto get_c_str = [](const std::string& str) { return str.c_str(); };
std::vector<const char*> input_name_cstrs(input_names.size());
std::transform(input_names.begin(), input_names.end(), input_name_cstrs.begin(), get_c_str);
std::vector<const char*> output_name_cstrs(output_names.size());
std::transform(output_names.begin(), output_names.end(), output_name_cstrs.begin(), get_c_str);
Ort::Session session(*ort_env, VARIADIC_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
auto ort_outputs = session.Run(Ort::RunOptions{}, input_name_cstrs.data(), ort_inputs.data(), ort_inputs.size(),
output_name_cstrs.data(), output_name_cstrs.size());
ASSERT_EQ(ort_outputs.size(), 3u);
// Validate outputs.
for (size_t i = 0; i < ort_outputs.size(); ++i) {
auto type_info = ort_outputs[i].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), expected_dims[i]);
ASSERT_EQ(type_info.GetElementCount(), 1u);
int64_t* lens_data = ort_outputs[i].GetTensorMutableData<int64_t>();
ASSERT_EQ(lens_data[0], expected_lens[i][0]);
}
}
TEST(CApiTest, variadic_undef_input_output_custom_op) {
// Create a custom op with 1 variadic input and 1 variadic output.
// Both the input and output are of undefined element type and allowed to differ in type (hetergeneous).
// The model passes in inputs (string, int64_t, and float) which are then echoed in
// reversed order (float, int64_t, string).
TemplatedCustomOp<MyCustomEchoReversedArgsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
false,
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
false);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
std::vector<Ort::Value> ort_inputs;
Ort::AllocatorWithDefaultOptions allocator;
Ort::ConstMemoryInfo mem_info = allocator.GetInfo();
std::vector<int64_t> input_dims = {1};
// Set string input.
std::string str_input("hello_ort");
Ort::Value& str_input_val = ort_inputs.emplace_back(
Ort::Value::CreateTensor(allocator, input_dims.data(), input_dims.size(),
ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING));
str_input_val.FillStringTensorElement(str_input.c_str(), 0);
// Set int64_t input.
std::array<int64_t, 1> int_inps = {23};
ort_inputs.emplace_back(Ort::Value::CreateTensor<int64_t>(mem_info, int_inps.data(), int_inps.size(),
input_dims.data(), input_dims.size()));
// Set float input.
std::array<float, 1> float_inps = {10.0f};
ort_inputs.emplace_back(Ort::Value::CreateTensor<float>(mem_info, float_inps.data(), float_inps.size(),
input_dims.data(), input_dims.size()));
constexpr std::array<const char*, 3> input_names = {"input_0", "input_1", "input_2"};
constexpr std::array<const char*, 3> output_names = {"output_0", "output_1", "output_2"};
Ort::Session session(*ort_env, VARIADIC_UNDEF_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
auto ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
output_names.data(), output_names.size());
ASSERT_EQ(ort_outputs.size(), 3u);
// Validate outputs.
// First output should be a float.
{
auto& ort_output = ort_outputs[0];
auto type_info = ort_output.GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), input_dims);
ASSERT_EQ(type_info.GetElementCount(), 1u);
ASSERT_EQ(type_info.GetElementType(), ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);
const float* out_ptr = ort_output.GetTensorData<float>();
ASSERT_EQ(out_ptr[0], float_inps[0]);
}
// Second output should be a int64_t.
{
auto& ort_output = ort_outputs[1];
auto type_info = ort_output.GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), input_dims);
ASSERT_EQ(type_info.GetElementCount(), 1u);
ASSERT_EQ(type_info.GetElementType(), ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64);
const int64_t* out_ptr = ort_output.GetTensorData<int64_t>();
ASSERT_EQ(out_ptr[0], int_inps[0]);
}
// Last output should be a string.
{
auto& ort_output = ort_outputs[2];
auto type_info = ort_output.GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), input_dims);
ASSERT_EQ(type_info.GetElementCount(), 1u);
ASSERT_EQ(type_info.GetElementType(), ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
const size_t str_len = ort_output.GetStringTensorElementLength(0);
ASSERT_EQ(str_len, str_input.size());
std::string str;
str.resize(str_len);
ort_output.GetStringTensorElement(str_len, 0, str.data());
ASSERT_EQ(str, str_input);
}
}
TEST(CApiTest, invalid_variadic_input_not_last_custom_op) {
// Create an invalid custom op with 2 inputs. The first input is variadic and the last is not.
// Expect an error because only the last input may be marked as variadic.
TemplatedCustomOp<MyCustomStringLengthsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING,
ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC,
OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_REQUIRED},
1,
true,
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64,
ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_REQUIRED,
OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
try {
Ort::Session session(*ort_env, VARIADIC_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
FAIL();
} catch (const Ort::Exception& excpt) {
ASSERT_THAT(excpt.what(), testing::HasSubstr("Only the last input to a custom op may be marked variadic."));
}
}
TEST(CApiTest, invalid_variadic_output_not_last_custom_op) {
// Create an invalid custom op with 2 outputs. The first output is variadic and the last is not.
// Expect an error because only the last output may be marked as variadic.
TemplatedCustomOp<MyCustomStringLengthsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING,
ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_REQUIRED,
OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true,
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64,
ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC,
OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_REQUIRED},
1,
true);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
try {
Ort::Session session(*ort_env, VARIADIC_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
FAIL();
} catch (const Ort::Exception& excpt) {
ASSERT_THAT(excpt.what(), testing::HasSubstr("Only the last output to a custom op may be marked variadic."));
}
}
TEST(CApiTest, invalid_variadic_input_min_arity_custom_op) {
// Create a custom op with a variadic input with a minimum arity of 4.
// Expect an error because the model passes in less than 4 inputs to the op.
TemplatedCustomOp<MyCustomStringLengthsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
4,
true,
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
try {
Ort::Session session(*ort_env, VARIADIC_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
FAIL();
} catch (const Ort::Exception& excpt) {
ASSERT_THAT(excpt.what(), testing::HasSubstr("Error Node (VariadicNode0) has input size 3 not in range [min=4"));
}
}
TEST(CApiTest, invalid_variadic_output_min_arity_custom_op) {
// Create a custom op with a variadic output with a minimum arity of 4.
// Expect an error because the model instantiates the op with less than 4 outputs.
TemplatedCustomOp<MyCustomStringLengthsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true,
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
4,
true);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
try {
Ort::Session session(*ort_env, VARIADIC_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
FAIL();
} catch (const Ort::Exception& excpt) {
ASSERT_THAT(excpt.what(), testing::HasSubstr("Error Node (VariadicNode0) has output size 3 not in range [min=4"));
}
}
TEST(CApiTest, invalid_variadic_input_homogeneity_custom_op) {
// Create a custom op with a homogeneous variadic input. The model has heterogeneous inputs,
// so we expect an error.
TemplatedCustomOp<MyCustomEchoReversedArgsKernel> custom_op(
"VariadicNode",
// Input config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
true, // Input homogeneity requirement will cause error!
// Output config
{ONNXTensorElementDataType::ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED},
{OrtCustomOpInputOutputCharacteristic::INPUT_OUTPUT_VARIADIC},
1,
false);
Ort::CustomOpDomain custom_op_domain("test");
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
try {
Ort::Session session(*ort_env, VARIADIC_UNDEF_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
FAIL();
} catch (const Ort::Exception& excpt) {
Single-schema-multi-kernel (#15184) The PR is to allow custom op of different input types to have same op name in a graph. The idea to go over all ops of same name and merge their input/output types into a type-inference function. With the enhancement, custom op node inside a graph can have same op-type given that the input/output types are different. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-04-27 20:39:59 +00:00
ASSERT_THAT(excpt.what(), testing::HasSubstr("Type Error: Type parameter (Input0) of Optype (VariadicNode) bound "
Support for custom op variadic inputs/outputs (#13946) ### Description Adds support for variadic inputs and outputs to custom operators. ### Motivation and Context Needed for custom ops that wrap external runtimes/models and maybe TensorRT plugins.
2022-12-23 19:41:15 +00:00
"to different types"));
}
}
Support optional inputs/outputs in custom op development (#6727)
2021-03-03 13:59:23 +00:00
TEST(CApiTest, optional_input_output_custom_op_handler) {
MyCustomOpWithOptionalInput custom_op{onnxruntime::kCpuExecutionProvider};
// `MyCustomOpFooBar` defines a custom op with atmost 3 inputs and the second input is optional.
// In this test, we are going to try and run 2 models - one with the optional input and one without
// the optional input.
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
Ort::CustomOpDomain custom_op_domain("test");
Support optional inputs/outputs in custom op development (#6727)
2021-03-03 13:59:23 +00:00
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
std::vector<Ort::Value> ort_inputs;
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
// input 0
std::vector<float> input_0_data = {1.f};
std::vector<int64_t> input_0_dims = {1};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_0_data.data()),
input_0_data.size(), input_0_dims.data(), input_0_dims.size()));
// input 1
std::vector<float> input_1_data = {1.f};
std::vector<int64_t> input_1_dims = {1};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_1_data.data()),
input_1_data.size(), input_1_dims.data(), input_1_dims.size()));
// input 2
std::vector<float> input_2_data = {1.f};
std::vector<int64_t> input_2_dims = {1};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_2_data.data()),
input_2_data.size(), input_2_dims.data(), input_2_dims.size()));
const char* output_name = "Y";
// Part 1: Model with optional input present
{
std::vector<const char*> input_names = {"X1", "X2", "X3"};
Ort::Session session(*ort_env, OPTIONAL_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI, session_options);
auto ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
&output_name, 1);
ASSERT_EQ(ort_outputs.size(), 1u);
// Validate results
std::vector<int64_t> y_dims = {1};
std::vector<float> values_y = {3.f};
auto type_info = ort_outputs[0].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), y_dims);
size_t total_len = type_info.GetElementCount();
ASSERT_EQ(values_y.size(), total_len);
float* f = ort_outputs[0].GetTensorMutableData<float>();
for (size_t i = 0; i != total_len; ++i) {
ASSERT_EQ(values_y[i], f[i]);
}
}
// Part 2: Model with optional input absent
{
std::vector<const char*> input_names = {"X1", "X2"};
ort_inputs.erase(ort_inputs.begin() + 2); // remove the last input in the container
Ort::Session session(*ort_env, OPTIONAL_INPUT_OUTPUT_CUSTOM_OP_MODEL_URI_2, session_options);
auto ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
&output_name, 1);
ASSERT_EQ(ort_outputs.size(), 1u);
// Validate results
std::vector<int64_t> y_dims = {1};
std::vector<float> values_y = {2.f};
auto type_info = ort_outputs[0].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), y_dims);
size_t total_len = type_info.GetElementCount();
ASSERT_EQ(values_y.size(), total_len);
float* f = ort_outputs[0].GetTensorMutableData<float>();
for (size_t i = 0; i != total_len; ++i) {
ASSERT_EQ(values_y[i], f[i]);
}
}
}
Support parsing an array of values stored as an attribute in a custom op (#6878)
2021-03-09 07:49:58 +00:00
TEST(CApiTest, custom_op_with_attributes_handler) {
MyCustomOpWithAttributes custom_op{onnxruntime::kCpuExecutionProvider};
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
Ort::CustomOpDomain custom_op_domain("test");
Support parsing an array of values stored as an attribute in a custom op (#6878)
2021-03-09 07:49:58 +00:00
custom_op_domain.Add(&custom_op);
Ort::SessionOptions session_options;
session_options.Add(custom_op_domain);
Ort::Session session(*ort_env, CUSTOM_OP_MODEL_WITH_ATTRIBUTES_URI, session_options);
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
std::vector<const char*> input_names;
// input 0 (float type)
input_names.emplace_back("X");
std::vector<float> input_0_data = {1.f};
std::vector<int64_t> input_0_dims = {1};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_0_data.data()),
input_0_data.size(), input_0_dims.data(), input_0_dims.size()));
// Run
const char* output_name = "Y";
auto ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
&output_name, 1);
ASSERT_EQ(ort_outputs.size(), 1u);
// Validate results
std::vector<int64_t> y_dims = {1};
std::vector<float> values_y = {15.f};
auto type_info = ort_outputs[0].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), y_dims);
size_t total_len = type_info.GetElementCount();
ASSERT_EQ(values_y.size(), total_len);
float* f = ort_outputs[0].GetTensorMutableData<float>();
for (size_t i = 0; i != total_len; ++i) {
ASSERT_EQ(values_y[i], f[i]);
}
}
Support optional inputs/outputs in custom op development (#6727)
2021-03-03 13:59:23 +00:00
Allow a custom op with the same name to be registered for several providers. (#3400)
2020-04-02 22:38:51 +00:00
// Tests registration of a custom op of the same name for both CPU and CUDA EPs
#ifdef USE_CUDA
TEST(CApiTest, RegisterCustomOpForCPUAndCUDA) {
std::cout << "Tests registration of a custom op of the same name for both CPU and CUDA EPs" << std::endl;
std::vector<Input> inputs(1);
Input& input = inputs[0];
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {3, 2};
std::vector<float> expected_values_y = {2.0f, 4.0f, 6.0f, 8.0f, 10.0f, 12.0f};
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
MyCustomOp custom_op_cpu{onnxruntime::kCpuExecutionProvider};
Update CUDA custom op unit tests to account for recent ORT change (#6971)
2021-03-12 06:22:45 +00:00
// We are going to test session creation only - hence it is not a problem to use the default stream as the compute stream for the custom op
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
MyCustomOp custom_op_cuda{onnxruntime::kCudaExecutionProvider};
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
Ort::CustomOpDomain custom_op_domain("test");
Allow a custom op with the same name to be registered for several providers. (#3400)
2020-04-02 22:38:51 +00:00
custom_op_domain.Add(&custom_op_cpu);
custom_op_domain.Add(&custom_op_cuda);
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, CUSTOM_OP_MODEL_URI, inputs, "Y", expected_dims_y,
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
expected_values_y, 1, custom_op_domain, nullptr, true);
Allow a custom op with the same name to be registered for several providers. (#3400)
2020-04-02 22:38:51 +00:00
}
#endif
FasterTransformer model wrapper using custom op (#15013) ### Description <!-- Describe your changes. --> We are introducing the FasterTransfomer model-level integration using ORT [custom op runtime wrapper](https://github.com/microsoft/onnxruntime/pull/13427). In order to make the FT wrapper/integration work, two things need to be done: - New API `KernelInfoGetConstantInput_tensor`. (Done in this PR) During custom op kernel initialization, it needs to get the model weights (saved as node's constant inputs) ready for FT's weights instantiation. What's why we need to add this new API to make kernel info capable of getting constant inputs. - Custom op and custom op kernel to wrap FT model. (Will provide in onnxruntime extensions or inference examples) During custom op kernel initialization, it can fetch attributes from kernel info to determine which kind of FT model instance create. During custom op kernel compute/inference, it can get input/output from kernel context and then assign input/output buffers for model instance to run.
2023-03-20 16:05:30 +00:00
#if (!defined(ORT_MINIMAL_BUILD)) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS)
TEST(CApiTest, test_custom_op_get_const_input) {
const auto* model_path = TSTR("testdata/test_kernel_info_get_const_input.onnx");
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
std::vector<const char*> input_names;
// input 0 (float type)
input_names.emplace_back("input1");
std::vector<float> input_0_data = {1.0f, 1.0f, 1.0f, 1.0f};
std::vector<int64_t> input_0_dims = {1, 4};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_0_data.data()),
input_0_data.size(), input_0_dims.data(), input_0_dims.size()));
const char* output_name = "output";
const ORTCHAR_T* lib_name;
#if defined(_WIN32)
lib_name = ORT_TSTR("custom_op_get_const_input_test_library.dll");
#elif defined(__APPLE__)
lib_name = ORT_TSTR("libcustom_op_get_const_input_test_library.dylib");
#else
lib_name = ORT_TSTR("./libcustom_op_get_const_input_test_library.so");
#endif
Ort::SessionOptions session_opts;
session_opts.RegisterCustomOpsLibrary(lib_name);
Ort::Session session(*ort_env, model_path, session_opts);
auto default_allocator = std::make_unique<MockedOrtAllocator>();
session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
Run clang-format in CI (#15524) ### Description Run clang-format in CI. Formatted all c/c++, objective-c/c++ files. Excluded ``` 'onnxruntime/core/mlas/**', 'onnxruntime/contrib_ops/cuda/bert/tensorrt_fused_multihead_attention/**', ``` because they contain assembly or is data heavy ### Motivation and Context Coding style consistency
2023-04-18 16:26:58 +00:00
&output_name, 1);
FasterTransformer model wrapper using custom op (#15013) ### Description <!-- Describe your changes. --> We are introducing the FasterTransfomer model-level integration using ORT [custom op runtime wrapper](https://github.com/microsoft/onnxruntime/pull/13427). In order to make the FT wrapper/integration work, two things need to be done: - New API `KernelInfoGetConstantInput_tensor`. (Done in this PR) During custom op kernel initialization, it needs to get the model weights (saved as node's constant inputs) ready for FT's weights instantiation. What's why we need to add this new API to make kernel info capable of getting constant inputs. - Custom op and custom op kernel to wrap FT model. (Will provide in onnxruntime extensions or inference examples) During custom op kernel initialization, it can fetch attributes from kernel info to determine which kind of FT model instance create. During custom op kernel compute/inference, it can get input/output from kernel context and then assign input/output buffers for model instance to run.
2023-03-20 16:05:30 +00:00
}
#endif
Fix schema type constraint for custom operators (#17497) ### Description onnxruntime may raise an error "type inference failed" but when a custom operator sets IsHomogeneous to false in its schema. This change make sure that TypeInferenceFunction and schema type constraints are aligned to prevent that from happening. --------- Co-authored-by: Xavier Dupre <xadupre@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>
2024-01-04 19:27:46 +00:00
#if !defined(ORT_MINIMAL_BUILD) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS)
#if defined(__ANDROID__)
// Disable on android because custom op libraries are not copied to the emulator.
TEST(CApiTest, DISABLED_test_custom_op_local_function) {
#else
TEST(CApiTest, test_custom_op_local_function) {
#endif // defined(__ANDROID__)
const auto* model_path = TSTR("testdata/custom_op_local_function/custom_ops_type_inference_fails_0.onnx");
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
std::vector<const char*> input_names;
// input 0 (float type)
input_names.emplace_back("X");
std::vector<float> input_0_data = {1.0f, 2.0f, 3.0f, 4.0f};
std::vector<int64_t> input_0_dims = {2, 2};
ort_inputs.emplace_back(
Ort::Value::CreateTensor<float>(info, const_cast<float*>(input_0_data.data()),
input_0_data.size(), input_0_dims.data(), input_0_dims.size()));
const char* output_name = "Y";
const ORTCHAR_T* lib_name;
#if defined(_WIN32)
lib_name = ORT_TSTR("custom_op_local_function.dll");
#elif defined(__APPLE__)
lib_name = ORT_TSTR("libcustom_op_local_function.dylib");
#else
lib_name = ORT_TSTR("./libcustom_op_local_function.so");
#endif
Ort::SessionOptions session_opts;
session_opts.RegisterCustomOpsLibrary(lib_name);
Ort::Session session(*ort_env, model_path, session_opts);
auto default_allocator = std::make_unique<MockedOrtAllocator>();
session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(),
&output_name, 1);
}
#endif // !defined(ORT_MINIMAL_BUILD) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS)
Custom Op runtime wrapper (#13427) ### Description Adds the below C APIs to support custom ops that wrap an entire model to be inferenced with an external runtime. The current SNPE EP is an example of an EP that could be ported to use a custom op wrapper. Ex: The custom op stores the serialized SNPE DLC binary as a string attribute. The SNPE model is built when the kernel is created. The model is inferenced with SNPE APIs on call to the kernel's compute method. #### C APIs | API | Description | Why | | --- | --- | --- | | `KernelInfo_GetInputCount` | Gets number of inputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputCount` | Gets number of outputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputName` | Gets an input's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputName` | Gets an output's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputTypeInfo` | Gets the type/shape information for an input. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputTypeInfo` | Gets the type/shape information for an output. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfoGetAttribute_tensor` | Get a OrtValue tensor stored as an attribute in the graph node | Extract serialized models, weights, etc. | | `GetSessionConfigEntry` | Get a session configuration value | Need to be able to get session-time configurations from within custom op | | `HasSessionConfigEntry` | Check if session configuration entry exists. | Need to be able to get session-time configurations from within custom op | #### Why so many KernelInfo APIs?<sup>1</sup> Similar APIs currently exist for `OrtKernelContext`, but not `OrtKernelInfo`. Note that `OrtKernelContext` is passed to the custom op on call to its kernel's compute() function. However, `OrtKernelInfo` is available on kernel creation, which occurs when the session is created. Having these APIs available from `OrtKernelInfo` allows an operator to trade-off computation time for session-creation time, and vice versa. Operators that must build expensive state may prefer to do it during session creation time instead of compute-time. SNPE is an example of an EP that needs to be able to query `KernelInfo` for the name, type, and shape of inputs and outputs in order to build the model from the serialized DLC data. This is an expensive operation. Other providers (e.g., OpenVINO) are able to query i/o info from the serialized model, so they do not strictly need these APIs. However, the APIs can still be used to validate the expected I/O characteristics. Additionally, several of our CPU contrib ops currently use the same internal version of these KernelInfo APIs (Ex: [qlinear_softmax](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/quantization/qlinear_softmax.cc#L71)). If custom ops are also meant to be a test bed for future ops, then all custom ops (not just runtime wrappers) would benefit from the addition of these public KernelInfo APIs (IMO). #### Example of usage in a custom OP From `onnxruntime/test/testdata/custom_op_openvino_wrapper_library/openvino_wrapper.h` ```c++ struct CustomOpOpenVINO : Ort::CustomOpBase<CustomOpOpenVINO, KernelOpenVINO> { explicit CustomOpOpenVINO(Ort::ConstSessionOptions session_options); CustomOpOpenVINO(const CustomOpOpenVINO&) = delete; CustomOpOpenVINO& operator=(const CustomOpOpenVINO&) = delete; void* CreateKernel(const OrtApi& api, const OrtKernelInfo* info) const; constexpr const char* GetName() const noexcept { return "OpenVINO_Wrapper"; } constexpr const char* GetExecutionProviderType() const noexcept { return "CPUExecutionProvider"; } // IMPORTANT: In order to wrap a generic runtime-specific model, the custom operator // must have a non-homogeneous variadic input and output. constexpr size_t GetInputTypeCount() const noexcept { return 1; } constexpr size_t GetOutputTypeCount() const noexcept { return 1; } constexpr ONNXTensorElementDataType GetInputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr ONNXTensorElementDataType GetOutputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr OrtCustomOpInputOutputCharacteristic GetInputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr OrtCustomOpInputOutputCharacteristic GetOutputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr bool GetVariadicInputHomogeneity() const noexcept { return false; // heterogenous } constexpr bool GetVariadicOutputHomogeneity() const noexcept { return false; // heterogeneous } std::vector<std::string> GetSessionConfigKeys() const { return {"device_type"}; } private: std::unordered_map<std::string, std::string> session_configs_; }; ``` #### How to create a session: ```c++ Ort::Env env; Ort::SessionOptions session_opts; Ort::CustomOpConfigs custom_op_configs; // Create local session config entries for the custom op. custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU"); // Register custom op library and pass in the custom op configs (optional). session_opts.RegisterCustomOpsLibrary(lib_name, custom_op_configs); Ort::Session session(env, model_path.data(), session_opts); ``` ### Motivation and Context Allows creation of simple "wrapper" EPs outside of the main ORT code base.
2023-01-18 17:09:32 +00:00
#if defined(USE_OPENVINO) && (!defined(ORT_MINIMAL_BUILD) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS))
TEST(CApiTest, test_custom_op_openvino_wrapper_library) {
// Tests a custom operator that wraps an OpenVINO MNIST model (.xml and .bin files serialized into node attributes).
// The custom op extracts the serialized .xml/.bin bytes and creates an in-memory OpenVINO model
// during kernel creation. The custom op is passed an image of a hand-drawn "1" as an input during computation, which
// is then inferenced using OpenVINO C++ APIs.
std::vector<Input> inputs(1);
inputs[0].name = "Input3";
inputs[0].dims = {1, 1, 28, 28};
// Float image with the digit "1".
inputs[0].values = {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.75f, 1.0f, 0.75f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.85f, 0.99f, 0.85f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 1.0f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 0.99f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.98f, 1.0f, 0.98f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.99f, 1.0f, 0.99f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.94f, 0.99f, 0.94f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.1f, 0.75f, 0.75f, 0.75f, 0.1f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f,
0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f};
// prepare expected outputs
std::vector<int64_t> expected_output_dims = {1, 10};
// Digit 1 (index 1) has the highest probability (before applying softmax)
std::vector<float> expected_vals = {-5.34957457f, 13.1904755f, -4.79670954f, -3.59232116f, 2.31260920f,
-4.27866220f, -4.31867933f, 0.587718308f, -2.33952785f, -3.88515306f};
const ORTCHAR_T* lib_name;
#if defined(_WIN32)
lib_name = ORT_TSTR("custom_op_openvino_wrapper_library.dll");
#elif defined(__APPLE__)
lib_name = ORT_TSTR("libcustom_op_openvino_wrapper_library.dylib");
#else
lib_name = ORT_TSTR("./libcustom_op_openvino_wrapper_library.so");
#endif
Add logging APIs for custom operators (#14416) ### Description Add logging APIs for custom ops. This PR introduces a `OrtLogger` type, which can be retrieved from a `OrtKernelInfo` or `OrtKernelContext`. The kernel info's logger is the session logger stored in the execution provider. The kernel context's logger is a run logger. ### Motivation and Context Allows custom ops to log information in a manner consistent with built-in ops. Example usage in custom op: ```C++ struct MyCustomKernel { MyCustomKernel(const OrtApi& api, const OrtKernelInfo* info) { Ort::ConstKernelInfo kinfo(info); this->logger_ = kinfo.GetLogger(); // ... ORT_CXX_LOGF_NOEXCEPT(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_ERROR, "Error: %s", err_msg); } void Compute(OrtKernelContext* context) { ORT_CXX_LOG(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_VERBOSE, "Calling compute..."); // ... } // ... private: Ort::Logger logger_; }; ```
2023-03-17 22:05:28 +00:00
// Run with custom op session configurations.
{
Ort::SessionOptions session_opts;
Ort::CustomOpConfigs custom_op_configs;
Custom Op runtime wrapper (#13427) ### Description Adds the below C APIs to support custom ops that wrap an entire model to be inferenced with an external runtime. The current SNPE EP is an example of an EP that could be ported to use a custom op wrapper. Ex: The custom op stores the serialized SNPE DLC binary as a string attribute. The SNPE model is built when the kernel is created. The model is inferenced with SNPE APIs on call to the kernel's compute method. #### C APIs | API | Description | Why | | --- | --- | --- | | `KernelInfo_GetInputCount` | Gets number of inputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputCount` | Gets number of outputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputName` | Gets an input's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputName` | Gets an output's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputTypeInfo` | Gets the type/shape information for an input. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputTypeInfo` | Gets the type/shape information for an output. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfoGetAttribute_tensor` | Get a OrtValue tensor stored as an attribute in the graph node | Extract serialized models, weights, etc. | | `GetSessionConfigEntry` | Get a session configuration value | Need to be able to get session-time configurations from within custom op | | `HasSessionConfigEntry` | Check if session configuration entry exists. | Need to be able to get session-time configurations from within custom op | #### Why so many KernelInfo APIs?<sup>1</sup> Similar APIs currently exist for `OrtKernelContext`, but not `OrtKernelInfo`. Note that `OrtKernelContext` is passed to the custom op on call to its kernel's compute() function. However, `OrtKernelInfo` is available on kernel creation, which occurs when the session is created. Having these APIs available from `OrtKernelInfo` allows an operator to trade-off computation time for session-creation time, and vice versa. Operators that must build expensive state may prefer to do it during session creation time instead of compute-time. SNPE is an example of an EP that needs to be able to query `KernelInfo` for the name, type, and shape of inputs and outputs in order to build the model from the serialized DLC data. This is an expensive operation. Other providers (e.g., OpenVINO) are able to query i/o info from the serialized model, so they do not strictly need these APIs. However, the APIs can still be used to validate the expected I/O characteristics. Additionally, several of our CPU contrib ops currently use the same internal version of these KernelInfo APIs (Ex: [qlinear_softmax](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/quantization/qlinear_softmax.cc#L71)). If custom ops are also meant to be a test bed for future ops, then all custom ops (not just runtime wrappers) would benefit from the addition of these public KernelInfo APIs (IMO). #### Example of usage in a custom OP From `onnxruntime/test/testdata/custom_op_openvino_wrapper_library/openvino_wrapper.h` ```c++ struct CustomOpOpenVINO : Ort::CustomOpBase<CustomOpOpenVINO, KernelOpenVINO> { explicit CustomOpOpenVINO(Ort::ConstSessionOptions session_options); CustomOpOpenVINO(const CustomOpOpenVINO&) = delete; CustomOpOpenVINO& operator=(const CustomOpOpenVINO&) = delete; void* CreateKernel(const OrtApi& api, const OrtKernelInfo* info) const; constexpr const char* GetName() const noexcept { return "OpenVINO_Wrapper"; } constexpr const char* GetExecutionProviderType() const noexcept { return "CPUExecutionProvider"; } // IMPORTANT: In order to wrap a generic runtime-specific model, the custom operator // must have a non-homogeneous variadic input and output. constexpr size_t GetInputTypeCount() const noexcept { return 1; } constexpr size_t GetOutputTypeCount() const noexcept { return 1; } constexpr ONNXTensorElementDataType GetInputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr ONNXTensorElementDataType GetOutputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr OrtCustomOpInputOutputCharacteristic GetInputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr OrtCustomOpInputOutputCharacteristic GetOutputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr bool GetVariadicInputHomogeneity() const noexcept { return false; // heterogenous } constexpr bool GetVariadicOutputHomogeneity() const noexcept { return false; // heterogeneous } std::vector<std::string> GetSessionConfigKeys() const { return {"device_type"}; } private: std::unordered_map<std::string, std::string> session_configs_; }; ``` #### How to create a session: ```c++ Ort::Env env; Ort::SessionOptions session_opts; Ort::CustomOpConfigs custom_op_configs; // Create local session config entries for the custom op. custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU"); // Register custom op library and pass in the custom op configs (optional). session_opts.RegisterCustomOpsLibrary(lib_name, custom_op_configs); Ort::Session session(env, model_path.data(), session_opts); ``` ### Motivation and Context Allows creation of simple "wrapper" EPs outside of the main ORT code base.
2023-01-18 17:09:32 +00:00
Add logging APIs for custom operators (#14416) ### Description Add logging APIs for custom ops. This PR introduces a `OrtLogger` type, which can be retrieved from a `OrtKernelInfo` or `OrtKernelContext`. The kernel info's logger is the session logger stored in the execution provider. The kernel context's logger is a run logger. ### Motivation and Context Allows custom ops to log information in a manner consistent with built-in ops. Example usage in custom op: ```C++ struct MyCustomKernel { MyCustomKernel(const OrtApi& api, const OrtKernelInfo* info) { Ort::ConstKernelInfo kinfo(info); this->logger_ = kinfo.GetLogger(); // ... ORT_CXX_LOGF_NOEXCEPT(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_ERROR, "Error: %s", err_msg); } void Compute(OrtKernelContext* context) { ORT_CXX_LOG(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_VERBOSE, "Calling compute..."); // ... } // ... private: Ort::Logger logger_; }; ```
2023-03-17 22:05:28 +00:00
custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU");
session_opts.RegisterCustomOpsLibrary(lib_name, custom_op_configs);
Custom Op runtime wrapper (#13427) ### Description Adds the below C APIs to support custom ops that wrap an entire model to be inferenced with an external runtime. The current SNPE EP is an example of an EP that could be ported to use a custom op wrapper. Ex: The custom op stores the serialized SNPE DLC binary as a string attribute. The SNPE model is built when the kernel is created. The model is inferenced with SNPE APIs on call to the kernel's compute method. #### C APIs | API | Description | Why | | --- | --- | --- | | `KernelInfo_GetInputCount` | Gets number of inputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputCount` | Gets number of outputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputName` | Gets an input's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputName` | Gets an output's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputTypeInfo` | Gets the type/shape information for an input. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputTypeInfo` | Gets the type/shape information for an output. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfoGetAttribute_tensor` | Get a OrtValue tensor stored as an attribute in the graph node | Extract serialized models, weights, etc. | | `GetSessionConfigEntry` | Get a session configuration value | Need to be able to get session-time configurations from within custom op | | `HasSessionConfigEntry` | Check if session configuration entry exists. | Need to be able to get session-time configurations from within custom op | #### Why so many KernelInfo APIs?<sup>1</sup> Similar APIs currently exist for `OrtKernelContext`, but not `OrtKernelInfo`. Note that `OrtKernelContext` is passed to the custom op on call to its kernel's compute() function. However, `OrtKernelInfo` is available on kernel creation, which occurs when the session is created. Having these APIs available from `OrtKernelInfo` allows an operator to trade-off computation time for session-creation time, and vice versa. Operators that must build expensive state may prefer to do it during session creation time instead of compute-time. SNPE is an example of an EP that needs to be able to query `KernelInfo` for the name, type, and shape of inputs and outputs in order to build the model from the serialized DLC data. This is an expensive operation. Other providers (e.g., OpenVINO) are able to query i/o info from the serialized model, so they do not strictly need these APIs. However, the APIs can still be used to validate the expected I/O characteristics. Additionally, several of our CPU contrib ops currently use the same internal version of these KernelInfo APIs (Ex: [qlinear_softmax](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/quantization/qlinear_softmax.cc#L71)). If custom ops are also meant to be a test bed for future ops, then all custom ops (not just runtime wrappers) would benefit from the addition of these public KernelInfo APIs (IMO). #### Example of usage in a custom OP From `onnxruntime/test/testdata/custom_op_openvino_wrapper_library/openvino_wrapper.h` ```c++ struct CustomOpOpenVINO : Ort::CustomOpBase<CustomOpOpenVINO, KernelOpenVINO> { explicit CustomOpOpenVINO(Ort::ConstSessionOptions session_options); CustomOpOpenVINO(const CustomOpOpenVINO&) = delete; CustomOpOpenVINO& operator=(const CustomOpOpenVINO&) = delete; void* CreateKernel(const OrtApi& api, const OrtKernelInfo* info) const; constexpr const char* GetName() const noexcept { return "OpenVINO_Wrapper"; } constexpr const char* GetExecutionProviderType() const noexcept { return "CPUExecutionProvider"; } // IMPORTANT: In order to wrap a generic runtime-specific model, the custom operator // must have a non-homogeneous variadic input and output. constexpr size_t GetInputTypeCount() const noexcept { return 1; } constexpr size_t GetOutputTypeCount() const noexcept { return 1; } constexpr ONNXTensorElementDataType GetInputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr ONNXTensorElementDataType GetOutputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr OrtCustomOpInputOutputCharacteristic GetInputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr OrtCustomOpInputOutputCharacteristic GetOutputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr bool GetVariadicInputHomogeneity() const noexcept { return false; // heterogenous } constexpr bool GetVariadicOutputHomogeneity() const noexcept { return false; // heterogeneous } std::vector<std::string> GetSessionConfigKeys() const { return {"device_type"}; } private: std::unordered_map<std::string, std::string> session_configs_; }; ``` #### How to create a session: ```c++ Ort::Env env; Ort::SessionOptions session_opts; Ort::CustomOpConfigs custom_op_configs; // Create local session config entries for the custom op. custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU"); // Register custom op library and pass in the custom op configs (optional). session_opts.RegisterCustomOpsLibrary(lib_name, custom_op_configs); Ort::Session session(env, model_path.data(), session_opts); ``` ### Motivation and Context Allows creation of simple "wrapper" EPs outside of the main ORT code base.
2023-01-18 17:09:32 +00:00
Add logging APIs for custom operators (#14416) ### Description Add logging APIs for custom ops. This PR introduces a `OrtLogger` type, which can be retrieved from a `OrtKernelInfo` or `OrtKernelContext`. The kernel info's logger is the session logger stored in the execution provider. The kernel context's logger is a run logger. ### Motivation and Context Allows custom ops to log information in a manner consistent with built-in ops. Example usage in custom op: ```C++ struct MyCustomKernel { MyCustomKernel(const OrtApi& api, const OrtKernelInfo* info) { Ort::ConstKernelInfo kinfo(info); this->logger_ = kinfo.GetLogger(); // ... ORT_CXX_LOGF_NOEXCEPT(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_ERROR, "Error: %s", err_msg); } void Compute(OrtKernelContext* context) { ORT_CXX_LOG(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_VERBOSE, "Calling compute..."); // ... } // ... private: Ort::Logger logger_; }; ```
2023-03-17 22:05:28 +00:00
Ort::Session session(*ort_env, CUSTOM_OP_OPENVINO_WRAPPER_LIB_TEST_MODEL_URI, session_opts);
auto default_allocator = std::make_unique<MockedOrtAllocator>();
Custom Op runtime wrapper (#13427) ### Description Adds the below C APIs to support custom ops that wrap an entire model to be inferenced with an external runtime. The current SNPE EP is an example of an EP that could be ported to use a custom op wrapper. Ex: The custom op stores the serialized SNPE DLC binary as a string attribute. The SNPE model is built when the kernel is created. The model is inferenced with SNPE APIs on call to the kernel's compute method. #### C APIs | API | Description | Why | | --- | --- | --- | | `KernelInfo_GetInputCount` | Gets number of inputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputCount` | Gets number of outputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputName` | Gets an input's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputName` | Gets an output's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputTypeInfo` | Gets the type/shape information for an input. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputTypeInfo` | Gets the type/shape information for an output. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfoGetAttribute_tensor` | Get a OrtValue tensor stored as an attribute in the graph node | Extract serialized models, weights, etc. | | `GetSessionConfigEntry` | Get a session configuration value | Need to be able to get session-time configurations from within custom op | | `HasSessionConfigEntry` | Check if session configuration entry exists. | Need to be able to get session-time configurations from within custom op | #### Why so many KernelInfo APIs?<sup>1</sup> Similar APIs currently exist for `OrtKernelContext`, but not `OrtKernelInfo`. Note that `OrtKernelContext` is passed to the custom op on call to its kernel's compute() function. However, `OrtKernelInfo` is available on kernel creation, which occurs when the session is created. Having these APIs available from `OrtKernelInfo` allows an operator to trade-off computation time for session-creation time, and vice versa. Operators that must build expensive state may prefer to do it during session creation time instead of compute-time. SNPE is an example of an EP that needs to be able to query `KernelInfo` for the name, type, and shape of inputs and outputs in order to build the model from the serialized DLC data. This is an expensive operation. Other providers (e.g., OpenVINO) are able to query i/o info from the serialized model, so they do not strictly need these APIs. However, the APIs can still be used to validate the expected I/O characteristics. Additionally, several of our CPU contrib ops currently use the same internal version of these KernelInfo APIs (Ex: [qlinear_softmax](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/quantization/qlinear_softmax.cc#L71)). If custom ops are also meant to be a test bed for future ops, then all custom ops (not just runtime wrappers) would benefit from the addition of these public KernelInfo APIs (IMO). #### Example of usage in a custom OP From `onnxruntime/test/testdata/custom_op_openvino_wrapper_library/openvino_wrapper.h` ```c++ struct CustomOpOpenVINO : Ort::CustomOpBase<CustomOpOpenVINO, KernelOpenVINO> { explicit CustomOpOpenVINO(Ort::ConstSessionOptions session_options); CustomOpOpenVINO(const CustomOpOpenVINO&) = delete; CustomOpOpenVINO& operator=(const CustomOpOpenVINO&) = delete; void* CreateKernel(const OrtApi& api, const OrtKernelInfo* info) const; constexpr const char* GetName() const noexcept { return "OpenVINO_Wrapper"; } constexpr const char* GetExecutionProviderType() const noexcept { return "CPUExecutionProvider"; } // IMPORTANT: In order to wrap a generic runtime-specific model, the custom operator // must have a non-homogeneous variadic input and output. constexpr size_t GetInputTypeCount() const noexcept { return 1; } constexpr size_t GetOutputTypeCount() const noexcept { return 1; } constexpr ONNXTensorElementDataType GetInputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr ONNXTensorElementDataType GetOutputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr OrtCustomOpInputOutputCharacteristic GetInputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr OrtCustomOpInputOutputCharacteristic GetOutputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr bool GetVariadicInputHomogeneity() const noexcept { return false; // heterogenous } constexpr bool GetVariadicOutputHomogeneity() const noexcept { return false; // heterogeneous } std::vector<std::string> GetSessionConfigKeys() const { return {"device_type"}; } private: std::unordered_map<std::string, std::string> session_configs_; }; ``` #### How to create a session: ```c++ Ort::Env env; Ort::SessionOptions session_opts; Ort::CustomOpConfigs custom_op_configs; // Create local session config entries for the custom op. custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU"); // Register custom op library and pass in the custom op configs (optional). session_opts.RegisterCustomOpsLibrary(lib_name, custom_op_configs); Ort::Session session(env, model_path.data(), session_opts); ``` ### Motivation and Context Allows creation of simple "wrapper" EPs outside of the main ORT code base.
2023-01-18 17:09:32 +00:00
Add logging APIs for custom operators (#14416) ### Description Add logging APIs for custom ops. This PR introduces a `OrtLogger` type, which can be retrieved from a `OrtKernelInfo` or `OrtKernelContext`. The kernel info's logger is the session logger stored in the execution provider. The kernel context's logger is a run logger. ### Motivation and Context Allows custom ops to log information in a manner consistent with built-in ops. Example usage in custom op: ```C++ struct MyCustomKernel { MyCustomKernel(const OrtApi& api, const OrtKernelInfo* info) { Ort::ConstKernelInfo kinfo(info); this->logger_ = kinfo.GetLogger(); // ... ORT_CXX_LOGF_NOEXCEPT(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_ERROR, "Error: %s", err_msg); } void Compute(OrtKernelContext* context) { ORT_CXX_LOG(this->logger_, OrtLoggingLevel::ORT_LOGGING_LEVEL_VERBOSE, "Calling compute..."); // ... } // ... private: Ort::Logger logger_; }; ```
2023-03-17 22:05:28 +00:00
RunSession(default_allocator.get(), session,
inputs,
"Plus214_Output_0",
expected_output_dims,
expected_vals,
nullptr);
}
// Run without specifying any custom op session configurations.
// Expect custom op to use "CPU" as OpenVINO's default backend.
{
Ort::SessionOptions session_opts;
session_opts.RegisterCustomOpsLibrary(lib_name);
Ort::Session session(*ort_env, CUSTOM_OP_OPENVINO_WRAPPER_LIB_TEST_MODEL_URI, session_opts);
auto default_allocator = std::make_unique<MockedOrtAllocator>();
RunSession(default_allocator.get(), session,
inputs,
"Plus214_Output_0",
expected_output_dims,
expected_vals,
nullptr);
}
Custom Op runtime wrapper (#13427) ### Description Adds the below C APIs to support custom ops that wrap an entire model to be inferenced with an external runtime. The current SNPE EP is an example of an EP that could be ported to use a custom op wrapper. Ex: The custom op stores the serialized SNPE DLC binary as a string attribute. The SNPE model is built when the kernel is created. The model is inferenced with SNPE APIs on call to the kernel's compute method. #### C APIs | API | Description | Why | | --- | --- | --- | | `KernelInfo_GetInputCount` | Gets number of inputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputCount` | Gets number of outputs from `OrtKernelInfo`. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputName` | Gets an input's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputName` | Gets an output's name. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetInputTypeInfo` | Gets the type/shape information for an input. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfo_GetOutputTypeInfo` | Gets the type/shape information for an output. | Query I/O characteristics during kernel creation<sup>1</sup> | | `KernelInfoGetAttribute_tensor` | Get a OrtValue tensor stored as an attribute in the graph node | Extract serialized models, weights, etc. | | `GetSessionConfigEntry` | Get a session configuration value | Need to be able to get session-time configurations from within custom op | | `HasSessionConfigEntry` | Check if session configuration entry exists. | Need to be able to get session-time configurations from within custom op | #### Why so many KernelInfo APIs?<sup>1</sup> Similar APIs currently exist for `OrtKernelContext`, but not `OrtKernelInfo`. Note that `OrtKernelContext` is passed to the custom op on call to its kernel's compute() function. However, `OrtKernelInfo` is available on kernel creation, which occurs when the session is created. Having these APIs available from `OrtKernelInfo` allows an operator to trade-off computation time for session-creation time, and vice versa. Operators that must build expensive state may prefer to do it during session creation time instead of compute-time. SNPE is an example of an EP that needs to be able to query `KernelInfo` for the name, type, and shape of inputs and outputs in order to build the model from the serialized DLC data. This is an expensive operation. Other providers (e.g., OpenVINO) are able to query i/o info from the serialized model, so they do not strictly need these APIs. However, the APIs can still be used to validate the expected I/O characteristics. Additionally, several of our CPU contrib ops currently use the same internal version of these KernelInfo APIs (Ex: [qlinear_softmax](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/quantization/qlinear_softmax.cc#L71)). If custom ops are also meant to be a test bed for future ops, then all custom ops (not just runtime wrappers) would benefit from the addition of these public KernelInfo APIs (IMO). #### Example of usage in a custom OP From `onnxruntime/test/testdata/custom_op_openvino_wrapper_library/openvino_wrapper.h` ```c++ struct CustomOpOpenVINO : Ort::CustomOpBase<CustomOpOpenVINO, KernelOpenVINO> { explicit CustomOpOpenVINO(Ort::ConstSessionOptions session_options); CustomOpOpenVINO(const CustomOpOpenVINO&) = delete; CustomOpOpenVINO& operator=(const CustomOpOpenVINO&) = delete; void* CreateKernel(const OrtApi& api, const OrtKernelInfo* info) const; constexpr const char* GetName() const noexcept { return "OpenVINO_Wrapper"; } constexpr const char* GetExecutionProviderType() const noexcept { return "CPUExecutionProvider"; } // IMPORTANT: In order to wrap a generic runtime-specific model, the custom operator // must have a non-homogeneous variadic input and output. constexpr size_t GetInputTypeCount() const noexcept { return 1; } constexpr size_t GetOutputTypeCount() const noexcept { return 1; } constexpr ONNXTensorElementDataType GetInputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr ONNXTensorElementDataType GetOutputType(size_t /* index */) const noexcept { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } constexpr OrtCustomOpInputOutputCharacteristic GetInputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr OrtCustomOpInputOutputCharacteristic GetOutputCharacteristic(size_t /* index */) const noexcept { return INPUT_OUTPUT_VARIADIC; } constexpr bool GetVariadicInputHomogeneity() const noexcept { return false; // heterogenous } constexpr bool GetVariadicOutputHomogeneity() const noexcept { return false; // heterogeneous } std::vector<std::string> GetSessionConfigKeys() const { return {"device_type"}; } private: std::unordered_map<std::string, std::string> session_configs_; }; ``` #### How to create a session: ```c++ Ort::Env env; Ort::SessionOptions session_opts; Ort::CustomOpConfigs custom_op_configs; // Create local session config entries for the custom op. custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU"); // Register custom op library and pass in the custom op configs (optional). session_opts.RegisterCustomOpsLibrary(lib_name, custom_op_configs); Ort::Session session(env, model_path.data(), session_opts); ``` ### Motivation and Context Allows creation of simple "wrapper" EPs outside of the main ORT code base.
2023-01-18 17:09:32 +00:00
}
#endif // defined(USE_OPENVINO) && (!defined(ORT_MINIMAL_BUILD) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS))
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// It has memory leak. The OrtCustomOpDomain created in custom_op_library.cc:RegisterCustomOps function was not freed
Re-enable custom op shared library test for debug builds (#5475)
2020-10-20 00:14:31 +00:00
#if defined(__ANDROID__)
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
TEST(CApiTest, DISABLED_test_custom_op_library) {
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// To accomodate a reduced op build pipeline
Fix a Win GPU reduced ops pipeline (#10202)
2022-01-06 17:46:34 +00:00
#elif defined(REDUCED_OPS_BUILD) && defined(USE_CUDA)
TEST(CApiTest, DISABLED_test_custom_op_library) {
Move model tests to onnxruntime_test_all (#4521) 1. Move model tests to onnxruntime_test_all 2. Publish TestResults of Windows CI build.
2020-07-15 23:46:18 +00:00
#else
TEST(CApiTest, test_custom_op_library) {
Fix static analysis warnings found by VC++ (#3530) 1. Fix static analysis warnings found by VC++ 2. Add a new pipeline for static analysis 3. Merge all the windows CI build into one single yaml file.(Easier to queue them all). 4. Make DNNL build faster by disabling building the tests and examples. 5. Enable custom op unitest.
2020-04-16 08:46:47 +00:00
#endif
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
std::cout << "Running inference using custom op shared library" << std::endl;
std::vector<Input> inputs(2);
inputs[0].name = "input_1";
inputs[0].dims = {3, 5};
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
inputs[0].values = {1.1f, 2.2f, 3.3f, 4.4f, 5.5f,
6.6f, 7.7f, 8.8f, 9.9f, 10.0f,
11.1f, 12.2f, 13.3f, 14.4f, 15.5f};
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
inputs[1].name = "input_2";
inputs[1].dims = {3, 5};
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
inputs[1].values = {15.5f, 14.4f, 13.3f, 12.2f, 11.1f,
10.0f, 9.9f, 8.8f, 7.7f, 6.6f,
5.5f, 4.4f, 3.3f, 2.2f, 1.1f};
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {3, 5};
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
std::vector<int32_t> expected_values_y =
{17, 17, 17, 17, 17,
17, 18, 18, 18, 17,
17, 17, 17, 17, 17};
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
onnxruntime::PathString lib_name;
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
#if defined(_WIN32)
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
lib_name = ORT_TSTR("custom_op_library.dll");
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
#elif defined(__APPLE__)
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
lib_name = ORT_TSTR("libcustom_op_library.dylib");
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
#else
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
lib_name = ORT_TSTR("./libcustom_op_library.so");
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
#endif
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
Add example of registering custom cuda op as shared lib (#10025)
2022-01-05 17:22:15 +00:00
#ifdef USE_CUDA
TestInference<int32_t>(*ort_env, CUSTOM_OP_LIBRARY_TEST_MODEL_URI, inputs, "output", expected_dims_y,
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
expected_values_y, 1, nullptr, lib_name.c_str());
EP context for custom op (#16454) Implement infrastructures to allow EP resources surfaced to custom ops. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-08-16 20:03:40 +00:00
#elif USE_ROCM
TestInference<int32_t>(*ort_env, CUSTOM_OP_LIBRARY_TEST_MODEL_URI, inputs, "output", expected_dims_y,
expected_values_y, 3, nullptr, lib_name.c_str());
Add example of registering custom cuda op as shared lib (#10025)
2022-01-05 17:22:15 +00:00
#else
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<int32_t>(*ort_env, CUSTOM_OP_LIBRARY_TEST_MODEL_URI, inputs, "output", expected_dims_y,
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
expected_values_y, 0, nullptr, lib_name.c_str());
Add example of registering custom cuda op as shared lib (#10025)
2022-01-05 17:22:15 +00:00
#endif
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
}
Re-enable custom op shared library test for debug builds (#5475)
2020-10-20 00:14:31 +00:00
Enable Address Sanitizer in CI (#19073) ### Description 1. Add two build jobs for enabling Address Sanitizer in CI. One for Windows CPU, One for Linux CPU. 2. Set default compiler flags/linker flags in build.py for normal Windows/Linux/MacOS build. This can help control compiler flags in a more centralized way. 3. All Windows binaries in our official packages will be built with "/PROFILE" flag. Symbols of onnxruntime.dll can be found at [Microsoft public symbol server](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/microsoft-public-symbols). Limitations: 1. On Linux Address Sanitizer ignores RPATH settings in ELF binaries. Therefore once Address Sanitizer is enabled, before running tests we need to manually set LD_LIBRARY_PATH properly otherwise libonnxruntime.so may not be able to find custom ops and shared EPs. 4. On Linux we also need to set LD_PRELOAD before running some tests(if the main executable, like python, is not built with address sanitizer. On Windows we do not need to. 5. On Windows before running python tests we should manually copy address sanitizer DLL to the onnxruntime/capi directory, because python 3.8 and above has enabled "Safe DLL Search Mode" that wouldn't use the information provided by PATH env. 6. On Linux Address Sanitizer found a lot of memory leaks from our python binding code. Therefore right now we cannot enable Address Sanitizer when building ONNX Runtime with python binding. 7. Address Sanitizer itself uses a lot of memory address space and delays memory deallocations, which is easy to cause OOM issues in 32-bit applications. We cannot run all the tests in onnxruntime_test_all in 32-bit mode with Address Sanitizer due to this reason. However, we still can run individual tests in such a way. We just cannot run all of them in one process. ### Motivation and Context To catch memory issues.
2024-01-12 15:24:40 +00:00
// Has memory leak
#if defined(__ANDROID__) || defined(ABSL_HAVE_ADDRESS_SANITIZER)
Custom op shape inference API (#17737) Add c/cxx API to allow custom ops do shape inference. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-10-13 19:57:42 +00:00
TEST(CApiTest, DISABLED_test_custom_op_shape_infer_attr) {
// To accomodate a reduced op build pipeline
#elif defined(REDUCED_OPS_BUILD) && defined(USE_CUDA)
TEST(CApiTest, DISABLED_test_custom_op_shape_infer_attr) {
#else
TEST(CApiTest, test_custom_op_shape_infer_attr) {
#endif
std::vector<Input> inputs(1);
inputs[0].name = "input_0";
inputs[0].dims = {5};
inputs[0].values = {1.f, 2.f, 3.f, 4.f, 5.f};
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {5};
std::vector<float> expected_values_y = {6.f, 12.f, 18.f, 24.f, 30.f};
onnxruntime::PathString lib_name;
#if defined(_WIN32)
lib_name = ORT_TSTR("custom_op_library.dll");
#elif defined(__APPLE__)
lib_name = ORT_TSTR("libcustom_op_library.dylib");
#else
lib_name = ORT_TSTR("./libcustom_op_library.so");
#endif
TestInference<float>(*ort_env, CUSTOM_OP_LIBRARY_ATTR_TESTER_URI, inputs, "output_0", expected_dims_y,
expected_values_y, 0, nullptr, lib_name.c_str());
}
Patch lite custom op API (#17605) A few enhancements: - Support compute returning status; - Support variadic; --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-09-26 21:02:18 +00:00
// It has memory leak. The OrtCustomOpDomain created in custom_op_library.cc:RegisterCustomOps function was not freed
#if defined(__ANDROID__)
TEST(CApiTest, test_custom_op_library_copy_variadic) {
// To accomodate a reduced op build pipeline
#elif defined(REDUCED_OPS_BUILD) && defined(USE_CUDA)
TEST(CApiTest, test_custom_op_library_copy_variadic) {
#else
TEST(CApiTest, test_custom_op_library_copy_variadic) {
#endif
std::cout << "Running inference using custom op shared library" << std::endl;
std::vector<Input> inputs(2);
inputs[0].name = "input_0";
inputs[0].dims = {15};
inputs[0].values = {1.1f, 2.2f, 3.3f, 4.4f, 5.5f,
6.6f, 7.7f, 8.8f, 9.9f, 10.0f,
11.1f, 12.2f, 13.3f, 14.4f, 15.5f};
inputs[1].name = "input_1";
inputs[1].dims = {15};
inputs[1].values = {15.5f, 14.4f, 13.3f, 12.2f, 11.1f,
10.0f, 9.9f, 8.8f, 7.7f, 6.6f,
5.5f, 4.4f, 3.3f, 2.2f, 1.1f};
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {15};
std::vector<float> expected_values_y = inputs[1].values;
onnxruntime::PathString lib_name;
#if defined(_WIN32)
lib_name = ORT_TSTR("custom_op_library.dll");
#elif defined(__APPLE__)
lib_name = ORT_TSTR("libcustom_op_library.dylib");
#else
lib_name = ORT_TSTR("./libcustom_op_library.so");
#endif
TestInference<float>(*ort_env, CUSTOM_OP_LIBRARY_COPY_TENSOR_ARRAY_2,
inputs, "output_1", expected_dims_y,
expected_values_y, 0, nullptr, lib_name.c_str());
inputs.push_back({});
inputs[2].name = "input_2";
inputs[2].dims = {15};
inputs[2].values = {6.6f, 7.7f, 8.8f, 9.9f, 10.0f,
1.1f, 2.2f, 3.3f, 4.4f, 5.5f,
11.1f, 12.2f, 13.3f, 14.4f, 15.5f};
expected_values_y = inputs[2].values;
TestInference<float>(*ort_env, CUSTOM_OP_LIBRARY_COPY_TENSOR_ARRAY_3,
inputs, "output_2", expected_dims_y,
expected_values_y, 0, nullptr, lib_name.c_str());
}
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
#if !defined(DISABLE_FLOAT8_TYPES)
struct InputF8 {
const char* name = nullptr;
std::vector<int64_t> dims;
std::vector<Ort::Float8E4M3FN_t> values;
};
// See test test_custom_op_library_float8.
#if defined(__ANDROID__)
TEST(CApiTest, DISABLED_test_custom_op_library_float8) {
#elif defined(REDUCED_OPS_BUILD) && defined(USE_CUDA)
TEST(CApiTest, DISABLED_test_custom_op_library_float8) {
#else
TEST(CApiTest, test_custom_op_library_float8) {
#endif
std::cout << "Running inference using custom op shared library" << std::endl;
std::vector<InputF8> inputs(2);
inputs[0].name = "X";
inputs[0].dims = {2};
inputs[0].values = {0, 1};
inputs[1].name = "Y";
inputs[1].dims = {2};
inputs[1].values = {3, 4};
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {2};
std::vector<Ort::Float8E4M3FN_t> expected_values_y = {0, 1};
onnxruntime::PathString lib_name;
#if defined(_WIN32)
lib_name = ORT_TSTR("custom_op_library.dll");
#elif defined(__APPLE__)
lib_name = ORT_TSTR("libcustom_op_library.dylib");
#else
lib_name = ORT_TSTR("./libcustom_op_library.so");
#endif
TestInference<Ort::Float8E4M3FN_t, Ort::Float8E4M3FN_t, InputF8>(*ort_env, CUSTOM_OP_LIBRARY_TEST_MODEL_FLOAT8_URI, inputs,
"Z", expected_dims_y,
expected_values_y, 0, nullptr, lib_name.c_str());
}
#endif
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
#if !defined(ORT_MINIMAL_BUILD) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS)
#if defined(__ANDROID__)
// Disable on android because custom op libraries are not copied to the emulator.
TEST(CApiTest, DISABLED_test_custom_op_library_registration_error) {
#else
TEST(CApiTest, test_custom_op_library_registration_error) {
#endif // defined(__ANDROID__)
// Loads a custom op library with a RegisterCustomOps function that returns an error status.
// This test tries to register the library with the session options and expects an error.
const ORTCHAR_T* lib_name;
#if defined(_WIN32)
lib_name = ORT_TSTR("custom_op_invalid_library.dll");
#elif defined(__APPLE__)
lib_name = ORT_TSTR("libcustom_op_invalid_library.dylib");
Re-enable custom op shared library test for debug builds (#5475)
2020-10-20 00:14:31 +00:00
#else
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
lib_name = ORT_TSTR("./libcustom_op_invalid_library.so");
Re-enable custom op shared library test for debug builds (#5475)
2020-10-20 00:14:31 +00:00
#endif
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
Ort::SessionOptions session_options;
try {
session_options.RegisterCustomOpsLibrary(lib_name);
FAIL();
} catch (const Ort::Exception& exception) {
ASSERT_THAT(exception.what(), testing::HasSubstr("Failure from custom op library's RegisterCustomOps()"));
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
}
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
#endif // !defined(ORT_MINIMAL_BUILD) || defined(ORT_MINIMAL_BUILD_CUSTOM_OPS)
CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library.
2019-11-21 23:45:49 +00:00
FixPyOpSegFault&MakeItStaticLib (#4600) * remove pyop wrapper * add py threading logic * fix doc * fix doc * fix doc * format doc * format doc * format doc * reenable test Co-authored-by: RandySheriffH <rashuai@microsoft.com>
2020-07-28 18:45:25 +00:00
#if defined(ENABLE_LANGUAGE_INTEROP_OPS)
EnrichPyOpUT (#4681) * cancel night build on pyop * enrich PyOp UTs * init script only once * remove space * update models * Show usage of kwargs in doc
2020-08-05 21:11:56 +00:00
std::once_flag my_module_flag;
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
void PrepareModule() {
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
std::ofstream module("mymodule.py");
module << "class MyKernel:" << std::endl;
Simplify some CustomOp code (#1206) test_inference.cc reformatted with clang-format
2019-06-12 00:00:04 +00:00
module << "\t"
<< "def __init__(self,A,B,C):" << std::endl;
module << "\t\t"
<< "self.a,self.b,self.c = A,B,C" << std::endl;
module << "\t"
<< "def compute(self,x):" << std::endl;
module << "\t\t"
<< "return x*2" << std::endl;
EnrichPyOpUT (#4681) * cancel night build on pyop * enrich PyOp UTs * init script only once * remove space * update models * Show usage of kwargs in doc
2020-08-05 21:11:56 +00:00
module << "class MyKernel_2:" << std::endl;
module << "\t"
<< "def __init__(self,A,B):" << std::endl;
module << "\t\t"
<< "self.a,self.b = A,B" << std::endl;
module << "\t"
<< "def compute(self,x):" << std::endl;
module << "\t\t"
<< "return x*4" << std::endl;
module << "class MyKernel_3:" << std::endl;
module << "\t"
<< "def __init__(self,A,B):" << std::endl;
module << "\t\t"
<< "self.a,self.b = A,B" << std::endl;
module << "\t"
<< "def compute(self,*kwargs):" << std::endl;
module << "\t\t"
<< "return kwargs[0]*5" << std::endl;
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
module.close();
EnrichPyOpUT (#4681) * cancel night build on pyop * enrich PyOp UTs * init script only once * remove space * update models * Show usage of kwargs in doc
2020-08-05 21:11:56 +00:00
}
TEST(CApiTest, test_pyop) {
std::call_once(my_module_flag, PrepareModule);
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
std::vector<Input> inputs(1);
Input& input = inputs[0];
input.name = "X";
input.dims = {2, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f};
std::vector<int64_t> expected_dims_y = {2, 2};
std::vector<float> expected_values_y = {2.0f, 4.0f, 6.0f, 8.0f};
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, PYOP_FLOAT_MODEL_URI, inputs, "Y", expected_dims_y, expected_values_y, 0,
nullptr, nullptr);
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
}
EnrichPyOpUT (#4681) * cancel night build on pyop * enrich PyOp UTs * init script only once * remove space * update models * Show usage of kwargs in doc
2020-08-05 21:11:56 +00:00
TEST(CApiTest, test_pyop_multi) {
std::call_once(my_module_flag, PrepareModule);
std::vector<Input> inputs(1);
Input& input = inputs[0];
input.name = "X";
input.dims = {2, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f};
std::vector<int64_t> expected_dims_y = {2, 2};
std::vector<float> expected_values_y = {8.0f, 16.0f, 24.0f, 32.0f};
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, PYOP_MULTI_MODEL_URI, inputs, "Z", expected_dims_y, expected_values_y, 0,
nullptr, nullptr);
EnrichPyOpUT (#4681) * cancel night build on pyop * enrich PyOp UTs * init script only once * remove space * update models * Show usage of kwargs in doc
2020-08-05 21:11:56 +00:00
}
TEST(CApiTest, test_pyop_kwarg) {
std::call_once(my_module_flag, PrepareModule);
std::vector<Input> inputs(1);
Input& input = inputs[0];
input.name = "X";
input.dims = {2, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f};
std::vector<int64_t> expected_dims_y = {2, 2};
std::vector<float> expected_values_y = {25.0f, 50.0f, 75.0f, 100.0f};
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, PYOP_KWARG_MODEL_URI, inputs, "Z", expected_dims_y, expected_values_y, 0,
nullptr, nullptr);
EnrichPyOpUT (#4681) * cancel night build on pyop * enrich PyOp UTs * init script only once * remove space * update models * Show usage of kwargs in doc
2020-08-05 21:11:56 +00:00
}
implement python opeartor (#1045) * implement python opeartor * format code * remove dup * limit type * format code * cancel default logging func * add comment * fix compile err * fix comments * switch to c++ style cast * implement interop framework * fix format * move includes * cancel needless linking * fix comment * add UT * exclude def
2019-06-07 18:50:23 +00:00
#endif
Abbreviate ONNXRuntime as Ort in all of our public APIs (#175) Applies to all public headers and macros, plus many internal ones. There are still some internal things with OnnxRuntime in the name, but this fixes all public functions & macros.
2018-12-14 22:54:23 +00:00
#ifdef ORT_RUN_EXTERNAL_ONNX_TESTS
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
TEST(CApiTest, create_session_without_session_option) {
Sync with internal master.
2018-11-23 04:56:43 +00:00
constexpr PATH_TYPE model_uri = TSTR("../models/opset8/test_squeezenet/model.onnx");
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
Ort::Session ret(*ort_env, model_uri, Ort::SessionOptions{nullptr});
Sync with internal master.
2018-11-23 04:56:43 +00:00
ASSERT_NE(nullptr, ret);
}
#endif
Ryanunderhill/ocr custom op (#744) * Adding a custom op interface to the C API to remove shared library dependency. * Remove old custom op test * Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method * Add a nicer C++ API for custom ops and switch the tests to use it.
2019-04-06 01:53:20 +00:00
CiPipeline@ReducedOpsBuild (#4917) * cancel night build on pyop * setup ci pipeline for build of reduced ops * add back c# test * remove debugging print * add testing model * add more arg in pipeline script * disable pipeline trigger temporarily * fix yaml format * fix yaml format * fix pipeline error * rid c# test * add ops for test cases * add Conv from domain com.microsoft.nchwc * remove --reduce_ops * fix typo * remove --build_java * add test case for excluded op * update doc with --skip_test * formatting code, renaming files and simplify yaml * remove debug build from yaml * remove surplus ops from included_ops.txt * add MinSizeRel build to yaml * rename test cases and models * exclude ir test from minimum build * restrict ir test to be only applied to reduced ops build
2020-09-01 04:21:18 +00:00
#ifdef REDUCED_OPS_BUILD
TEST(ReducedOpsBuildTest, test_excluded_ops) {
Add ability to track per operator types in reduced build config. (#6428) * Add ability to generate configuration that includes required types for individual operators, to allow build size reduction based on that. - Add python bindings for ORT format models - Add script to update bindings and help info - Add parsing of ORT format models - Add ability to enable type reduction to config generation - Update build.py to only allow operator/type reduction via config - simpler to require config to be generated first - can't mix a type aware (ORT format model only) and non-type aware config as that may result in insufficient types being enabled - Add script to create reduced build config - Update CIs
2021-01-28 21:59:51 +00:00
// In reduced ops build, test a model containing ops not included in required_ops.config cannot be loaded.
// See onnxruntime/test/testdata/reduced_build_test.readme.txt for more details of the setup
constexpr PATH_TYPE model_uri = TSTR("testdata/reduced_build_test.onnx_model_with_excluded_ops");
Fix reduce pipeline by replacing model (#5813) * update model and better comment * fix parameter Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2020-11-15 04:17:23 +00:00
std::vector<Input> inputs = {{"X", {3}, {-1.0f, 2.0f, -3.0f}}};
std::vector<int64_t> expected_dims_y = {3};
std::vector<float> expected_values_y = {0.1f, 0.1f, 0.1f};
CiPipeline@ReducedOpsBuild (#4917) * cancel night build on pyop * setup ci pipeline for build of reduced ops * add back c# test * remove debugging print * add testing model * add more arg in pipeline script * disable pipeline trigger temporarily * fix yaml format * fix yaml format * fix pipeline error * rid c# test * add ops for test cases * add Conv from domain com.microsoft.nchwc * remove --reduce_ops * fix typo * remove --build_java * add test case for excluded op * update doc with --skip_test * formatting code, renaming files and simplify yaml * remove debug build from yaml * remove surplus ops from included_ops.txt * add MinSizeRel build to yaml * rename test cases and models * exclude ir test from minimum build * restrict ir test to be only applied to reduced ops build
2020-09-01 04:21:18 +00:00
bool failed = false;
try {
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// only test model loading, exception expected
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
TestInference<float>(*ort_env, model_uri, inputs, "Y", expected_dims_y, expected_values_y, 0,
Improve custom op library handle cleanup (#14099) ### Description - Adds a new C API `OrtApi::RegisterCustomOpsLibrary_V2` that manages the lifetime of dynamic library handles (i.e., calls `dlclose` or `FreeLibrary`). - Deprecates C API `OrtApi::RegisterCustomOpsLibrary`. - Adds C++ API wrapper for convenient registering of custom op libraries. - `PySessionOptions` is now an alias of `OrtSessionOptions` ### Motivation and Context The current API for registering custom op libraries loads dynamic libraries but requires users to handle the release of the corresponding library handles. Additionally, the user has to make sure to release the library handle _after_ the session has been destroyed (or the program segfaults). The new API automatically cleans up the library and allows the user to write more straightforward code.
2023-01-05 01:56:29 +00:00
nullptr, nullptr, true);
CiPipeline@ReducedOpsBuild (#4917) * cancel night build on pyop * setup ci pipeline for build of reduced ops * add back c# test * remove debugging print * add testing model * add more arg in pipeline script * disable pipeline trigger temporarily * fix yaml format * fix yaml format * fix pipeline error * rid c# test * add ops for test cases * add Conv from domain com.microsoft.nchwc * remove --reduce_ops * fix typo * remove --build_java * add test case for excluded op * update doc with --skip_test * formatting code, renaming files and simplify yaml * remove debug build from yaml * remove surplus ops from included_ops.txt * add MinSizeRel build to yaml * rename test cases and models * exclude ir test from minimum build * restrict ir test to be only applied to reduced ops build
2020-09-01 04:21:18 +00:00
} catch (const Ort::Exception& e) {
failed = e.GetOrtErrorCode() == ORT_NOT_IMPLEMENTED;
}
ASSERT_EQ(failed, true);
}
#endif
Expose IOBinding features via C/C++/C# language bindings. (#4646) Expose I/O Binding in C/C++/C# Expose OrtAllocator, OrtMemoryAllocation, OrtMemoryInfo and OrtIoBinding
2020-08-10 20:33:49 +00:00
TEST(CApiTest, get_allocator_cpu) {
Ort::SessionOptions session_options;
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CPU(session_options, 1));
Ort::Session session(*ort_env, NAMED_AND_ANON_DIM_PARAM_URI, session_options);
Ort::MemoryInfo info_cpu = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemTypeDefault);
Ort::Allocator cpu_allocator(session, info_cpu);
// CPU OrtMemoryInfo does not return OrtArenaAllocator on x86 but rather a device allocator
// which causes MemoryInfo that is used to request the allocator and the actual instance
// of MemoryInfo returned from the allocator exactly match, although they are functionally equivalent.
auto allocator_info = cpu_allocator.GetInfo();
ASSERT_EQ(info_cpu.GetAllocatorName(), allocator_info.GetAllocatorName());
ASSERT_EQ(info_cpu.GetDeviceId(), allocator_info.GetDeviceId());
ASSERT_EQ(info_cpu.GetMemoryType(), allocator_info.GetDeviceId());
void* p = cpu_allocator.Alloc(1024);
ASSERT_NE(p, nullptr);
cpu_allocator.Free(p);
auto mem_allocation = cpu_allocator.GetAllocation(1024);
ASSERT_NE(nullptr, mem_allocation.get());
ASSERT_EQ(1024U, mem_allocation.size());
}
#ifdef USE_CUDA
TEST(CApiTest, get_allocator_cuda) {
Ort::SessionOptions session_options;
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, 0));
Ort::Session session(*ort_env, NAMED_AND_ANON_DIM_PARAM_URI, session_options);
Ort::MemoryInfo info_cuda("Cuda", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
Ort::Allocator cuda_allocator(session, info_cuda);
auto allocator_info = cuda_allocator.GetInfo();
ASSERT_TRUE(info_cuda == allocator_info);
void* p = cuda_allocator.Alloc(1024);
ASSERT_NE(p, nullptr);
cuda_allocator.Free(p);
auto mem_allocation = cuda_allocator.GetAllocation(1024);
ASSERT_NE(nullptr, mem_allocation.get());
ASSERT_EQ(1024U, mem_allocation.size());
}
#endif
TEST(CApiTest, io_binding) {
Ort::SessionOptions session_options;
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CPU(session_options, 1));
Ort::Session session(*ort_env, MODEL_URI, session_options);
Ort::MemoryInfo info_cpu = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemTypeDefault);
const std::array<int64_t, 2> x_shape = {3, 2};
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
std::array<float, 3 * 2> x_values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
Expose IOBinding features via C/C++/C# language bindings. (#4646) Expose I/O Binding in C/C++/C# Expose OrtAllocator, OrtMemoryAllocation, OrtMemoryInfo and OrtIoBinding
2020-08-10 20:33:49 +00:00
Ort::Value bound_x = Ort::Value::CreateTensor(info_cpu, x_values.data(), x_values.size(),
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
x_shape.data(), x_shape.size());
Expose IOBinding features via C/C++/C# language bindings. (#4646) Expose I/O Binding in C/C++/C# Expose OrtAllocator, OrtMemoryAllocation, OrtMemoryInfo and OrtIoBinding
2020-08-10 20:33:49 +00:00
const std::array<float, 3 * 2> expected_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
const std::array<int64_t, 2> y_shape = {3, 2};
std::array<float, 3 * 2> y_values;
Ort::Value bound_y = Ort::Value::CreateTensor(info_cpu, y_values.data(), y_values.size(),
y_shape.data(), y_shape.size());
Ort::IoBinding binding(session);
binding.BindInput("X", bound_x);
binding.BindOutput("Y", bound_y);
session.Run(Ort::RunOptions(), binding);
// Check the values against the bound raw memory
ASSERT_TRUE(std::equal(std::begin(y_values), std::end(y_values), std::begin(expected_y)));
// Now compare values via GetOutputValues
{
std::vector<Ort::Value> output_values = binding.GetOutputValues();
ASSERT_EQ(output_values.size(), 1U);
const Ort::Value& Y_value = output_values[0];
ASSERT_TRUE(Y_value.IsTensor());
Ort::TensorTypeAndShapeInfo type_info = Y_value.GetTensorTypeAndShapeInfo();
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, type_info.GetElementType());
auto count = type_info.GetElementCount();
ASSERT_EQ(expected_y.size(), count);
const float* values = Y_value.GetTensorData<float>();
ASSERT_TRUE(std::equal(values, values + count, std::begin(expected_y)));
}
{
std::vector<std::string> output_names = binding.GetOutputNames();
ASSERT_EQ(1U, output_names.size());
ASSERT_EQ(output_names[0].compare("Y"), 0);
}
// Now replace binding of Y with an on device binding instead of pre-allocated memory.
// This is when we can not allocate an OrtValue due to unknown dimensions
{
Ort::MemoryInfo info_cpu_dev("Cpu", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
binding.BindOutput("Y", info_cpu_dev);
session.Run(Ort::RunOptions(), binding);
}
// Check the output value allocated based on the device binding.
{
std::vector<Ort::Value> output_values = binding.GetOutputValues();
ASSERT_EQ(output_values.size(), 1U);
const Ort::Value& Y_value = output_values[0];
ASSERT_TRUE(Y_value.IsTensor());
Ort::TensorTypeAndShapeInfo type_info = Y_value.GetTensorTypeAndShapeInfo();
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, type_info.GetElementType());
auto count = type_info.GetElementCount();
ASSERT_EQ(expected_y.size(), count);
const float* values = Y_value.GetTensorData<float>();
ASSERT_TRUE(std::equal(values, values + count, std::begin(expected_y)));
}
binding.ClearBoundInputs();
binding.ClearBoundOutputs();
}
Fix allocator issue for TensorRT IOBinding (#6240) * Fix issue: https://github.com/microsoft/onnxruntime/issues/6094 Root cause: we didn't expose the OrtMemoryInfo for TRT, so it will cause issue if user want use IObinding for Tensorrt. Short term fix, add the OrtMemoryInfo for TRT. Long term should unify the allocator for CUDA and TRT
2021-01-01 04:15:43 +00:00
#if defined(USE_CUDA) || defined(USE_TENSORRT)
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
TEST(CApiTest, io_binding_cuda) {
Ort::SessionOptions session_options;
Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248)
2021-01-06 06:18:03 +00:00
#ifdef USE_TENSORRT
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_Tensorrt(session_options, 0));
#else
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, 0));
#endif
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
Ort::Session session(*ort_env, MODEL_URI, session_options);
Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248)
2021-01-06 06:18:03 +00:00
Ort::MemoryInfo info_cuda("Cuda", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
Fix allocator issue for TensorRT IOBinding (#6240) * Fix issue: https://github.com/microsoft/onnxruntime/issues/6094 Root cause: we didn't expose the OrtMemoryInfo for TRT, so it will cause issue if user want use IObinding for Tensorrt. Short term fix, add the OrtMemoryInfo for TRT. Long term should unify the allocator for CUDA and TRT
2021-01-01 04:15:43 +00:00
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
Ort::Allocator cuda_allocator(session, info_cuda);
auto allocator_info = cuda_allocator.GetInfo();
ASSERT_TRUE(info_cuda == allocator_info);
const std::array<int64_t, 2> x_shape = {3, 2};
std::array<float, 3 * 2> x_values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
Deprecate CustomApi and refactor public API for better safety and consistency (#13215) ### Description Deprecate CustomOpApi and refactor dependencies for exception safety and eliminate memory leaks. Refactor API classes for clear ownership and semantics. Introduce `InitProviderOrtApi()` ### Motivation and Context Make public API better and safer. Special note about `Ort::Unowned`. The class suffers from the following problems: 1. It is not able to hold const pointers to the underlying C objects. This forces users to `const_cast` and circumvent constness of the returned object. The user is now able to call mutating interfaces on the object which violates invariants and may be a thread-safety issue. It also enables to take ownership of the pointer and destroy it unintentionally (see examples below). 2. The objects that are unowned cannot be copied and that makes coding inconvenient and at times unsafe. 3. It directly inherits from the type it `unowns`. All of the above creates great conditions for inadvertent unowned object mutations and destructions. Consider the following examples of object slicing, one of them is from a real customer issue and the other one I accidentally coded myself (and I am supposed to know how this works). None of the below can be solved by aftermarket patches and can be hard to diagnose. #### Example 1 slicing of argument ```cpp void SlicingOnArgument(Ort::Value& value) { // This will take possession of the input and if the argument // is Ort::Unowned<Ort::Value> it would again double free the ptr // regardless if it was const or not since we cast it away. Ort::Value output_values[] = {std::move(value)}; } void main() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; // onowned is destroyed when the call returns. SlicingOnArgument(unowned); } ``` #### Example 2 slicing of return value ```cpp // The return will be sliced to Ort::Value that would own and relase (double free the ptr) Ort::Value SlicingOnReturn() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; return unowned; } ```
2022-10-06 21:57:37 +00:00
auto input_data = cuda_allocator.GetAllocation(x_values.size() * sizeof(float));
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
ASSERT_NE(input_data.get(), nullptr);
cudaMemcpy(input_data.get(), x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);
// Create an OrtValue tensor backed by data on CUDA memory
Ort::Value bound_x = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(input_data.get()), x_values.size(),
x_shape.data(), x_shape.size());
const std::array<int64_t, 2> expected_y_shape = {3, 2};
const std::array<float, 3 * 2> expected_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
Deprecate CustomApi and refactor public API for better safety and consistency (#13215) ### Description Deprecate CustomOpApi and refactor dependencies for exception safety and eliminate memory leaks. Refactor API classes for clear ownership and semantics. Introduce `InitProviderOrtApi()` ### Motivation and Context Make public API better and safer. Special note about `Ort::Unowned`. The class suffers from the following problems: 1. It is not able to hold const pointers to the underlying C objects. This forces users to `const_cast` and circumvent constness of the returned object. The user is now able to call mutating interfaces on the object which violates invariants and may be a thread-safety issue. It also enables to take ownership of the pointer and destroy it unintentionally (see examples below). 2. The objects that are unowned cannot be copied and that makes coding inconvenient and at times unsafe. 3. It directly inherits from the type it `unowns`. All of the above creates great conditions for inadvertent unowned object mutations and destructions. Consider the following examples of object slicing, one of them is from a real customer issue and the other one I accidentally coded myself (and I am supposed to know how this works). None of the below can be solved by aftermarket patches and can be hard to diagnose. #### Example 1 slicing of argument ```cpp void SlicingOnArgument(Ort::Value& value) { // This will take possession of the input and if the argument // is Ort::Unowned<Ort::Value> it would again double free the ptr // regardless if it was const or not since we cast it away. Ort::Value output_values[] = {std::move(value)}; } void main() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; // onowned is destroyed when the call returns. SlicingOnArgument(unowned); } ``` #### Example 2 slicing of return value ```cpp // The return will be sliced to Ort::Value that would own and relase (double free the ptr) Ort::Value SlicingOnReturn() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; return unowned; } ```
2022-10-06 21:57:37 +00:00
auto output_data = cuda_allocator.GetAllocation(expected_y.size() * sizeof(float));
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
ASSERT_NE(output_data.get(), nullptr);
// Create an OrtValue tensor backed by data on CUDA memory
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
Ort::Value bound_y = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(output_data.get()),
expected_y.size(), expected_y_shape.data(), expected_y_shape.size());
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
Ort::IoBinding binding(session);
binding.BindInput("X", bound_x);
binding.BindOutput("Y", bound_y);
Expose IOBinding SynchronizeInputs/Outputs via C/C++/C# And Python APIs (#9823) Add C/C++ APIs for SynchronizeBoundInputs/Outputs Add python bindings Expose SynchronizeBoundInputs/Outputs to C# API
2021-11-22 17:45:31 +00:00
// Sychronize to make sure the copy on default stream is done since TensorRT isn't using default stream.
binding.SynchronizeInputs();
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
session.Run(Ort::RunOptions(), binding);
Expose IOBinding SynchronizeInputs/Outputs via C/C++/C# And Python APIs (#9823) Add C/C++ APIs for SynchronizeBoundInputs/Outputs Add python bindings Expose SynchronizeBoundInputs/Outputs to C# API
2021-11-22 17:45:31 +00:00
binding.SynchronizeOutputs();
Add a CUDA based IOBinding test (#5572)
2020-10-26 17:57:36 +00:00
// Check the values against the bound raw memory (needs copying from device to host first)
std::array<float, 3 * 2> y_values_0;
cudaMemcpy(y_values_0.data(), output_data.get(), sizeof(float) * y_values_0.size(), cudaMemcpyDeviceToHost);
ASSERT_TRUE(std::equal(std::begin(y_values_0), std::end(y_values_0), std::begin(expected_y)));
// Now compare values via GetOutputValues
{
std::vector<Ort::Value> output_values = binding.GetOutputValues();
ASSERT_EQ(output_values.size(), 1U);
const Ort::Value& Y_value = output_values[0];
ASSERT_TRUE(Y_value.IsTensor());
Ort::TensorTypeAndShapeInfo type_info = Y_value.GetTensorTypeAndShapeInfo();
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, type_info.GetElementType());
auto count = type_info.GetElementCount();
ASSERT_EQ(expected_y.size(), count);
const float* values = Y_value.GetTensorData<float>();
std::array<float, 3 * 2> y_values_1;
cudaMemcpy(y_values_1.data(), values, sizeof(float) * y_values_1.size(), cudaMemcpyDeviceToHost);
ASSERT_TRUE(std::equal(std::begin(y_values_1), std::end(y_values_1), std::begin(expected_y)));
}
{
std::vector<std::string> output_names = binding.GetOutputNames();
ASSERT_EQ(1U, output_names.size());
ASSERT_EQ(output_names[0].compare("Y"), 0);
}
// Now replace binding of Y with an on device binding instead of pre-allocated memory.
// This is when we can not allocate an OrtValue due to unknown dimensions
{
binding.BindOutput("Y", info_cuda);
session.Run(Ort::RunOptions(), binding);
}
// Check the output value allocated based on the device binding.
{
std::vector<Ort::Value> output_values = binding.GetOutputValues();
ASSERT_EQ(output_values.size(), 1U);
const Ort::Value& Y_value = output_values[0];
ASSERT_TRUE(Y_value.IsTensor());
Ort::TensorTypeAndShapeInfo type_info = Y_value.GetTensorTypeAndShapeInfo();
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, type_info.GetElementType());
auto count = type_info.GetElementCount();
ASSERT_EQ(expected_y.size(), count);
const float* values = Y_value.GetTensorData<float>();
std::array<float, 3 * 2> y_values_2;
cudaMemcpy(y_values_2.data(), values, sizeof(float) * y_values_2.size(), cudaMemcpyDeviceToHost);
ASSERT_TRUE(std::equal(std::begin(y_values_2), std::end(y_values_2), std::begin(expected_y)));
}
// Clean up
binding.ClearBoundInputs();
binding.ClearBoundOutputs();
}
#endif
CUDA graph support for TRT EP (#16081) CUDA EP already supports [CUDA graph](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs), also we observed some models can benefit from using CUDA graph with `trtexec`. Therefore, this PR enables the CUDA graph support for TRT EP. The implementation is based on https://github.com/microsoft/onnxruntime/pull/9978 with the same [constraints](https://github.com/microsoft/onnxruntime/pull/9978) as below: - Models with control-flow ops (i.e. If, Loop and Scan ops) are not supported. - Usage of CUDA Graphs is limited to models where-in all the model ops (graph nodes) can be partitioned to the TRT EP. - The input/output types of models need to be tensors. - Shapes of inputs/outputs cannot change across inference calls. - IObinding is required.
2023-06-21 16:36:45 +00:00
#if defined(USE_CUDA) || defined(USE_TENSORRT)
Relax node placement check for CUDA Graph usage (#16358)
2023-06-15 21:03:08 +00:00
TEST(CApiTest, basic_cuda_graph) {
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
const auto& api = Ort::GetApi();
CUDA graph support for TRT EP (#16081) CUDA EP already supports [CUDA graph](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs), also we observed some models can benefit from using CUDA graph with `trtexec`. Therefore, this PR enables the CUDA graph support for TRT EP. The implementation is based on https://github.com/microsoft/onnxruntime/pull/9978 with the same [constraints](https://github.com/microsoft/onnxruntime/pull/9978) as below: - Models with control-flow ops (i.e. If, Loop and Scan ops) are not supported. - Usage of CUDA Graphs is limited to models where-in all the model ops (graph nodes) can be partitioned to the TRT EP. - The input/output types of models need to be tensors. - Shapes of inputs/outputs cannot change across inference calls. - IObinding is required.
2023-06-21 16:36:45 +00:00
Ort::SessionOptions session_options;
#if defined(USE_TENSORRT)
// Enable cuda graph in TRT provider option.
OrtTensorRTProviderOptionsV2* trt_options;
ASSERT_TRUE(api.CreateTensorRTProviderOptions(&trt_options) == nullptr);
std::unique_ptr<OrtTensorRTProviderOptionsV2, decltype(api.ReleaseTensorRTProviderOptions)>
rel_trt_options(trt_options, api.ReleaseTensorRTProviderOptions);
std::vector<const char*> keys{"trt_cuda_graph_enable"};
std::vector<const char*> values{"1"};
ASSERT_TRUE(api.UpdateTensorRTProviderOptions(rel_trt_options.get(), keys.data(), values.data(), keys.size()) == nullptr);
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
CUDA graph support for TRT EP (#16081) CUDA EP already supports [CUDA graph](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs), also we observed some models can benefit from using CUDA graph with `trtexec`. Therefore, this PR enables the CUDA graph support for TRT EP. The implementation is based on https://github.com/microsoft/onnxruntime/pull/9978 with the same [constraints](https://github.com/microsoft/onnxruntime/pull/9978) as below: - Models with control-flow ops (i.e. If, Loop and Scan ops) are not supported. - Usage of CUDA Graphs is limited to models where-in all the model ops (graph nodes) can be partitioned to the TRT EP. - The input/output types of models need to be tensors. - Shapes of inputs/outputs cannot change across inference calls. - IObinding is required.
2023-06-21 16:36:45 +00:00
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_TensorRT_V2(
static_cast<OrtSessionOptions*>(session_options),
rel_trt_options.get()) == nullptr);
#else
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
// Enable cuda graph in cuda provider option.
OrtCUDAProviderOptionsV2* cuda_options = nullptr;
ASSERT_TRUE(api.CreateCUDAProviderOptions(&cuda_options) == nullptr);
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
std::unique_ptr<OrtCUDAProviderOptionsV2, decltype(api.ReleaseCUDAProviderOptions)>
rel_cuda_options(cuda_options, api.ReleaseCUDAProviderOptions);
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
std::vector<const char*> keys{"enable_cuda_graph"};
std::vector<const char*> values{"1"};
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
ASSERT_TRUE(api.UpdateCUDAProviderOptions(rel_cuda_options.get(), keys.data(), values.data(), 1) == nullptr);
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_CUDA_V2(
Fix bug in MemcpyToHost (#10816)
2022-03-10 15:02:27 +00:00
static_cast<OrtSessionOptions*>(session_options),
rel_cuda_options.get()) == nullptr);
CUDA graph support for TRT EP (#16081) CUDA EP already supports [CUDA graph](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs), also we observed some models can benefit from using CUDA graph with `trtexec`. Therefore, this PR enables the CUDA graph support for TRT EP. The implementation is based on https://github.com/microsoft/onnxruntime/pull/9978 with the same [constraints](https://github.com/microsoft/onnxruntime/pull/9978) as below: - Models with control-flow ops (i.e. If, Loop and Scan ops) are not supported. - Usage of CUDA Graphs is limited to models where-in all the model ops (graph nodes) can be partitioned to the TRT EP. - The input/output types of models need to be tensors. - Shapes of inputs/outputs cannot change across inference calls. - IObinding is required.
2023-06-21 16:36:45 +00:00
#endif
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
Ort::Session session(*ort_env, MODEL_URI, session_options);
Ort::MemoryInfo info_cuda("Cuda", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
Ort::Allocator cuda_allocator(session, info_cuda);
auto allocator_info = cuda_allocator.GetInfo();
ASSERT_TRUE(info_cuda == allocator_info);
const std::array<int64_t, 2> x_shape = {3, 2};
std::array<float, 3 * 2> x_values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
Deprecate CustomApi and refactor public API for better safety and consistency (#13215) ### Description Deprecate CustomOpApi and refactor dependencies for exception safety and eliminate memory leaks. Refactor API classes for clear ownership and semantics. Introduce `InitProviderOrtApi()` ### Motivation and Context Make public API better and safer. Special note about `Ort::Unowned`. The class suffers from the following problems: 1. It is not able to hold const pointers to the underlying C objects. This forces users to `const_cast` and circumvent constness of the returned object. The user is now able to call mutating interfaces on the object which violates invariants and may be a thread-safety issue. It also enables to take ownership of the pointer and destroy it unintentionally (see examples below). 2. The objects that are unowned cannot be copied and that makes coding inconvenient and at times unsafe. 3. It directly inherits from the type it `unowns`. All of the above creates great conditions for inadvertent unowned object mutations and destructions. Consider the following examples of object slicing, one of them is from a real customer issue and the other one I accidentally coded myself (and I am supposed to know how this works). None of the below can be solved by aftermarket patches and can be hard to diagnose. #### Example 1 slicing of argument ```cpp void SlicingOnArgument(Ort::Value& value) { // This will take possession of the input and if the argument // is Ort::Unowned<Ort::Value> it would again double free the ptr // regardless if it was const or not since we cast it away. Ort::Value output_values[] = {std::move(value)}; } void main() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; // onowned is destroyed when the call returns. SlicingOnArgument(unowned); } ``` #### Example 2 slicing of return value ```cpp // The return will be sliced to Ort::Value that would own and relase (double free the ptr) Ort::Value SlicingOnReturn() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; return unowned; } ```
2022-10-06 21:57:37 +00:00
auto input_data = cuda_allocator.GetAllocation(x_values.size() * sizeof(float));
Add ability to register custom ops by specifying a function name (#14177) ### Description <!-- Describe your changes. --> Use dlsym/GetProcAddress to lookup a custom ops registration function by name and call it. This will be better on mobile platforms where the custom ops library is linked against, and there isn't necessarily a filesystem that a library path can be loaded from. Alternative is to wire up passing in the address of the function, but that has multiple complications which differ by platform. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable using ort and ort-ext packages on mobile platforms. Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-01-12 05:11:34 +00:00
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
ASSERT_NE(input_data.get(), nullptr);
cudaMemcpy(input_data.get(), x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);
// Create an OrtValue tensor backed by data on CUDA memory
Ort::Value bound_x = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(input_data.get()), x_values.size(),
x_shape.data(), x_shape.size());
const std::array<int64_t, 2> expected_y_shape = {3, 2};
std::array<float, 3 * 2> expected_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
Deprecate CustomApi and refactor public API for better safety and consistency (#13215) ### Description Deprecate CustomOpApi and refactor dependencies for exception safety and eliminate memory leaks. Refactor API classes for clear ownership and semantics. Introduce `InitProviderOrtApi()` ### Motivation and Context Make public API better and safer. Special note about `Ort::Unowned`. The class suffers from the following problems: 1. It is not able to hold const pointers to the underlying C objects. This forces users to `const_cast` and circumvent constness of the returned object. The user is now able to call mutating interfaces on the object which violates invariants and may be a thread-safety issue. It also enables to take ownership of the pointer and destroy it unintentionally (see examples below). 2. The objects that are unowned cannot be copied and that makes coding inconvenient and at times unsafe. 3. It directly inherits from the type it `unowns`. All of the above creates great conditions for inadvertent unowned object mutations and destructions. Consider the following examples of object slicing, one of them is from a real customer issue and the other one I accidentally coded myself (and I am supposed to know how this works). None of the below can be solved by aftermarket patches and can be hard to diagnose. #### Example 1 slicing of argument ```cpp void SlicingOnArgument(Ort::Value& value) { // This will take possession of the input and if the argument // is Ort::Unowned<Ort::Value> it would again double free the ptr // regardless if it was const or not since we cast it away. Ort::Value output_values[] = {std::move(value)}; } void main() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; // onowned is destroyed when the call returns. SlicingOnArgument(unowned); } ``` #### Example 2 slicing of return value ```cpp // The return will be sliced to Ort::Value that would own and relase (double free the ptr) Ort::Value SlicingOnReturn() { const OrtValue* ptr = nullptr; // some value does not matter Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)}; return unowned; } ```
2022-10-06 21:57:37 +00:00
auto output_data = cuda_allocator.GetAllocation(expected_y.size() * sizeof(float));
Add ability to register custom ops by specifying a function name (#14177) ### Description <!-- Describe your changes. --> Use dlsym/GetProcAddress to lookup a custom ops registration function by name and call it. This will be better on mobile platforms where the custom ops library is linked against, and there isn't necessarily a filesystem that a library path can be loaded from. Alternative is to wire up passing in the address of the function, but that has multiple complications which differ by platform. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable using ort and ort-ext packages on mobile platforms. Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-01-12 05:11:34 +00:00
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
ASSERT_NE(output_data.get(), nullptr);
// Create an OrtValue tensor backed by data on CUDA memory
Ort::Value bound_y = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(output_data.get()),
expected_y.size(), expected_y_shape.data(), expected_y_shape.size());
Relax node placement check for CUDA Graph usage (#16358)
2023-06-15 21:03:08 +00:00
// Create IoBinding for inputs and outputs.
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
Ort::IoBinding binding(session);
binding.BindInput("X", bound_x);
binding.BindOutput("Y", bound_y);
// One regular run for necessary memory allocation and graph capturing
session.Run(Ort::RunOptions(), binding);
// Check the values against the bound raw memory (needs copying from device to host first)
std::array<float, 3 * 2> y_values;
cudaMemcpy(y_values.data(), output_data.get(), sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
// Replay the captured CUDA graph
session.Run(Ort::RunOptions(), binding);
cudaMemcpy(y_values.data(), output_data.get(), sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
// Change the input and replay the CUDA graph again.
x_values = {10.0f, 20.0f, 30.0f, 40.0f, 50.0f, 60.0f};
cudaMemcpy(input_data.get(), x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
binding.SynchronizeInputs();
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
session.Run(Ort::RunOptions(), binding);
cudaMemcpy(y_values.data(), output_data.get(), sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
expected_y = {10.0f, 40.0f, 90.0f, 160.0f, 250.0f, 360.0f};
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
// Clean up
binding.ClearBoundInputs();
binding.ClearBoundOutputs();
}
Relax node placement check for CUDA Graph usage (#16358)
2023-06-15 21:03:08 +00:00
Fix Reduced Ops pipeline (#16612)
2023-07-06 21:32:59 +00:00
#ifndef REDUCED_OPS_BUILD
// The following test uses some ops not supported in the reduced ops build
Relax node placement check for CUDA Graph usage (#16358)
2023-06-15 21:03:08 +00:00
TEST(CApiTest, cuda_graph_with_shape_nodes) {
const auto& api = Ort::GetApi();
// Enable cuda graph in cuda provider option.
OrtCUDAProviderOptionsV2* cuda_options = nullptr;
ASSERT_TRUE(api.CreateCUDAProviderOptions(&cuda_options) == nullptr);
std::unique_ptr<OrtCUDAProviderOptionsV2, decltype(api.ReleaseCUDAProviderOptions)>
rel_cuda_options(cuda_options, api.ReleaseCUDAProviderOptions);
std::vector<const char*> keys{"enable_cuda_graph"};
std::vector<const char*> values{"1"};
ASSERT_TRUE(api.UpdateCUDAProviderOptions(rel_cuda_options.get(), keys.data(), values.data(), 1) == nullptr);
Ort::SessionOptions session_options;
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_CUDA_V2(
static_cast<OrtSessionOptions*>(session_options),
rel_cuda_options.get()) == nullptr);
// Successful loading of the ONNX model with shape nodes with cuda graph feature enabled
Ort::Session session(*ort_env, TSTR("testdata/cuda_graph_with_shape_nodes.onnx"), session_options);
}
Support CUDA Graph in the CUDA EP (#9978)
2022-03-07 04:47:31 +00:00
#endif
Fix Reduced Ops pipeline (#16612)
2023-07-06 21:32:59 +00:00
#endif
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
TEST(CApiTest, create_tensor) {
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
const char* s[] = {"abc", "kmp"};
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
constexpr int64_t expected_len = 2;
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto default_allocator = std::make_unique<MockedOrtAllocator>();
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
Ort::Value tensor = Ort::Value::CreateTensor(default_allocator.get(), &expected_len, 1,
ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
Move CXX API global into the header (#2228)
2019-10-23 21:15:53 +00:00
Ort::ThrowOnError(Ort::GetApi().FillStringTensor(tensor, s, expected_len));
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
auto shape_info = tensor.GetTensorTypeAndShapeInfo();
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
const auto len = shape_info.GetElementCount();
ASSERT_EQ(len, static_cast<size_t>(expected_len));
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
std::vector<int64_t> shape_array(len);
size_t data_len = tensor.GetStringTensorDataLength();
std::string result(data_len, '\0');
std::vector<size_t> offsets(len);
tensor.GetStringTensorContent((void*)result.data(), data_len, offsets.data(), offsets.size());
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
}
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
TEST(CApiTest, fill_string_tensor) {
const char* s[] = {"abc", "kmp"};
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
constexpr int64_t expected_len = 2;
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto default_allocator = std::make_unique<MockedOrtAllocator>();
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
Ort::Value tensor = Ort::Value::CreateTensor(default_allocator.get(), &expected_len, 1,
ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
for (size_t i = 0; i < expected_len; i++) {
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
tensor.FillStringTensorElement(s[i], i);
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
}
auto shape_info = tensor.GetTensorTypeAndShapeInfo();
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
const auto len = shape_info.GetElementCount();
ASSERT_EQ(len, static_cast<size_t>(expected_len));
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
}
[C# ] Improve string marshalling and reduce GC pressure (#15545) ### Description Reduce a number of auxillary objects created to reduce GC pressure. Eliminate GCHandle type of memory pinning in most of the places. Improve string marshalling by allocating unmanaged memory that does not require pinning. Change native methods from `IntPtr` to `byte[]` (marshalling pinning is more efficient). Allocate input/output UTF-8 names in unmanaged heap for the lifetime of InferenceSession. So we do not keep converting them and pinning on every Run. Introduce a new native API that allows to allocate and convert/copy strings directly into a native tensor. The PR delivers around 50% latency improvements and less GC pauses. Inspired by: https://github.com/microsoft/onnxruntime/pull/15520 ### Motivation and Context Client experience GC pressure and performance degradation when dealing with string tensors. Co-Authored-By: @tannergooding
2023-04-20 22:12:51 +00:00
TEST(CApiTest, fill_string_tensor_directly) {
constexpr std::string_view s[] = {"abc", "kmp"};
constexpr int64_t expected_len = 2;
MockedOrtAllocator default_allocator;
Ort::Value tensor = Ort::Value::CreateTensor(&default_allocator, &expected_len, 1U,
ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
for (size_t i = 0; i < expected_len; i++) {
[C# ] Improve string marshalling and reduce GC pressure (#15545) ### Description Reduce a number of auxillary objects created to reduce GC pressure. Eliminate GCHandle type of memory pinning in most of the places. Improve string marshalling by allocating unmanaged memory that does not require pinning. Change native methods from `IntPtr` to `byte[]` (marshalling pinning is more efficient). Allocate input/output UTF-8 names in unmanaged heap for the lifetime of InferenceSession. So we do not keep converting them and pinning on every Run. Introduce a new native API that allows to allocate and convert/copy strings directly into a native tensor. The PR delivers around 50% latency improvements and less GC pauses. Inspired by: https://github.com/microsoft/onnxruntime/pull/15520 ### Motivation and Context Client experience GC pressure and performance degradation when dealing with string tensors. Co-Authored-By: @tannergooding
2023-04-20 22:12:51 +00:00
auto* buffer = tensor.GetResizedStringTensorElementBuffer(i, s[i].size());
memcpy(buffer, s[i].data(), s[i].size());
}
auto shape_info = tensor.GetTensorTypeAndShapeInfo();
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
const auto len = shape_info.GetElementCount();
ASSERT_EQ(len, static_cast<size_t>(expected_len));
[C# ] Improve string marshalling and reduce GC pressure (#15545) ### Description Reduce a number of auxillary objects created to reduce GC pressure. Eliminate GCHandle type of memory pinning in most of the places. Improve string marshalling by allocating unmanaged memory that does not require pinning. Change native methods from `IntPtr` to `byte[]` (marshalling pinning is more efficient). Allocate input/output UTF-8 names in unmanaged heap for the lifetime of InferenceSession. So we do not keep converting them and pinning on every Run. Introduce a new native API that allows to allocate and convert/copy strings directly into a native tensor. The PR delivers around 50% latency improvements and less GC pauses. Inspired by: https://github.com/microsoft/onnxruntime/pull/15520 ### Motivation and Context Client experience GC pressure and performance degradation when dealing with string tensors. Co-Authored-By: @tannergooding
2023-04-20 22:12:51 +00:00
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
for (size_t i = 0; i < expected_len; i++) {
[C# ] Improve string marshalling and reduce GC pressure (#15545) ### Description Reduce a number of auxillary objects created to reduce GC pressure. Eliminate GCHandle type of memory pinning in most of the places. Improve string marshalling by allocating unmanaged memory that does not require pinning. Change native methods from `IntPtr` to `byte[]` (marshalling pinning is more efficient). Allocate input/output UTF-8 names in unmanaged heap for the lifetime of InferenceSession. So we do not keep converting them and pinning on every Run. Introduce a new native API that allows to allocate and convert/copy strings directly into a native tensor. The PR delivers around 50% latency improvements and less GC pauses. Inspired by: https://github.com/microsoft/onnxruntime/pull/15520 ### Motivation and Context Client experience GC pressure and performance degradation when dealing with string tensors. Co-Authored-By: @tannergooding
2023-04-20 22:12:51 +00:00
auto element = tensor.GetStringTensorElement(i);
ASSERT_EQ(s[i], element);
}
}
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
TEST(CApiTest, get_string_tensor_element) {
const char* s[] = {"abc", "kmp"};
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
constexpr int64_t expected_len = 2;
constexpr int64_t element_index = 0;
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto default_allocator = std::make_unique<MockedOrtAllocator>();
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
Ort::Value tensor = Ort::Value::CreateTensor(default_allocator.get(), &expected_len, 1,
ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
tensor.FillStringTensor(s, expected_len);
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
auto expected_string = s[element_index];
Replace functions with secured version for OSX compliance (#7586) * replace strlen with strnlen * replace vsnprintf with vsnprintf_l * add macro * switch to std numeric::limits * apply uint16 max * fix build err * fix mac build * define MAX_STR_LEN * define MAX_STR_LEN * fix typo * trim empty lines * apply constexpr * fix typo * add namespace * fix build err * rename global constant Co-authored-by: Randy <Randy@randysmac.attlocal.net> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: Randy <Randy@randysmac.local>
2021-07-08 18:02:36 +00:00
size_t expected_string_len = strnlen(expected_string, onnxruntime::kMaxStrLen);
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
std::string result(expected_string_len, '\0');
tensor.GetStringTensorElement(expected_string_len, element_index, (void*)result.data());
ASSERT_STREQ(result.c_str(), expected_string);
auto string_len = tensor.GetStringTensorElementLength(element_index);
ASSERT_EQ(expected_string_len, string_len);
}
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
TEST(CApiTest, create_tensor_with_data) {
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
float values[] = {3.0f, 1.0f, 2.f, 0.f};
constexpr size_t values_length = sizeof(values) / sizeof(values[0]);
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
Part 2 of renaming AllocatorInfo to MemoryInfo. (#1804) * Mention OrtCreateSessionFromArray in C API doc * Part 2 of renaming AllocatorInfo to MemoryInfo. * pr comments * fix comment
2019-09-12 15:19:29 +00:00
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
Fix inconsistent dimension data type in C-API (#726) * update dimension type * update dimension type for items added after 0.2.1 * fix gpu build
2019-03-29 07:23:25 +00:00
std::vector<int64_t> dims = {4};
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
Ort::Value tensor = Ort::Value::CreateTensor<float>(info, values, values_length, dims.data(), dims.size());
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
const float* new_pointer = tensor.GetTensorData<float>();
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
ASSERT_EQ(new_pointer, values);
Ryanunderhill/cxx api2 (#1091) More C++ API improvements and cleanup Add templates to tensor creation Add run method that allows preallocated outputs Simplify CreateTensor<T> to multiply by sizeof(T) Convert io_types code Optimize away vector copies in Session::Run
2019-05-24 18:15:51 +00:00
auto type_info = tensor.GetTypeInfo();
auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
Sync with internal master.
2018-11-23 04:56:43 +00:00
ASSERT_NE(tensor_info, nullptr);
Fix some warnings on Windows (#2560) 1. Enable warning "4503" # Decorated name length exceeded. 2. Enable warning "4146" # unary minus operator applied to unsigned type. 3. Enable float64 support for the Softmax operator 4. Enable compliance checks for Windows x86 32bits build 5. Use TryBatchParallelFor to replace some fallback code in mlas pooling.cc 6. Fix Android CI pipeline.
2020-01-22 23:59:11 +00:00
ASSERT_EQ(1u, tensor_info.GetDimensionsCount());
Initial bootstrap commit.
2018-11-20 00:48:22 +00:00
}
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
TEST(CApiTest, create_tensor_with_data_float16) {
// Example with C++. However, what we are feeding underneath is really
// a continuous buffer of uint16_t
// Use 3rd party libraries such as Eigen to convert floats and doubles to float16 types.
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
constexpr float values[] = {1.f, 2.f, 3.f, 4.f, 5.f};
constexpr size_t values_length = std::size(values);
constexpr uint16_t expected_values[values_length] = {15360, 16384, 16896, 17408, 17664};
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
std::vector<Ort::Float16_t> fp16_values;
fp16_values.reserve(values_length);
std::transform(std::begin(values), std::end(values), std::back_inserter(fp16_values),
[](float fl) { return Ort::Float16_t(fl); });
for (size_t i = 0; i < values_length; ++i) {
ASSERT_EQ(expected_values[i], fp16_values[i].val);
}
constexpr int64_t dims = static_cast<int64_t>(values_length);
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
Ort::Value tensor = Ort::Value::CreateTensor<Ort::Float16_t>(info, fp16_values.data(), values_length, &dims, 1u);
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
const auto* new_pointer = tensor.GetTensorData<Ort::Float16_t>();
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
ASSERT_EQ(new_pointer, fp16_values.data());
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
auto type_info = tensor.GetTypeInfo();
auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
const auto element_count = tensor_info.GetElementCount();
ASSERT_EQ(values_length, element_count);
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
ASSERT_NE(tensor_info, nullptr);
ASSERT_EQ(1u, tensor_info.GetDimensionsCount());
ASSERT_EQ(tensor_info.GetElementType(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16);
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
const Ort::Float16_t& value_at_1 = tensor.At<Ort::Float16_t>({1});
ASSERT_EQ(expected_values[1], value_at_1.val);
std::vector<float> output_values;
output_values.reserve(values_length);
const auto data_span = gsl::make_span(tensor.GetTensorData<Ort::Float16_t>(), element_count);
std::transform(data_span.begin(), data_span.end(), std::back_inserter(output_values),
[](const Ort::Float16_t& fp16) { return static_cast<float>(fp16); });
for (size_t i = 0; i < values_length; ++i) {
ASSERT_NEAR(values[i], output_values[i], 1e-3);
}
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
}
TEST(CApiTest, create_tensor_with_data_bfloat16) {
// Example with C++. However, what we are feeding underneath is really
// a continuous buffer of uint16_t
// Conversion from float to bfloat16 is simple. Strip off half of the bytes from float.
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
constexpr float values[] = {1.f, 2.f, 3.f, 4.f, 5.f};
constexpr size_t values_length = std::size(values);
constexpr uint16_t expected_values[] = {16256, 16384, 16448, 16512, 16544}; // 1.f, 2.f, 3.f, 4.f, 5.f
constexpr int64_t dims = static_cast<int64_t>(values_length);
std::vector<Ort::BFloat16_t> b16_values;
b16_values.reserve(values_length);
std::transform(std::begin(values), std::end(values), std::back_inserter(b16_values),
[](float fl) { return Ort::BFloat16_t(fl); });
for (size_t i = 0; i < values_length; ++i) {
ASSERT_EQ(expected_values[i], b16_values[i].val);
}
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
Ort::Value tensor = Ort::Value::CreateTensor<Ort::BFloat16_t>(info, b16_values.data(), values_length, &dims, 1u);
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
const auto* new_pointer = tensor.GetTensorData<Ort::BFloat16_t>();
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
ASSERT_EQ(new_pointer, b16_values.data());
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
auto type_info = tensor.GetTypeInfo();
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
const auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
const auto element_count = tensor_info.GetElementCount();
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
ASSERT_NE(tensor_info, nullptr);
ASSERT_EQ(1u, tensor_info.GetDimensionsCount());
ASSERT_EQ(tensor_info.GetElementType(), ONNX_TENSOR_ELEMENT_DATA_TYPE_BFLOAT16);
[C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) ### Description Introduce `Float16/BFloat16` support for C# and C++ APIs. User should be able to perform conversions from `float` to/from `Float16/BFloat16`, compare values and tests for `NaN, Inifnity, and whether the number is denormalized.` ### Motivation and Context User filed issues such as: https://github.com/microsoft/onnxruntime/issues/14303
2023-07-14 17:46:52 +00:00
const Ort::BFloat16_t& value_at_1 = tensor.At<Ort::BFloat16_t>({1});
ASSERT_EQ(expected_values[1], value_at_1.val);
std::vector<float> output_values;
output_values.reserve(values_length);
const auto data_span = gsl::make_span(tensor.GetTensorData<Ort::BFloat16_t>(), element_count);
std::transform(data_span.begin(), data_span.end(), std::back_inserter(output_values),
[](const Ort::BFloat16_t& b16) { return static_cast<float>(b16); });
for (size_t i = 0; i < values_length; ++i) {
ASSERT_NEAR(values[i], output_values[i], 1e-3);
}
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs (#5716) Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs that contain uint16_t values for float16 and bfloat16. These will serve as type dispatching types for C++ API. They are of uint16_t size and arrays of these types can be used to create Tensors of the corresponding types. Make documentation Doxygen compliant.
2020-11-07 00:41:26 +00:00
}
Support custom ops taking float 8 tensors as inputs and outputs (#16323) ### Description C API for custom ops does not support float 8 types. This PR changes that. ### Motivation and Context The list of operators supporting float 8 is very limited. It should be extended to custom ops to let developpers add customized operators for these specific types.
2023-07-06 12:36:06 +00:00
#if !defined(DISABLE_FLOAT8_TYPES)
TEST(CApiTest, create_tensor_with_data_float8) {
Ort::Float8E4M3FN_t values[] = {0, 1, 2, 3, 4};
constexpr size_t values_length = sizeof(values) / sizeof(values[0]);
std::vector<int64_t> dims = {static_cast<int64_t>(values_length)};
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
Ort::Value tensor = Ort::Value::CreateTensor<Ort::Float8E4M3FN_t>(info, values, values_length, dims.data(), dims.size());
const auto* new_pointer = tensor.GetTensorData<Ort::Float8E4M3FN_t>();
ASSERT_EQ(new_pointer, values);
auto type_info = tensor.GetTypeInfo();
auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
ASSERT_NE(tensor_info, nullptr);
ASSERT_EQ(1u, tensor_info.GetDimensionsCount());
ASSERT_EQ(tensor_info.GetElementType(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT8E4M3FN);
Ort::Float8E4M3FN_t value_at_1 = tensor.At<Ort::Float8E4M3FN_t>({1});
ASSERT_EQ(values[1], value_at_1);
}
#endif
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
TEST(CApiTest, access_tensor_data_elements) {
/**
* Create a 2x3 data blob that looks like:
Remove onnxruntime_session_options_config_keys.h from c_api (#5772) * Remove seesion config keys header from c_api * remove copy session config header in release package * Keep the session option config header in the package
2020-11-12 17:12:13 +00:00
*
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
* 0 1 2
* 3 4 5
*/
std::vector<int64_t> shape = {2, 3};
int element_count = 6; // 2*3
std::vector<float> values(element_count);
for (int i = 0; i < element_count; i++)
values[i] = static_cast<float>(i);
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
Ort::Value tensor = Ort::Value::CreateTensor<float>(info, values.data(), values.size(), shape.data(), shape.size());
float expected_value = 0;
Refactor TensorAt, prepare for release (#5180) * Refactor TensorAt locations* must be const and int64_t since our dims are int64_t Remove unnecessary copy of locations. Remove unnecesary casting and C-casting. Simplify implementation. Add a check for string type. Make CXX api return T& to fully expose C API in C++, const std::vector& by value as it covers more ground and eliminate redundant copy. Eliminate inner loop, compute strides first.
2020-09-16 17:20:45 +00:00
for (int64_t row = 0; row < shape[0]; row++) {
for (int64_t col = 0; col < shape[1]; col++) {
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
ASSERT_EQ(expected_value++, tensor.At<float>({row, col}));
}
}
}
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
TEST(CApiTest, override_initializer) {
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto allocator = std::make_unique<MockedOrtAllocator>();
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
// CreateTensor which is not owning this ptr
bool Label_input[] = {true};
std::vector<int64_t> dims = {1, 1};
Ort::Value label_input_tensor = Ort::Value::CreateTensor<bool>(info, Label_input, 1U, dims.data(), dims.size());
std::string f2_data{"f2_string"};
// Place a string into Tensor OrtValue and assign to the
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
Ort::Value f2_input_tensor = Ort::Value::CreateTensor(allocator.get(), dims.data(), dims.size(),
ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
const char* const input_char_string[] = {f2_data.c_str()};
Add GetStringTensorElement, GetStringTensorElementLength and FillStringTensorElement API (#4374) Add new string tensor APIs and unit tests
2020-07-25 04:35:46 +00:00
f2_input_tensor.FillStringTensor(input_char_string, 1U);
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
Ort::SessionOptions session_options;
CMake changes (#2961) 1. Add support for vstest. 2. Add support for vcpkg. To use it: ```bat vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows mkdir build cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON ``` 3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule. 4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest. 4. Fix the post merge pipeline(Mainly for test coverage report). 5. Ignore the compile warning from the Featurizer library code 6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows. 7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it. 8. Move opaque api tests to onnxruntime_test_all 9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks. 10. Delete the onnxruntime::test::TestEnvironment class. 11. Add a DLLmain for onnxruntime.dll. 12. Allow dynamic link to libprotobuf
2020-02-04 03:33:14 +00:00
Ort::Session session(*ort_env, OVERRIDABLE_INITIALIZER_MODEL_URI, session_options);
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
// Get Overrideable initializers
size_t init_count = session.GetOverridableInitializerCount();
ASSERT_EQ(init_count, 1U);
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto f1_init_name = session.GetOverridableInitializerNameAllocated(0, allocator.get());
ASSERT_TRUE(strcmp("F1", f1_init_name.get()) == 0);
}
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
Ort::TypeInfo init_type_info = session.GetOverridableInitializerTypeInfo(0);
ASSERT_EQ(ONNX_TYPE_TENSOR, init_type_info.GetONNXType());
// Let's override the initializer
float f11_input_data[] = {2.0f};
Ort::Value f11_input_tensor = Ort::Value::CreateTensor<float>(info, f11_input_data, 1U, dims.data(), dims.size());
std::vector<Ort::Value> ort_inputs;
ort_inputs.push_back(std::move(label_input_tensor));
ort_inputs.push_back(std::move(f2_input_tensor));
ort_inputs.push_back(std::move(f11_input_tensor));
std::vector<const char*> input_names = {"Label", "F2", "F1"};
Refactor onnx_test_runner (#5169) Refactor onnx_test_runner for better object ownership, code readability and maintainability.
2020-09-18 20:19:35 +00:00
const char* output_names[] = {"Label0", "F20", "F11"};
Expose GetOverridableInitializers via Python and C/C++ API (#1878) Implement GetOverridableInitializers() Add unit test for initializer override. Expose in Python and C/C++ API
2019-09-19 22:43:28 +00:00
std::vector<Ort::Value> ort_outputs = session.Run(Ort::RunOptions{nullptr}, input_names.data(),
ort_inputs.data(), ort_inputs.size(),
output_names, countof(output_names));
ASSERT_EQ(ort_outputs.size(), 3U);
// Expecting the last output would be the overridden value of the initializer
auto type_info = ort_outputs[2].GetTensorTypeAndShapeInfo();
ASSERT_EQ(type_info.GetShape(), dims);
ASSERT_EQ(type_info.GetElementType(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);
ASSERT_EQ(type_info.GetElementCount(), 1U);
float* output_data = ort_outputs[2].GetTensorMutableData<float>();
ASSERT_EQ(*output_data, f11_input_data[0]);
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
}
TEST(CApiTest, end_profiling) {
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto allocator = std::make_unique<MockedOrtAllocator>();
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
// Create session with profiling enabled (profiling is automatically turned on)
Ort::SessionOptions session_options_1;
#ifdef _WIN32
session_options_1.EnableProfiling(L"profile_prefix");
#else
session_options_1.EnableProfiling("profile_prefix");
#endif
Ort::Session session_1(*ort_env, MODEL_WITH_CUSTOM_MODEL_METADATA, session_options_1);
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto profile_file = session_1.EndProfilingAllocated(allocator.get());
ASSERT_TRUE(std::string(profile_file.get()).find("profile_prefix") != std::string::npos);
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
// Create session with profiling disabled
Ort::SessionOptions session_options_2;
#ifdef _WIN32
session_options_2.DisableProfiling();
#else
session_options_2.DisableProfiling();
#endif
Ort::Session session_2(*ort_env, MODEL_WITH_CUSTOM_MODEL_METADATA, session_options_2);
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto profile_file = session_2.EndProfilingAllocated(allocator.get());
ASSERT_TRUE(std::string(profile_file.get()) == std::string());
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
}
Add GetStartTime() for profiler to get private profiling_start_time_ (#4994) * add GetStartTime() for profiler * add function in inference_session * remove qualified name * add the api in cxx_api.h * rename starttime to StartTimeNs, expost profiling object * rename GetProfilingStartTime * move Ortapis to the right place * move to the end * add const for session * const the right place * use const auto instead of const auto* for session * remove const for auto getstarttime * remove const for auto getstarttime add unit tests * nit: update test name and add comments
2020-09-16 07:17:04 +00:00
TEST(CApiTest, get_profiling_start_time) {
// Test whether the C_API can access the profiler's start time
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
Ort::SessionOptions session_options;
#ifdef _WIN32
session_options.EnableProfiling(L"profile_prefix");
#else
session_options.EnableProfiling("profile_prefix");
#endif
uint64_t before_start_time = std::chrono::duration_cast<std::chrono::nanoseconds>(
Add valgrind support to our cmake files (#5296)
2020-09-28 16:31:08 +00:00
std::chrono::high_resolution_clock::now().time_since_epoch())
.count(); // get current time
Add GetStartTime() for profiler to get private profiling_start_time_ (#4994) * add GetStartTime() for profiler * add function in inference_session * remove qualified name * add the api in cxx_api.h * rename starttime to StartTimeNs, expost profiling object * rename GetProfilingStartTime * move Ortapis to the right place * move to the end * add const for session * const the right place * use const auto instead of const auto* for session * remove const for auto getstarttime * remove const for auto getstarttime add unit tests * nit: update test name and add comments
2020-09-16 07:17:04 +00:00
Ort::Session session_1(*ort_env, MODEL_WITH_CUSTOM_MODEL_METADATA, session_options);
uint64_t profiling_start_time = session_1.GetProfilingStartTimeNs();
uint64_t after_start_time = std::chrono::duration_cast<std::chrono::nanoseconds>(
Add valgrind support to our cmake files (#5296)
2020-09-28 16:31:08 +00:00
std::chrono::high_resolution_clock::now().time_since_epoch())
.count();
Add GetStartTime() for profiler to get private profiling_start_time_ (#4994) * add GetStartTime() for profiler * add function in inference_session * remove qualified name * add the api in cxx_api.h * rename starttime to StartTimeNs, expost profiling object * rename GetProfilingStartTime * move Ortapis to the right place * move to the end * add const for session * const the right place * use const auto instead of const auto* for session * remove const for auto getstarttime * remove const for auto getstarttime add unit tests * nit: update test name and add comments
2020-09-16 07:17:04 +00:00
// the profiler's start time needs to be between before_time and after_time
Add valgrind support to our cmake files (#5296)
2020-09-28 16:31:08 +00:00
ASSERT_TRUE(before_start_time <= profiling_start_time && profiling_start_time <= after_start_time);
Add GetStartTime() for profiler to get private profiling_start_time_ (#4994) * add GetStartTime() for profiler * add function in inference_session * remove qualified name * add the api in cxx_api.h * rename starttime to StartTimeNs, expost profiling object * rename GetProfilingStartTime * move Ortapis to the right place * move to the end * add const for session * const the right place * use const auto instead of const auto* for session * remove const for auto getstarttime * remove const for auto getstarttime add unit tests * nit: update test name and add comments
2020-09-16 07:17:04 +00:00
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
TEST(CApiTest, model_metadata) {
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto allocator = std::make_unique<MockedOrtAllocator>();
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
// The following all tap into the c++ APIs which internally wrap over C APIs
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
// The following section tests a model containing all metadata supported via the APIs
{
Ort::SessionOptions session_options;
Ort::Session session(*ort_env, MODEL_WITH_CUSTOM_MODEL_METADATA, session_options);
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
// Fetch model metadata
auto model_metadata = session.GetModelMetadata();
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto producer_name = model_metadata.GetProducerNameAllocated(allocator.get());
ASSERT_TRUE(strcmp("Hari", producer_name.get()) == 0);
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto graph_name = model_metadata.GetGraphNameAllocated(allocator.get());
ASSERT_TRUE(strcmp("matmul test", graph_name.get()) == 0);
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto domain = model_metadata.GetDomainAllocated(allocator.get());
ASSERT_TRUE(strcmp("", domain.get()) == 0);
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto description = model_metadata.GetDescriptionAllocated(allocator.get());
ASSERT_TRUE(strcmp("This is a test model with a valid ORT config Json", description.get()) == 0);
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto graph_description = model_metadata.GetGraphDescriptionAllocated(allocator.get());
ASSERT_TRUE(strcmp("graph description", graph_description.get()) == 0);
}
Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248)
2021-01-06 06:18:03 +00:00
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
int64_t version = model_metadata.GetVersion();
ASSERT_TRUE(version == 1);
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto custom_metadata_map_keys = model_metadata.GetCustomMetadataMapKeysAllocated(allocator.get());
ASSERT_EQ(custom_metadata_map_keys.size(), 2U);
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
auto lookup_value_1 = model_metadata.LookupCustomMetadataMapAllocated("ort_config", allocator.get());
ASSERT_TRUE(strcmp(lookup_value_1.get(),
Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build.
2021-01-25 00:41:00 +00:00
"{\"session_options\": {\"inter_op_num_threads\": 5, \"intra_op_num_threads\": 2, "
"\"graph_optimization_level\": 99, \"enable_profiling\": 1}}") == 0);
Support accessing a model's metadata in C# (#4867) Implement access to model's metadata in C#
2020-08-25 18:13:49 +00:00
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
auto lookup_value_2 = model_metadata.LookupCustomMetadataMapAllocated("dummy_key", allocator.get());
ASSERT_TRUE(strcmp(lookup_value_2.get(), "dummy_value") == 0);
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
// key doesn't exist in custom metadata map
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
auto lookup_value_3 = model_metadata.LookupCustomMetadataMapAllocated("key_doesnt_exist", allocator.get());
Support accessing a model's metadata in C# (#4867) Implement access to model's metadata in C#
2020-08-25 18:13:49 +00:00
ASSERT_TRUE(lookup_value_3 == nullptr);
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
}
// The following section tests a model with some missing metadata info
// Adding this just to make sure the API implementation is able to handle empty/missing info
{
Ort::SessionOptions session_options;
Ort::Session session(*ort_env, MODEL_URI, session_options);
// Fetch model metadata
auto model_metadata = session.GetModelMetadata();
// Model description is empty
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto description = model_metadata.GetDescriptionAllocated(allocator.get());
ASSERT_TRUE(strcmp("", description.get()) == 0);
}
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248)
2021-01-06 06:18:03 +00:00
// Graph description is empty
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
{
auto graph_description = model_metadata.GetGraphDescriptionAllocated(allocator.get());
ASSERT_TRUE(strcmp("", graph_description.get()) == 0);
}
Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248)
2021-01-06 06:18:03 +00:00
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
// Model does not contain custom metadata map
Deprecate APIs returning raw ptrs and provide replacements (#11922) Provider better documentation
2022-06-24 16:50:04 +00:00
auto custom_metadata_map_keys = model_metadata.GetCustomMetadataMapKeysAllocated(allocator.get());
ASSERT_TRUE(custom_metadata_map_keys.empty());
Support listing keys in custom metadata map via C/C++ API (#3477) * Support listing keys in custom metadata map via C/C++ API * nit * PR feedback * Nit
2020-04-15 19:14:03 +00:00
}
Support a few new C/C++ APIs (#2794) * Initial commit * More changes * More changes * Changes * More changes * More changes * More changes * More changes * Updates * Fix break * PR feedback * Nit * Resolve conflicts * More changes
2020-02-11 00:18:42 +00:00
}
Added GetAvailableProviders() to C API (#4247) * Added GetAvailableProviders to C API * Fix API version and Windows build error * Changed function name * Changed ORT_API_VERSION to 4 * Moved all_providers array to constants.h * Move check for providers to constants.h * Changed name of array to avoid warning * Address review comment * Added unit test
2020-06-22 02:10:25 +00:00
TEST(CApiTest, get_available_providers) {
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
const OrtApi* g_ort = OrtGetApiBase()->GetApi(ORT_API_VERSION);
Added GetAvailableProviders() to C API (#4247) * Added GetAvailableProviders to C API * Fix API version and Windows build error * Changed function name * Changed ORT_API_VERSION to 4 * Moved all_providers array to constants.h * Move check for providers to constants.h * Changed name of array to avoid warning * Address review comment * Added unit test
2020-06-22 02:10:25 +00:00
int len = 0;
Add new api function At() (#4457) * add modern standards to function arguments * add first version of At for better tensor element access
2020-08-12 01:34:03 +00:00
char** providers;
Added GetAvailableProviders() to C API (#4247) * Added GetAvailableProviders to C API * Fix API version and Windows build error * Changed function name * Changed ORT_API_VERSION to 4 * Moved all_providers array to constants.h * Move check for providers to constants.h * Changed name of array to avoid warning * Address review comment * Added unit test
2020-06-22 02:10:25 +00:00
ASSERT_EQ(g_ort->GetAvailableProviders(&providers, &len), nullptr);
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
ASSERT_GT(len, 0);
Add ability to track per operator types in reduced build config. (#6428) * Add ability to generate configuration that includes required types for individual operators, to allow build size reduction based on that. - Add python bindings for ORT format models - Add script to update bindings and help info - Add parsing of ORT format models - Add ability to enable type reduction to config generation - Update build.py to only allow operator/type reduction via config - simpler to require config to be generated first - can't mix a type aware (ORT format model only) and non-type aware config as that may result in insufficient types being enabled - Add script to create reduced build config - Update CIs
2021-01-28 21:59:51 +00:00
ASSERT_STREQ(providers[len - 1], "CPUExecutionProvider");
Added GetAvailableProviders() to C API (#4247) * Added GetAvailableProviders to C API * Fix API version and Windows build error * Changed function name * Changed ORT_API_VERSION to 4 * Moved all_providers array to constants.h * Move check for providers to constants.h * Changed name of array to avoid warning * Address review comment * Added unit test
2020-06-22 02:10:25 +00:00
ASSERT_EQ(g_ort->ReleaseAvailableProviders(providers, len), nullptr);
}
Add C++ wrapper for GetAvailableProviders() C API (#4313)
2020-06-25 07:41:55 +00:00
TEST(CApiTest, get_available_providers_cpp) {
std::vector<std::string> providers = Ort::GetAvailableProviders();
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
ASSERT_FALSE(providers.empty());
ASSERT_EQ(providers.back(), "CPUExecutionProvider");
Support more C# APIs (#5608)
2020-10-31 02:19:50 +00:00
#ifdef USE_CUDA
// CUDA EP will exist in the list but its position may vary based on other EPs included in the build
Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up.
2021-01-07 18:10:55 +00:00
ASSERT_TRUE(std::find(providers.begin(), providers.end(), "CUDAExecutionProvider") != providers.end());
Support more C# APIs (#5608)
2020-10-31 02:19:50 +00:00
#endif
Add C++ wrapper for GetAvailableProviders() C API (#4313)
2020-06-25 07:41:55 +00:00
}
Add GetVersionSting API for C++, C# and Python (#14873) ### Description Added APIs. ### Motivation and Context Addresses https://github.com/microsoft/onnxruntime/issues/14584 Cc: @Craigacp cp
2023-03-03 01:11:07 +00:00
TEST(CApiTest, get_version_string_cpp) {
Adust GetVersionString() GetBuildInfoString() signatures and move them to OrtApi (#15921) ### Description This PR partially reverts changes introduced in https://github.com/microsoft/onnxruntime/pull/15643 We make two API return std::string always in UTF-8. We also move the entry points from OrtApiBase to OrtApi to make them versioned. ### Motivation and Context `GetVersionString` always returns x.y.z numbers that are not subject to internationalization. `GetBuildInfoString` can hold international chars, but UTF-8 should be fine to contain those. We prefix them with u8"" in case the compiler default charset is not UTF-8. Furthermore, creating platform dependent APIs is discouraged. `ORTCHAR_T` is platform dependent and was created for paths only. On non-unix platforms would still produce `std::string` that can only contain UTF-8 The API was introduced after the latest release, and can still be adjusted.
2023-05-13 20:45:07 +00:00
auto version_string = Ort::GetVersionString();
Add GetVersionSting API for C++, C# and Python (#14873) ### Description Added APIs. ### Motivation and Context Addresses https://github.com/microsoft/onnxruntime/issues/14584 Cc: @Craigacp cp
2023-03-03 01:11:07 +00:00
ASSERT_FALSE(version_string.empty());
Adust GetVersionString() GetBuildInfoString() signatures and move them to OrtApi (#15921) ### Description This PR partially reverts changes introduced in https://github.com/microsoft/onnxruntime/pull/15643 We make two API return std::string always in UTF-8. We also move the entry points from OrtApiBase to OrtApi to make them versioned. ### Motivation and Context `GetVersionString` always returns x.y.z numbers that are not subject to internationalization. `GetBuildInfoString` can hold international chars, but UTF-8 should be fine to contain those. We prefix them with u8"" in case the compiler default charset is not UTF-8. Furthermore, creating platform dependent APIs is discouraged. `ORTCHAR_T` is platform dependent and was created for paths only. On non-unix platforms would still produce `std::string` that can only contain UTF-8 The API was introduced after the latest release, and can still be adjusted.
2023-05-13 20:45:07 +00:00
ASSERT_EQ(version_string, std::string(ORT_VERSION));
}
TEST(CApiTest, get_build_info_string) {
auto build_info_string = Ort::GetBuildInfoString();
ASSERT_FALSE(build_info_string.empty());
Add GetVersionSting API for C++, C# and Python (#14873) ### Description Added APIs. ### Motivation and Context Addresses https://github.com/microsoft/onnxruntime/issues/14584 Cc: @Craigacp cp
2023-03-03 01:11:07 +00:00
}
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
TEST(CApiTest, TestSharedAllocators) {
OrtEnv* env_ptr = (OrtEnv*)(*ort_env);
Allow multiple sessions to share an allocator, optimize constant folding memory usage, expose arena configs. (#4813) * Add support for sharing allocators * Incremental update * Address some PR comments, add unit tests, add documentation. * Address PR comments, add tests and some documentation. * Fix build and test issues * Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this. Also fixed the orttraining segfault. The segfault was because in the case of training session, the CPU exec prov is not available at the time the transformers are applied. Changed it to create a new one.
2020-08-22 17:03:17 +00:00
// prepare inputs
std::vector<Input> inputs(1);
Input& input = inputs.back();
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
auto allocator_for_input_memory_allocation = std::make_unique<MockedOrtAllocator>();
Allow multiple sessions to share an allocator, optimize constant folding memory usage, expose arena configs. (#4813) * Add support for sharing allocators * Incremental update * Address some PR comments, add unit tests, add documentation. * Address PR comments, add tests and some documentation. * Fix build and test issues * Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this. Also fixed the orttraining segfault. The segfault was because in the case of training session, the CPU exec prov is not available at the time the transformers are applied. Changed it to create a new one.
2020-08-22 17:03:17 +00:00
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
// prepare expected outputs
Allow multiple sessions to share an allocator, optimize constant folding memory usage, expose arena configs. (#4813) * Add support for sharing allocators * Incremental update * Address some PR comments, add unit tests, add documentation. * Address PR comments, add tests and some documentation. * Fix build and test issues * Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this. Also fixed the orttraining segfault. The segfault was because in the case of training session, the CPU exec prov is not available at the time the transformers are applied. Changed it to create a new one.
2020-08-22 17:03:17 +00:00
std::vector<int64_t> expected_dims_y = {3, 2};
std::vector<float> expected_values_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
// Create session options and configure it appropriately
Ort::SessionOptions session_options;
// Turn on sharing of the allocator between sessions
session_options.AddConfigEntry(kOrtSessionOptionsConfigUseEnvAllocators, "1");
Allow multiple sessions to share an allocator, optimize constant folding memory usage, expose arena configs. (#4813) * Add support for sharing allocators * Incremental update * Address some PR comments, add unit tests, add documentation. * Address PR comments, add tests and some documentation. * Fix build and test issues * Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this. Also fixed the orttraining segfault. The segfault was because in the case of training session, the CPU exec prov is not available at the time the transformers are applied. Changed it to create a new one.
2020-08-22 17:03:17 +00:00
const auto& api = Ort::GetApi();
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
// CASE 1: We test creating and registering an ORT-internal allocator implementation instance
// for sharing between sessions
{
OrtMemoryInfo* mem_info = nullptr;
ASSERT_TRUE(api.CreateCpuMemoryInfo(OrtArenaAllocator, OrtMemTypeDefault, &mem_info) == nullptr);
std::unique_ptr<OrtMemoryInfo, decltype(api.ReleaseMemoryInfo)> rel_info(mem_info, api.ReleaseMemoryInfo);
OrtArenaCfg* arena_cfg = nullptr;
ASSERT_TRUE(api.CreateArenaCfg(0, -1, -1, -1, &arena_cfg) == nullptr);
std::unique_ptr<OrtArenaCfg, decltype(api.ReleaseArenaCfg)> rel_arena_cfg(arena_cfg, api.ReleaseArenaCfg);
// This creates an ORT-internal allocator instance and registers it in the environment for sharing
// NOTE: On x86 builds arenas are not supported and will default to using non-arena based allocator
ASSERT_TRUE(api.CreateAndRegisterAllocator(env_ptr, mem_info, arena_cfg) == nullptr);
// Test that duplicates are handled
std::unique_ptr<OrtStatus, decltype(api.ReleaseStatus)> status_releaser(
api.CreateAndRegisterAllocator(env_ptr, mem_info, arena_cfg),
api.ReleaseStatus);
ASSERT_FALSE(status_releaser.get() == nullptr);
{
// create session 1
Ort::Session session1(*ort_env, MODEL_URI, session_options);
RunSession<float>(allocator_for_input_memory_allocation.get(),
session1,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
// create session 2
Ort::Session session2(*ort_env, MODEL_URI, session_options);
RunSession<float>(allocator_for_input_memory_allocation.get(),
session2,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
}
Expose knobs to create and share (CPU) allocators across sessions in C# and Python (#5634)
2020-11-21 22:12:33 +00:00
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
// Remove the registered shared allocator for part 2 of this test
// where-in we will register a custom allocator for the same device.
ASSERT_TRUE(api.UnregisterAllocator(env_ptr, mem_info) == nullptr);
}
Allow multiple sessions to share an allocator, optimize constant folding memory usage, expose arena configs. (#4813) * Add support for sharing allocators * Incremental update * Address some PR comments, add unit tests, add documentation. * Address PR comments, add tests and some documentation. * Fix build and test issues * Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this. Also fixed the orttraining segfault. The segfault was because in the case of training session, the CPU exec prov is not available at the time the transformers are applied. Changed it to create a new one.
2020-08-22 17:03:17 +00:00
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
// CASE 2: We test registering a custom allocator implementation
// for sharing between sessions
{
// This creates a custom allocator instance and registers it in the environment for sharing
// NOTE: This is a very basic allocator implementation. For optimal performance, allocations
// need to be aligned for certain devices/build configurations/math libraries.
// See docs/C_API.md for details.
MockedOrtAllocator custom_allocator;
ASSERT_TRUE(api.RegisterAllocator(env_ptr, &custom_allocator) == nullptr);
// Test that duplicates are handled
std::unique_ptr<OrtStatus, decltype(api.ReleaseStatus)>
status_releaser(
api.RegisterAllocator(env_ptr, &custom_allocator),
api.ReleaseStatus);
ASSERT_FALSE(status_releaser.get() == nullptr);
{
// Keep this scoped to destroy the underlying sessions after use
// This should trigger frees in our custom allocator
// create session 1
Ort::Session session1(*ort_env, MODEL_URI, session_options);
RunSession<float>(allocator_for_input_memory_allocation.get(),
session1,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
// create session 2
Ort::Session session2(*ort_env, MODEL_URI, session_options);
RunSession<float>(allocator_for_input_memory_allocation.get(),
session2,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
}
Allow multiple sessions to share an allocator, optimize constant folding memory usage, expose arena configs. (#4813) * Add support for sharing allocators * Incremental update * Address some PR comments, add unit tests, add documentation. * Address PR comments, add tests and some documentation. * Fix build and test issues * Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this. Also fixed the orttraining segfault. The segfault was because in the case of training session, the CPU exec prov is not available at the time the transformers are applied. Changed it to create a new one.
2020-08-22 17:03:17 +00:00
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
// Remove the registered shared allocator from the global environment
// (common to all tests) to prevent its accidental usage elsewhere
ASSERT_TRUE(api.UnregisterAllocator(env_ptr, custom_allocator.Info()) == nullptr);
Allow multiple sessions to share an allocator, optimize constant folding memory usage, expose arena configs. (#4813) * Add support for sharing allocators * Incremental update * Address some PR comments, add unit tests, add documentation. * Address PR comments, add tests and some documentation. * Fix build and test issues * Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this. Also fixed the orttraining segfault. The segfault was because in the case of training session, the CPU exec prov is not available at the time the transformers are applied. Changed it to create a new one.
2020-08-22 17:03:17 +00:00
Support plugging in custom user-defined allocators for sharing between sessions (#8059)
2021-07-22 17:17:35 +00:00
// Ensure that the registered custom allocator was indeed used for both sessions
// We should have seen 2 allocations per session (one for the sole initializer
// and one for the output). So, for two sessions, we should have seen 4 allocations.
size_t num_allocations = custom_allocator.NumAllocations();
ASSERT_TRUE(num_allocations == 4);
// Ensure that there was no leak
custom_allocator.LeakCheck();
}
ExecutionProvider API refactor - move allocator from EP level to SessionState level and indexed by OrtDevice (#15833) ### Description This PR is to refactor ExecutionProvider API for memory management, which is to move allocators from EP level to SessionState level and indexed by OrtDevice ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> This PR is to refactor ExecutionProvider API for memory management, which is to move allocators from EP level to SessionState level and indexed by OrtDevice. By this change, EP level will shift the burden of maintaining allocators, which will be user friendly for EP developers --------- Co-authored-by: Lei Cao <leca@microsoft.com@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
2023-06-20 00:44:45 +00:00
#ifdef USE_CUDA
{
OrtMemoryInfo* cuda_meminfo = nullptr;
ASSERT_TRUE(api.CreateMemoryInfo("Cuda", OrtArenaAllocator, 0, OrtMemTypeDefault, &cuda_meminfo) == nullptr);
std::unique_ptr<OrtMemoryInfo, decltype(api.ReleaseMemoryInfo)> rel_info(cuda_meminfo, api.ReleaseMemoryInfo);
OrtArenaCfg* arena_cfg = nullptr;
ASSERT_TRUE(api.CreateArenaCfg(0, -1, -1, -1, &arena_cfg) == nullptr);
std::unique_ptr<OrtArenaCfg, decltype(api.ReleaseArenaCfg)> rel_arena_cfg(arena_cfg, api.ReleaseArenaCfg);
std::vector<const char*> keys, values;
ASSERT_TRUE(api.CreateAndRegisterAllocatorV2(env_ptr, onnxruntime::kCudaExecutionProvider, cuda_meminfo, arena_cfg, keys.data(), values.data(), 0) == nullptr);
// Test that duplicates are handled
std::unique_ptr<OrtStatus, decltype(api.ReleaseStatus)> status_releaser(
api.CreateAndRegisterAllocatorV2(env_ptr, onnxruntime::kCudaExecutionProvider, cuda_meminfo, arena_cfg, keys.data(), values.data(), 0),
api.ReleaseStatus);
ASSERT_FALSE(status_releaser.get() == nullptr);
{
// create session 1
Ort::SessionOptions cuda_session_options;
cuda_session_options.AddConfigEntry(kOrtSessionOptionsConfigUseEnvAllocators, "1");
cuda_session_options.AppendExecutionProvider_CUDA(OrtCUDAProviderOptions{});
Ort::Session session1(*ort_env, MODEL_URI, cuda_session_options);
RunSession<float>(allocator_for_input_memory_allocation.get(),
session1,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
// create session 2
Ort::Session session2(*ort_env, MODEL_URI, cuda_session_options);
RunSession<float>(allocator_for_input_memory_allocation.get(),
session2,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
}
ASSERT_TRUE(api.UnregisterAllocator(env_ptr, cuda_meminfo) == nullptr);
}
#endif
Missing header for `std::once_flag` and `std::call_once`. (#5010)
2020-09-02 07:46:59 +00:00
}
Allow sharing of initializers between sessions. (#5092) * Allow sharing of initializers between sessions. * Allow sharing of initializers between sessions (2). * Add test for C# * Add test for C#; address PR comments * Address PR comments Moved AddInitializer logic to internal session options Added tests for owned buffer Clarified documentation Fix bug where memory info and not device was getting compared * Fix test * Fix training build * Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
2020-09-21 21:09:37 +00:00
Some cosmetic changes (#7741)
2021-05-18 07:02:07 +00:00
TEST(CApiTest, TestSharingOfInitializerAndItsPrepackedVersion) {
Allow sharing of initializers between sessions. (#5092) * Allow sharing of initializers between sessions. * Allow sharing of initializers between sessions (2). * Add test for C# * Add test for C#; address PR comments * Address PR comments Moved AddInitializer logic to internal session options Added tests for owned buffer Clarified documentation Fix bug where memory info and not device was getting compared * Fix test * Fix training build * Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
2020-09-21 21:09:37 +00:00
// simple inference test
// prepare inputs
std::vector<Input> inputs(1);
Input& input = inputs.back();
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
// prepare expected inputs and outputs
Add ability for pre-packed weights of shared initializers to be shared across sessions (#7421)
2021-05-15 03:44:42 +00:00
std::vector<int64_t> expected_dims_y = {3, 1};
std::vector<float> expected_values_y = {4.0f, 10.0f, 16.0f};
Allow sharing of initializers between sessions. (#5092) * Allow sharing of initializers between sessions. * Allow sharing of initializers between sessions (2). * Add test for C# * Add test for C#; address PR comments * Address PR comments Moved AddInitializer logic to internal session options Added tests for owned buffer Clarified documentation Fix bug where memory info and not device was getting compared * Fix test * Fix training build * Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
2020-09-21 21:09:37 +00:00
Ort::SessionOptions session_options;
Ort::MemoryInfo mem_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Support sharing of initializers between session via the Python API (#5407)
2020-10-10 03:26:28 +00:00
// These values are different from the actual initializer values in the model
Add ability for pre-packed weights of shared initializers to be shared across sessions (#7421)
2021-05-15 03:44:42 +00:00
float data[] = {2.0f, 1.0f};
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
constexpr int data_len = sizeof(data) / sizeof(data[0]);
Add ability for pre-packed weights of shared initializers to be shared across sessions (#7421)
2021-05-15 03:44:42 +00:00
const int64_t shape[] = {2, 1};
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
constexpr size_t shape_len = sizeof(shape) / sizeof(shape[0]);
Allow sharing of initializers between sessions. (#5092) * Allow sharing of initializers between sessions. * Allow sharing of initializers between sessions (2). * Add test for C# * Add test for C#; address PR comments * Address PR comments Moved AddInitializer logic to internal session options Added tests for owned buffer Clarified documentation Fix bug where memory info and not device was getting compared * Fix test * Fix training build * Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
2020-09-21 21:09:37 +00:00
Ort::Value val = Ort::Value::CreateTensor<float>(mem_info, data, data_len, shape, shape_len);
session_options.AddInitializer("W", val);
Add ability for pre-packed weights of shared initializers to be shared across sessions (#7421)
2021-05-15 03:44:42 +00:00
const auto& api = Ort::GetApi();
OrtPrepackedWeightsContainer* prepacked_weights_container = nullptr;
ASSERT_TRUE(api.CreatePrepackedWeightsContainer(&prepacked_weights_container) == nullptr);
std::unique_ptr<OrtPrepackedWeightsContainer, decltype(api.ReleasePrepackedWeightsContainer)>
rel_prepacked_weights_container(prepacked_weights_container, api.ReleasePrepackedWeightsContainer);
Change onnxruntime::make_unique to std::make_unique (#7502) 1. Change onnxruntime::make_unique to std::make_unique 2. Add "-std=c++14" to ROCM EP's build flags.
2021-04-30 00:04:53 +00:00
auto default_allocator = std::make_unique<MockedOrtAllocator>();
Add ability for pre-packed weights of shared initializers to be shared across sessions (#7421)
2021-05-15 03:44:42 +00:00
Add C++ Session ctor taking model bytes and OrtPrepackedWeightsContainer (#12333)
2022-07-29 19:32:43 +00:00
// create session 1 (using model path)
Add ability for pre-packed weights of shared initializers to be shared across sessions (#7421)
2021-05-15 03:44:42 +00:00
Ort::Session session1(*ort_env, MATMUL_MODEL_URI, session_options, prepacked_weights_container);
Allow sharing of initializers between sessions. (#5092) * Allow sharing of initializers between sessions. * Allow sharing of initializers between sessions (2). * Add test for C# * Add test for C#; address PR comments * Address PR comments Moved AddInitializer logic to internal session options Added tests for owned buffer Clarified documentation Fix bug where memory info and not device was getting compared * Fix test * Fix training build * Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
2020-09-21 21:09:37 +00:00
RunSession<float>(default_allocator.get(),
session1,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
Add C++ Session ctor taking model bytes and OrtPrepackedWeightsContainer (#12333)
2022-07-29 19:32:43 +00:00
// create session 2 (using model bytes)
std::ifstream model_file_stream(MATMUL_MODEL_URI, std::ios::in | std::ios::binary);
ASSERT_TRUE(model_file_stream.good());
model_file_stream.seekg(0, std::ios::end);
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
const auto size = onnxruntime::narrow<size_t>(model_file_stream.tellg());
Add C++ Session ctor taking model bytes and OrtPrepackedWeightsContainer (#12333)
2022-07-29 19:32:43 +00:00
model_file_stream.seekg(0, std::ios::beg);
std::vector<char> file_contents(size, 0);
model_file_stream.read(&file_contents[0], size);
model_file_stream.close();
Ort::Session session2(*ort_env, file_contents.data(), size, session_options, prepacked_weights_container);
Allow sharing of initializers between sessions. (#5092) * Allow sharing of initializers between sessions. * Allow sharing of initializers between sessions (2). * Add test for C# * Add test for C#; address PR comments * Address PR comments Moved AddInitializer logic to internal session options Added tests for owned buffer Clarified documentation Fix bug where memory info and not device was getting compared * Fix test * Fix training build * Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
2020-09-21 21:09:37 +00:00
RunSession<float>(default_allocator.get(),
session2,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
Add valgrind support to our cmake files (#5296)
2020-09-28 16:31:08 +00:00
}
Fix Linux/Mac error message on input type mismatch (#6256)
2021-01-06 06:21:24 +00:00
#ifndef ORT_NO_RTTI
TEST(CApiTest, TestIncorrectInputTypeToModel_Tensors) {
// simple inference test
// prepare inputs (incorrect type)
Ort::MemoryInfo mem_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
double data[] = {2., 1., 4., 3., 6., 5.};
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
constexpr int data_len = sizeof(data) / sizeof(data[0]);
Fix Linux/Mac error message on input type mismatch (#6256)
2021-01-06 06:21:24 +00:00
const int64_t shape[] = {3, 2};
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
constexpr size_t shape_len = sizeof(shape) / sizeof(shape[0]);
Fix Linux/Mac error message on input type mismatch (#6256)
2021-01-06 06:21:24 +00:00
Ort::Value val = Ort::Value::CreateTensor<double>(mem_info, data, data_len, shape, shape_len);
std::vector<const char*> input_names{"X"};
const char* output_names[] = {"Y"};
Ort::SessionOptions session_options;
Ort::Session session(*ort_env, MODEL_URI, session_options);
bool exception_thrown = false;
try {
auto outputs = session.Run(Ort::RunOptions{nullptr}, input_names.data(), &val, 1, output_names, 1);
} catch (const Ort::Exception& ex) {
exception_thrown = true;
const char* exception_string = ex.what();
ASSERT_TRUE(strcmp(exception_string,
"Unexpected input data type. Actual: (tensor(double)) , expected: (tensor(float))") == 0);
}
ASSERT_TRUE(exception_thrown);
}
TEST(CApiTest, TestIncorrectInputTypeToModel_SequenceTensors) {
// simple inference test
// prepare inputs (incorrect type)
Ort::MemoryInfo mem_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
double data[] = {2., 1., 4., 3., 6., 5.};
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
constexpr int data_len = sizeof(data) / sizeof(data[0]);
Fix Linux/Mac error message on input type mismatch (#6256)
2021-01-06 06:21:24 +00:00
const int64_t shape[] = {2, 3};
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
constexpr size_t shape_len = sizeof(shape) / sizeof(shape[0]);
Fix Linux/Mac error message on input type mismatch (#6256)
2021-01-06 06:21:24 +00:00
Ort::Value val = Ort::Value::CreateTensor<double>(mem_info, data, data_len, shape, shape_len);
std::vector<Ort::Value> seq;
seq.push_back(std::move(val));
Ort::Value seq_value = Ort::Value::CreateSequence(seq);
std::vector<const char*> input_names{"X"};
const char* output_names[] = {"Y"};
Ort::SessionOptions session_options;
Ort::Session session(*ort_env, SEQUENCE_MODEL_URI, session_options);
bool exception_thrown = false;
try {
auto outputs = session.Run(Ort::RunOptions{nullptr}, input_names.data(), &seq_value, 1, output_names, 1);
} catch (const Ort::Exception& ex) {
exception_thrown = true;
const char* exception_string = ex.what();
ASSERT_TRUE(strcmp(exception_string,
"Unexpected input data type. Actual: (seq(double)) , expected: (seq(float))") == 0);
}
ASSERT_TRUE(exception_thrown);
}
#endif
Add ability to allocate initialized tensor memory from non-arena memory (#7267)
2021-04-21 03:27:48 +00:00
Some cosmetic changes (#7741)
2021-05-18 07:02:07 +00:00
TEST(CApiTest, AllocateInitializersFromNonArenaMemory) {
Add ability to allocate initialized tensor memory from non-arena memory (#7267)
2021-04-21 03:27:48 +00:00
Ort::SessionOptions session_options;
#ifdef USE_CUDA
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, 0));
#else
// arena is enabled but the sole initializer will still be allocated from non-arena memory
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CPU(session_options, 1));
#endif
// disable using arena for the sole initializer in the model
session_options.AddConfigEntry(kOrtSessionOptionsUseDeviceAllocatorForInitializers, "1");
// This is mostly an usage example - if the logging level for the default logger is made INFO (by default it is at WARNING)
// when the Ort::Env instance is instantiated, logs pertaining to initializer memory being allocated from non-arena memory
// can be confirmed by seeing logs like "Reserving memory in BFCArena...".
Ort::Session session(*ort_env, MODEL_URI, session_options);
}
Add ability for memory arenas to "shrink" periodically (#7284)
2021-05-08 14:53:21 +00:00
#ifdef USE_CUDA
// Usage example showing how to use CreateArenaCfgV2() API to configure the default memory CUDA arena allocator
Some cosmetic changes (#7741)
2021-05-18 07:02:07 +00:00
TEST(CApiTest, ConfigureCudaArenaAndDemonstrateMemoryArenaShrinkage) {
Add ability for memory arenas to "shrink" periodically (#7284)
2021-05-08 14:53:21 +00:00
const auto& api = Ort::GetApi();
Ort::SessionOptions session_options;
New configuration to limit the arena extension (#15983) Add a configuration `max_power_of_two_extend_bytes ` to limit the arena extension size. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> In our real scenario, we observe that if the model is big enough the BfcArena will extend uncontrollable. As showed by the following figures, if a model uses more than 16GB memory, the BfcArena will totally apply for 32GB memory according to the `kNextPowerOfTwo` strategy. With the new strategy, the extension is limited. The default maximum extension size is 1GB. #### Without the new configuration After loading the model, ORT uses 32G GPU memory. ![image](https://github.com/microsoft/onnxruntime/assets/19584326/42b93c66-b957-4f20-a13b-d34cb390afff) #### With the new configuration After loading the model, ORT uses 23G GPU memory. ![image](https://github.com/microsoft/onnxruntime/assets/19584326/5abffeff-9ca3-4187-a262-37fd2764fe1b) Co-authored-by: Yuhong Guo <yuhong.gyh@antgroup.com>
2023-05-25 09:19:07 +00:00
const char* keys[] = {"max_mem", "arena_extend_strategy", "initial_chunk_size_bytes", "max_dead_bytes_per_chunk", "initial_growth_chunk_size_bytes", "max_power_of_two_extend_bytes"};
const size_t values[] = {0 /*let ort pick default max memory*/, 0, 1024, 0, 256, 1L << 24};
Add ability for memory arenas to "shrink" periodically (#7284)
2021-05-08 14:53:21 +00:00
OrtArenaCfg* arena_cfg = nullptr;
ASSERT_TRUE(api.CreateArenaCfgV2(keys, values, 5, &arena_cfg) == nullptr);
std::unique_ptr<OrtArenaCfg, decltype(api.ReleaseArenaCfg)> rel_arena_cfg(arena_cfg, api.ReleaseArenaCfg);
OrtCUDAProviderOptions cuda_provider_options = CreateDefaultOrtCudaProviderOptionsWithCustomStream(nullptr);
cuda_provider_options.default_memory_arena_cfg = arena_cfg;
session_options.AppendExecutionProvider_CUDA(cuda_provider_options);
Ort::Session session(*ort_env, MODEL_URI, session_options);
// Use a run option like this while invoking Run() to trigger a memory arena shrinkage post Run()
// This will shrink memory allocations left unused at the end of Run() and cap the arena growth
// This does come with associated costs as there are costs to cudaFree() but the goodness it offers
// is that the memory held by the arena (memory pool) is kept checked.
Ort::RunOptions run_option;
run_option.AddConfigEntry(kOrtRunOptionsConfigEnableMemoryArenaShrinkage, "gpu:0");
// To also trigger a cpu memory arena shrinkage along with the gpu arena shrinkage, use the following-
// (Memory arena for the CPU should not have been disabled)
// run_option.AddConfigEntry(kOrtRunOptionsConfigEnableMemoryArenaShrinkage, "cpu:0;gpu:0");
}
[js/react_native] Use a mobile ORT instead of a full ORT (#8042) * Change full ort to mobile ort * Update Android example to load mobile ort * Change the format of test models to ort * update ios to use mobile ort * revise README * use onnxruntime-mobile-c CocoaPods in a npm package
2021-06-15 20:36:05 +00:00
#endif
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
#ifdef USE_TENSORRT
[TensorRT EP] Fix bug for shape tensor input (#18253) When the model has "shape tensor" as one of the inputs and user provides explicit profile shapes for it, TRT EP doesn't correctly set the "shape tensor" input. Also, there is a bug for applying explicit profile shapes for the shape tensor input. Note: It seems the model has shape tensor input is a rare case. Most of the cases, the inputs are all execution tensor.
2023-11-03 23:07:50 +00:00
TEST(TensorrtExecutionProviderTest, ShapeTensorTest) {
const auto& api = Ort::GetApi();
// Test input tensor which is shape tensor with explicit trt profile shapes
Ort::SessionOptions session_options;
OrtTensorRTProviderOptionsV2* trt_options;
ASSERT_TRUE(api.CreateTensorRTProviderOptions(&trt_options) == nullptr);
std::unique_ptr<OrtTensorRTProviderOptionsV2, decltype(api.ReleaseTensorRTProviderOptions)>
rel_trt_options(trt_options, api.ReleaseTensorRTProviderOptions);
const char* trt_profile_min_shapes = "data:2x2,shape:4x1";
const char* trt_profile_max_shapes = "data:2x2,shape:4x1";
const char* trt_profile_opt_shapes = "data:2x2,shape:4x1";
std::vector<const char*> keys{"trt_profile_min_shapes", "trt_profile_max_shapes", "trt_profile_opt_shapes"};
std::vector<const char*> values{trt_profile_min_shapes, trt_profile_max_shapes, trt_profile_opt_shapes};
ASSERT_TRUE(api.UpdateTensorRTProviderOptions(rel_trt_options.get(), keys.data(), values.data(), keys.size()) == nullptr);
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_TensorRT_V2(
static_cast<OrtSessionOptions*>(session_options),
rel_trt_options.get()) == nullptr);
auto model_path = ORT_TSTR("testdata/trt_reshape.onnx");
std::vector<float> input_value_0{1.1f, 1.2f, 1.3f, 1.4f};
std::vector<int64_t> input_shape_0{2, 2};
std::vector<int64_t> input_value_1{4, 1};
std::vector<int64_t> input_shape_1{2};
std::vector<const char*> input_names{"data", "shape"};
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
ort_inputs.emplace_back(Ort::Value::CreateTensor<float>(info, input_value_0.data(), input_value_0.size(), input_shape_0.data(), input_shape_0.size()));
ort_inputs.emplace_back(Ort::Value::CreateTensor<int64_t>(info, input_value_1.data(), input_value_1.size(), input_shape_1.data(), input_shape_1.size()));
const char* output_names[] = {"reshaped"};
Ort::Session session(*ort_env, model_path, session_options);
session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(), output_names, countof(output_names));
// Test input tensor which is shape tensor with implicit trt profile shapes
Ort::SessionOptions session_options_2;
OrtTensorRTProviderOptionsV2* trt_options_2;
ASSERT_TRUE(api.CreateTensorRTProviderOptions(&trt_options_2) == nullptr);
std::unique_ptr<OrtTensorRTProviderOptionsV2, decltype(api.ReleaseTensorRTProviderOptions)>
rel_trt_options_2(trt_options_2, api.ReleaseTensorRTProviderOptions);
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_TensorRT_V2(
static_cast<OrtSessionOptions*>(session_options_2),
rel_trt_options_2.get()) == nullptr);
Ort::Session session_2(*ort_env, model_path, session_options_2);
session_2.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(), output_names, countof(output_names));
}
[TensorRT EP] Add unit test for user provided cuda stream (#17974) Add a unit test for testing user provided CUDA stream
2023-10-24 02:41:15 +00:00
TEST(CApiTest, TestExternalCUDAStreamWithIOBinding) {
const auto& api = Ort::GetApi();
Ort::SessionOptions session_options;
OrtTensorRTProviderOptionsV2* trt_options;
ASSERT_TRUE(api.CreateTensorRTProviderOptions(&trt_options) == nullptr);
std::unique_ptr<OrtTensorRTProviderOptionsV2, decltype(api.ReleaseTensorRTProviderOptions)>
rel_trt_options(trt_options, api.ReleaseTensorRTProviderOptions);
// updating provider option with user provided compute stream
cudaStream_t compute_stream = nullptr;
void* user_compute_stream = nullptr;
cudaStreamCreate(&compute_stream);
ASSERT_TRUE(api.UpdateTensorRTProviderOptionsWithValue(rel_trt_options.get(), "user_compute_stream", compute_stream) == nullptr);
ASSERT_TRUE(api.GetTensorRTProviderOptionsByName(rel_trt_options.get(), "user_compute_stream", &user_compute_stream) == nullptr);
ASSERT_TRUE(user_compute_stream == (void*)compute_stream);
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_TensorRT_V2(
static_cast<OrtSessionOptions*>(session_options),
rel_trt_options.get()) == nullptr);
Ort::Session session(*ort_env, MODEL_URI, session_options);
Ort::MemoryInfo info_cuda("Cuda", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
const std::array<int64_t, 2> x_shape = {3, 2};
std::array<float, 3 * 2> x_values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
/*
* Use cudaMallocHost() (pinned memory allocation) to create input/output tensors
*/
float* input_data;
cudaMallocHost(&input_data, 3 * 2 * sizeof(float));
ASSERT_NE(input_data, nullptr);
cudaMemcpy(input_data, x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);
std::cout << "pinned memory allocation" << std::endl;
std::cout << "input tesnor:" << std::endl;
for (int i = 0; i < 6; i++) {
std::cout << input_data[i] << std::endl;
}
// Create an OrtValue tensor backed by data on CUDA memory
Ort::Value bound_x = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(input_data), x_values.size(),
x_shape.data(), x_shape.size());
const std::array<int64_t, 2> expected_y_shape = {3, 2};
std::array<float, 3 * 2> expected_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
float* output_data;
cudaMallocHost(&output_data, 3 * 2 * sizeof(float));
ASSERT_NE(output_data, nullptr);
// Create an OrtValue tensor backed by data on CUDA memory
Ort::Value bound_y = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(output_data),
expected_y.size(), expected_y_shape.data(), expected_y_shape.size());
// Create IoBinding for inputs and outputs.
Ort::IoBinding binding(session);
binding.BindInput("X", bound_x);
binding.BindOutput("Y", bound_y);
/*
* Use cudaMalloc() (pageable memory allocation first and then implicit pinned memory allocation) to create input/output tensors
*/
float* input_data_2;
cudaMalloc(&input_data_2, 3 * 2 * sizeof(float));
ASSERT_NE(input_data_2, nullptr);
cudaMemcpy(input_data_2, x_values.data(), sizeof(float) * x_values.size(), cudaMemcpyHostToDevice);
// Create an OrtValue tensor backed by data on CUDA memory
Ort::Value bound_x_2 = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(input_data_2), x_values.size(),
x_shape.data(), x_shape.size());
float* output_data_2;
cudaMalloc(&output_data_2, 3 * 2 * sizeof(float));
ASSERT_NE(output_data_2, nullptr);
// Create an OrtValue tensor backed by data on CUDA memory
Ort::Value bound_y_2 = Ort::Value::CreateTensor(info_cuda, reinterpret_cast<float*>(output_data_2),
expected_y.size(), expected_y_shape.data(), expected_y_shape.size());
// Create IoBinding for inputs and outputs.
Ort::IoBinding binding_2(session);
binding_2.BindInput("X", bound_x_2);
binding_2.BindOutput("Y", bound_y_2);
// Run with first iobindings
session.Run(Ort::RunOptions(), binding);
// Check the values against the bound raw memory (needs copying from device to host first)
std::array<float, 3 * 2> y_values;
cudaMemcpy(y_values.data(), output_data, sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
std::cout << "pinned memory allocation" << std::endl;
std::cout << "output: " << std::endl;
for (auto y : y_values) {
std::cout << y << std::endl;
}
ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
// Run with second iobindings
session.Run(Ort::RunOptions(), binding_2);
// Check the values against the bound raw memory (needs copying from device to host first)
cudaMemcpy(y_values.data(), output_data_2, sizeof(float) * y_values.size(), cudaMemcpyDeviceToHost);
std::cout << "pageable memory allocation" << std::endl;
std::cout << "output: " << std::endl;
for (auto y : y_values) {
std::cout << y << std::endl;
}
ASSERT_THAT(y_values, ::testing::ContainerEq(expected_y));
// Clean up
binding.ClearBoundInputs();
binding.ClearBoundOutputs();
binding_2.ClearBoundInputs();
binding_2.ClearBoundOutputs();
cudaFreeHost(input_data);
cudaFreeHost(output_data);
cudaFree(input_data_2);
cudaFree(output_data_2);
cudaStreamDestroy(compute_stream);
}
Exposing new TRT build options (#15089) ### Description This will add a few TRT options, some of them are only available on TRT 8.6: - heuristics - sparsity - optimization level (8.6 only) - auxiliary stream (8.6 only) - tactic source selection I am no sure yet which tests is should add for these options. As those are mostly simple TRT flags i am not sure to what level i should test. For heuristics something similar to https://github.com/gedoensmax/onnxruntime/blob/44dda08b51e98bd86f6eafd09b7d00c3680c78a8/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc#L510-L538 should be possible for, but for all other essentially we would only be testing if there is a crash or not if the option is set. Also if i forgot some option that would be good to have feel free to speak up !
2023-04-14 16:47:36 +00:00
class CApiTensorRTTest : public testing::Test, public ::testing::WithParamInterface<std::string> {};
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
// This test uses CreateTensorRTProviderOptions/UpdateTensorRTProviderOptions APIs to configure and create a TensorRT Execution Provider
Exposing new TRT build options (#15089) ### Description This will add a few TRT options, some of them are only available on TRT 8.6: - heuristics - sparsity - optimization level (8.6 only) - auxiliary stream (8.6 only) - tactic source selection I am no sure yet which tests is should add for these options. As those are mostly simple TRT flags i am not sure to what level i should test. For heuristics something similar to https://github.com/gedoensmax/onnxruntime/blob/44dda08b51e98bd86f6eafd09b7d00c3680c78a8/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc#L510-L538 should be possible for, but for all other essentially we would only be testing if there is a crash or not if the option is set. Also if i forgot some option that would be good to have feel free to speak up !
2023-04-14 16:47:36 +00:00
TEST_P(CApiTensorRTTest, TestConfigureTensorRTProviderOptions) {
std::string param = GetParam();
size_t pos = param.find("=");
std::string option_name = param.substr(0, pos);
std::string option_value = param.substr(pos + 1);
ASSERT_NE(pos, std::string::npos);
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
const auto& api = Ort::GetApi();
OrtTensorRTProviderOptionsV2* trt_options;
OrtAllocator* allocator;
char* trt_options_str;
ASSERT_TRUE(api.CreateTensorRTProviderOptions(&trt_options) == nullptr);
std::unique_ptr<OrtTensorRTProviderOptionsV2, decltype(api.ReleaseTensorRTProviderOptions)> rel_trt_options(trt_options, api.ReleaseTensorRTProviderOptions);
const char* engine_cache_path = "./trt_engine_folder";
Add API for updating TRT EP provider option user compute stream (#16965) Add a generic `UpdateTensorRTProviderOptionsWithValue()` C API to update TensorRT provider options where its data type is pointer that can't be represented by string.
2023-08-04 22:14:43 +00:00
std::vector<const char*> keys{"device_id", "has_user_compute_stream", "trt_fp16_enable", "trt_int8_enable", "trt_engine_cache_enable",
Exposing new TRT build options (#15089) ### Description This will add a few TRT options, some of them are only available on TRT 8.6: - heuristics - sparsity - optimization level (8.6 only) - auxiliary stream (8.6 only) - tactic source selection I am no sure yet which tests is should add for these options. As those are mostly simple TRT flags i am not sure to what level i should test. For heuristics something similar to https://github.com/gedoensmax/onnxruntime/blob/44dda08b51e98bd86f6eafd09b7d00c3680c78a8/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc#L510-L538 should be possible for, but for all other essentially we would only be testing if there is a crash or not if the option is set. Also if i forgot some option that would be good to have feel free to speak up !
2023-04-14 16:47:36 +00:00
"trt_engine_cache_path", option_name.c_str()};
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
Add API for updating TRT EP provider option user compute stream (#16965) Add a generic `UpdateTensorRTProviderOptionsWithValue()` C API to update TensorRT provider options where its data type is pointer that can't be represented by string.
2023-08-04 22:14:43 +00:00
std::vector<const char*> values{"0", "0", "1", "0", "1",
Exposing new TRT build options (#15089) ### Description This will add a few TRT options, some of them are only available on TRT 8.6: - heuristics - sparsity - optimization level (8.6 only) - auxiliary stream (8.6 only) - tactic source selection I am no sure yet which tests is should add for these options. As those are mostly simple TRT flags i am not sure to what level i should test. For heuristics something similar to https://github.com/gedoensmax/onnxruntime/blob/44dda08b51e98bd86f6eafd09b7d00c3680c78a8/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc#L510-L538 should be possible for, but for all other essentially we would only be testing if there is a crash or not if the option is set. Also if i forgot some option that would be good to have feel free to speak up !
2023-04-14 16:47:36 +00:00
engine_cache_path, option_value.c_str()};
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
Exposing new TRT build options (#15089) ### Description This will add a few TRT options, some of them are only available on TRT 8.6: - heuristics - sparsity - optimization level (8.6 only) - auxiliary stream (8.6 only) - tactic source selection I am no sure yet which tests is should add for these options. As those are mostly simple TRT flags i am not sure to what level i should test. For heuristics something similar to https://github.com/gedoensmax/onnxruntime/blob/44dda08b51e98bd86f6eafd09b7d00c3680c78a8/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc#L510-L538 should be possible for, but for all other essentially we would only be testing if there is a crash or not if the option is set. Also if i forgot some option that would be good to have feel free to speak up !
2023-04-14 16:47:36 +00:00
ASSERT_TRUE(api.UpdateTensorRTProviderOptions(rel_trt_options.get(), keys.data(), values.data(), keys.size()) == nullptr);
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
ASSERT_TRUE(api.GetAllocatorWithDefaultOptions(&allocator) == nullptr);
ASSERT_TRUE(api.GetTensorRTProviderOptionsAsString(rel_trt_options.get(), allocator, &trt_options_str) == nullptr);
std::string s(trt_options_str);
ASSERT_TRUE(s.find(engine_cache_path) != std::string::npos);
Exposing new TRT build options (#15089) ### Description This will add a few TRT options, some of them are only available on TRT 8.6: - heuristics - sparsity - optimization level (8.6 only) - auxiliary stream (8.6 only) - tactic source selection I am no sure yet which tests is should add for these options. As those are mostly simple TRT flags i am not sure to what level i should test. For heuristics something similar to https://github.com/gedoensmax/onnxruntime/blob/44dda08b51e98bd86f6eafd09b7d00c3680c78a8/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc#L510-L538 should be possible for, but for all other essentially we would only be testing if there is a crash or not if the option is set. Also if i forgot some option that would be good to have feel free to speak up !
2023-04-14 16:47:36 +00:00
ASSERT_TRUE(s.find(param.c_str()) != std::string::npos);
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
ASSERT_TRUE(api.AllocatorFree(allocator, (void*)trt_options_str) == nullptr);
Ort::SessionOptions session_options;
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_TensorRT_V2(static_cast<OrtSessionOptions*>(session_options), rel_trt_options.get()) == nullptr);
// simple inference test
// prepare inputs
std::vector<Input> inputs(1);
Input& input = inputs.back();
input.name = "X";
input.dims = {3, 2};
input.values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
// prepare expected inputs and outputs
std::vector<int64_t> expected_dims_y = {3, 2};
std::vector<float> expected_values_y = {1.0f, 4.0f, 9.0f, 16.0f, 25.0f, 36.0f};
std::basic_string<ORTCHAR_T> model_uri = MODEL_URI;
// if session creation passes, model loads fine
Ort::Session session(*ort_env, model_uri.c_str(), session_options);
auto default_allocator = std::make_unique<MockedOrtAllocator>();
Rework allocator sharing to work for multiple devices. (#11700) * Rework allocator sharing to work for multiple devices. * Update SessionState to not use allocator name in matching for consistency with IExecutionProvider. The name doesn't have any clear meaning (e.g. we use the same name for the per-thread allocator in the CUDA EP as the shared allocate there and in the TRT EP). * NOTE: this means we will have one allocator per OrtMemType+OrtDevice. * Reverse order when doing allocator setup in SessionState. This will result in the CPU and CUDA EPs allocators being preferred (they are the most configurable), and also means the per-thread CUDA allocator for default GPU memory will be used even when TRT is enabled. * NOTE: Combined with the change to remove the allocator name from the key this will mean that if CUDA and TRT or ROCM and MIGraphX are both enabled the CUDA/ROCM per-thread allocator will be used to allocate GPU memory. * Use InsertAllocator instead of TryInsertAllocator. Each EP should be registered once, and we should only enter RegisterAllocator once, so the 'try' should not be required and would indicate an unexpected setup was involved. i.e. better to fail and figure out if we need to support that setup. * Add some clarifying comments around how replace allocator works. * Add unit testing for setup where EP has local allocator that may get out of sync with values in the IExecutionProvider base class. * Fix invalid check of whether data is on CPU to use device info instead of allocator name.
2022-06-09 07:38:38 +00:00
// without preallocated output tensor
Enable TRT provider option configuration for C# (updated version) (#7808) * prepare for C# to configure provider options * add c# code * revert modification * Add update provider info configuration in trt ep side * fix bugs * fix bug for compiler error C2259 * Add c# test * fix bug * fix bug * Properly deal with string * Add c# api for accepting trt provider options * fix bug * Modify C# test * add shared lib test * Add get provider options functionality * clean up * clean up * fix bug * fix bugs for CI * Fix bugs for CI and documentation * Move TRT EP provider options related functions out of C API * revert * fix bug * refactor * add check for provider options string * code refactor * fix CI bug * Fix CI bugs * clean up * fix bug * Fix bug for Post Analysis * fix accidental bug * Add API_IMPL_BEGIN/API_IMPL_END * clean up * code refactor * code refactor * fix CI fail * fix bug * use string append * Change the code to better handle strncpy and string append
2021-06-25 10:21:22 +00:00
RunSession(default_allocator.get(),
session,
inputs,
"Y",
expected_dims_y,
expected_values_y,
nullptr);
struct stat buffer;
ASSERT_TRUE(stat(engine_cache_path, &buffer) == 0);
}
Exposing new TRT build options (#15089) ### Description This will add a few TRT options, some of them are only available on TRT 8.6: - heuristics - sparsity - optimization level (8.6 only) - auxiliary stream (8.6 only) - tactic source selection I am no sure yet which tests is should add for these options. As those are mostly simple TRT flags i am not sure to what level i should test. For heuristics something similar to https://github.com/gedoensmax/onnxruntime/blob/44dda08b51e98bd86f6eafd09b7d00c3680c78a8/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc#L510-L538 should be possible for, but for all other essentially we would only be testing if there is a crash or not if the option is set. Also if i forgot some option that would be good to have feel free to speak up !
2023-04-14 16:47:36 +00:00
/*
* The TensorrtExecutionProviderOptionsTest can be used to test TRT options
*/
INSTANTIATE_TEST_SUITE_P(CApiTensorRTTest, CApiTensorRTTest,
::testing::Values("trt_build_heuristics_enable=1", "trt_sparsity_enable=1", "trt_builder_optimization_level=0", "trt_tactic_sources=-CUDNN,+CUBLAS", "trt_auxiliary_streams=2"));
Replace functions with secured version for OSX compliance (#7586) * replace strlen with strnlen * replace vsnprintf with vsnprintf_l * add macro * switch to std numeric::limits * apply uint16 max * fix build err * fix mac build * define MAX_STR_LEN * define MAX_STR_LEN * fix typo * trim empty lines * apply constexpr * fix typo * add namespace * fix build err * rename global constant Co-authored-by: Randy <Randy@randysmac.attlocal.net> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: Randy <Randy@randysmac.local>
2021-07-08 18:02:36 +00:00
#endif
Custom thread creation and join hooks (#9426)
2021-11-13 03:10:31 +00:00
Enable CUDA provider option configuration for C# (#10188)
2022-01-06 19:03:14 +00:00
#ifdef USE_CUDA
Add API for updating CUDA EP provider option user compute stream (#17037) Add a generic `UpdateCUDAProviderOptionsWithValue()` C API to update CUDA EP provider options where its data type is pointer that can't be represented by string. Note: Please see some comments for the similar [PR ](https://github.com/microsoft/onnxruntime/pull/16965)for TRT EP.
2023-08-09 16:24:19 +00:00
// This test uses CreateCUDAProviderOptions/UpdateCUDAProviderOptions/UpdateCUDAProviderOptionsWithValue APIs to configure and create a CUDA Execution Provider instance
Enable CUDA provider option configuration for C# (#10188)
2022-01-06 19:03:14 +00:00
TEST(CApiTest, TestConfigureCUDAProviderOptions) {
const auto& api = Ort::GetApi();
OrtCUDAProviderOptionsV2* cuda_options = nullptr;
ASSERT_TRUE(api.CreateCUDAProviderOptions(&cuda_options) == nullptr);
std::unique_ptr<OrtCUDAProviderOptionsV2, decltype(api.ReleaseCUDAProviderOptions)> rel_cuda_options(cuda_options, api.ReleaseCUDAProviderOptions);
Add API for updating CUDA EP provider option user compute stream (#17037) Add a generic `UpdateCUDAProviderOptionsWithValue()` C API to update CUDA EP provider options where its data type is pointer that can't be represented by string. Note: Please see some comments for the similar [PR ](https://github.com/microsoft/onnxruntime/pull/16965)for TRT EP.
2023-08-09 16:24:19 +00:00
// Only test updating OrtCUDAProviderOptionsV2 instance with user provided compute stream not running the inference
cudaStream_t compute_stream = nullptr;
void* user_compute_stream = nullptr;
cudaStreamCreateWithFlags(&compute_stream, cudaStreamNonBlocking);
ASSERT_TRUE(api.UpdateCUDAProviderOptionsWithValue(rel_cuda_options.get(), "user_compute_stream", compute_stream) == nullptr);
ASSERT_TRUE(api.GetCUDAProviderOptionsByName(rel_cuda_options.get(), "user_compute_stream", &user_compute_stream) == nullptr);
ASSERT_TRUE(user_compute_stream == (void*)compute_stream);
cudaStreamDestroy(compute_stream);
Enable CUDA provider option configuration for C# (#10188)
2022-01-06 19:03:14 +00:00
std::vector<const char*> keys{
Add API for updating CUDA EP provider option user compute stream (#17037) Add a generic `UpdateCUDAProviderOptionsWithValue()` C API to update CUDA EP provider options where its data type is pointer that can't be represented by string. Note: Please see some comments for the similar [PR ](https://github.com/microsoft/onnxruntime/pull/16965)for TRT EP.
2023-08-09 16:24:19 +00:00
"device_id", "has_user_compute_stream", "gpu_mem_limit", "arena_extend_strategy",
[CUDA] Optimize Conv and ConvGrad for Training (#10999) * Optimize Conv and ConvGrad for Training * add provider option to control * fix typo
2022-03-28 23:31:36 +00:00
"cudnn_conv_algo_search", "do_copy_in_default_stream", "cudnn_conv_use_max_workspace", "cudnn_conv1d_pad_to_nc1d"};
Enable CUDA provider option configuration for C# (#10188)
2022-01-06 19:03:14 +00:00
std::vector<const char*> values{
Add API for updating CUDA EP provider option user compute stream (#17037) Add a generic `UpdateCUDAProviderOptionsWithValue()` C API to update CUDA EP provider options where its data type is pointer that can't be represented by string. Note: Please see some comments for the similar [PR ](https://github.com/microsoft/onnxruntime/pull/16965)for TRT EP.
2023-08-09 16:24:19 +00:00
"0", "0", "1024", "kSameAsRequested",
Enable CUDA provider option configuration for C# (#10188)
2022-01-06 19:03:14 +00:00
"DEFAULT", "1", "1"};
ASSERT_TRUE(api.UpdateCUDAProviderOptions(rel_cuda_options.get(), keys.data(), values.data(), 6) == nullptr);
OrtAllocator* allocator;
ASSERT_TRUE(api.GetAllocatorWithDefaultOptions(&allocator) == nullptr);
char* cuda_options_str = nullptr;
ASSERT_TRUE(api.GetCUDAProviderOptionsAsString(rel_cuda_options.get(), allocator, &cuda_options_str) == nullptr);
Fix/Suppress some VC static analyzer warnings (#12713)
2022-08-25 06:39:40 +00:00
std::string s;
if (cuda_options_str != nullptr) {
s = std::string(cuda_options_str, strnlen(cuda_options_str, 2048));
}
Enable CUDA provider option configuration for C# (#10188)
2022-01-06 19:03:14 +00:00
ASSERT_TRUE(s.find("device_id=0") != std::string::npos);
ASSERT_TRUE(s.find("gpu_mem_limit=1024") != std::string::npos);
ASSERT_TRUE(s.find("arena_extend_strategy=kSameAsRequested") != std::string::npos);
ASSERT_TRUE(s.find("cudnn_conv_algo_search=DEFAULT") != std::string::npos);
ASSERT_TRUE(s.find("do_copy_in_default_stream=1") != std::string::npos);
ASSERT_TRUE(s.find("cudnn_conv_use_max_workspace=1") != std::string::npos);
[CUDA] Optimize Conv and ConvGrad for Training (#10999) * Optimize Conv and ConvGrad for Training * add provider option to control * fix typo
2022-03-28 23:31:36 +00:00
ASSERT_TRUE(s.find("cudnn_conv1d_pad_to_nc1d") != std::string::npos);
Enable CUDA provider option configuration for C# (#10188)
2022-01-06 19:03:14 +00:00
ASSERT_TRUE(api.AllocatorFree(allocator, (void*)cuda_options_str) == nullptr);
Ort::SessionOptions session_options;
ASSERT_TRUE(api.SessionOptionsAppendExecutionProvider_CUDA_V2(static_cast<OrtSessionOptions*>(session_options), rel_cuda_options.get()) == nullptr);
// if session creation passes, model loads fine
std::basic_string<ORTCHAR_T> model_uri = MODEL_URI;
Ort::Session session(*ort_env, model_uri.c_str(), session_options);
}
#endif
Custom thread creation and join hooks (#9426)
2021-11-13 03:10:31 +00:00
namespace TestPerSessionCustomThreadHooks {
std::vector<std::thread> threads;
int32_t custom_thread_creation_options = 5;
int32_t custom_creation_hook_called = 0;
int32_t custom_join_hook_called = 0;
OrtCustomThreadHandle CreateThreadCustomized(void* options, OrtThreadWorkerFn work_loop, void* param) {
if (*((int32_t*)options) == 5) {
custom_creation_hook_called += 1;
}
threads.push_back(std::thread(work_loop, param));
return reinterpret_cast<OrtCustomThreadHandle>(threads.back().native_handle());
}
void JoinThreadCustomized(OrtCustomThreadHandle handle) {
for (auto& t : threads) {
if (reinterpret_cast<OrtCustomThreadHandle>(t.native_handle()) == handle) {
custom_join_hook_called += 1;
t.join();
}
}
}
TEST(CApiTest, TestPerSessionCustomThreadPoolHooks) {
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
constexpr int32_t thread_count = 3;
Custom thread creation and join hooks (#9426)
2021-11-13 03:10:31 +00:00
Ort::SessionOptions session_options;
// test both intra and inter op thread pool
session_options.SetExecutionMode(ExecutionMode::ORT_PARALLEL);
session_options.SetIntraOpNumThreads(thread_count);
session_options.SetInterOpNumThreads(thread_count);
session_options.SetCustomCreateThreadFn(CreateThreadCustomized);
session_options.SetCustomThreadCreationOptions(&custom_thread_creation_options);
session_options.SetCustomJoinThreadFn(JoinThreadCustomized);
{
Ort::Session session(*ort_env, MODEL_URI, session_options);
}
ASSERT_TRUE(custom_creation_hook_called == (thread_count - 1) << 1);
ASSERT_TRUE(custom_join_hook_called == (thread_count - 1) << 1);
}
Fix typos in sources: operater, tranform, neccessary, trainig (#14907) ### Description While browsing the sources I found several typos here and there. I collected them to a single PR and fixed them. Namely these typos are: operater, tranform, neccessary, trainig. After fixing none of them was found anymore: $ git grep "operater" $ git grep "tranform" $ git grep "neccessary" $ git grep "trainig" $ ### Motivation and Context Since some of the typos are in example notebooks and markdown files, users can see them.
2023-03-14 05:45:04 +00:00
// Preventing resize transformer issue:
Cancel transpose optimizer for resize (#9870) * cancel transpose optimizer for resize * add UT * addressing comments * fix build err
2021-11-30 06:30:16 +00:00
// https://github.com/microsoft/onnxruntime/issues/9857
exclude test case from reduced-op build (#9895) Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2021-12-01 23:50:36 +00:00
#ifndef REDUCED_OPS_BUILD
Cancel transpose optimizer for resize (#9870) * cancel transpose optimizer for resize * add UT * addressing comments * fix build err
2021-11-30 06:30:16 +00:00
TEST(CApiTest, crop_and_resize) {
std::vector<float> input_value_0;
input_value_0.resize(2 * 36 * 36 * 3);
Change some const to constexpr in unit tests code(#10002)
2021-12-10 20:26:59 +00:00
for (ptrdiff_t i = 0; i < 36 * 36 * 3; ++i) {
Cancel transpose optimizer for resize (#9870) * cancel transpose optimizer for resize * add UT * addressing comments * fix build err
2021-11-30 06:30:16 +00:00
input_value_0[i] = 1.f;
input_value_0[i + 36 * 36 * 3] = 2.f;
}
std::vector<int64_t> input_shape_0{2, 36, 36, 3};
std::vector<int32_t> input_value_1{1, 0};
std::vector<int64_t> input_shape_1{2};
std::vector<const char*> input_names{"input:0", "input2:0"};
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
ort_inputs.emplace_back(Ort::Value::CreateTensor<float>(info, input_value_0.data(), input_value_0.size(), input_shape_0.data(), input_shape_0.size()));
ort_inputs.emplace_back(Ort::Value::CreateTensor<int32_t>(info, input_value_1.data(), input_value_1.size(), input_shape_1.data(), input_shape_1.size()));
Ort::SessionOptions session_options;
Ort::Session session(*ort_env, RESIZE_AND_CROP_MODEL_URI, session_options);
const char* output_names[] = {"output:0"};
std::vector<int64_t> output_shape{2, 20, 20, 3};
std::vector<Ort::Value> ort_outputs = session.Run(Ort::RunOptions{}, input_names.data(), ort_inputs.data(), ort_inputs.size(), output_names, countof(output_names));
ASSERT_EQ(ort_outputs.size(), 1U);
const auto& output_0 = ort_outputs[0];
ASSERT_TRUE(output_0.IsTensor());
auto output_type_shape = output_0.GetTensorTypeAndShapeInfo();
ASSERT_EQ(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, output_type_shape.GetElementType());
ASSERT_EQ(output_shape, output_type_shape.GetShape());
}
exclude test case from reduced-op build (#9895) Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2021-12-01 23:50:36 +00:00
#endif
Cancel transpose optimizer for resize (#9870) * cancel transpose optimizer for resize * add UT * addressing comments * fix build err
2021-11-30 06:30:16 +00:00
Custom thread creation and join hooks (#9426)
2021-11-13 03:10:31 +00:00
} // namespace TestPerSessionCustomThreadHooks
Fix race condition in CUDA, ROCm, and TensorRT EP GetKernelRegistry() implementations. (#10200) Make GetKernelRegistry() kernel registry initialization thread-safe.
2022-03-02 01:53:58 +00:00
#ifdef USE_CUDA
TEST(CApiTest, GitHubIssue10179) {
// https://github.com/microsoft/onnxruntime/issues/10179
// the issue was caused by a race condition in CUDAExecutionProvider::GetKernelRegistry()
// if the test runs to completion, consider that run successful
auto load_model_thread_fn = []() {
try {
const auto* model_path = MODEL_URI;
Ort::SessionOptions session_options{};
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, 0));
Ort::Session session{*ort_env, model_path, session_options};
} catch (const std::exception& e) {
std::cerr << "exception: " << e.what() << "\n";
throw e;
}
};
constexpr int num_threads = 4;
constexpr int num_iterations = 10;
for (int i = 0; i < num_iterations; ++i) {
std::vector<std::thread> threads(num_threads);
for (auto& thread : threads) {
thread = std::thread{load_model_thread_fn};
}
for (auto& thread : threads) {
thread.join();
}
}
}
Fix bug in MemcpyToHost (#10816)
2022-03-10 15:02:27 +00:00
Fix Reduced ops pipeline (#10861)
2022-03-15 03:42:14 +00:00
#endif
// Reduced Ops build doesn't support If (16) yet
#if !defined(REDUCED_OPS_BUILD) && defined(USE_CUDA)
Fix bug in MemcpyToHost (#10816)
2022-03-10 15:02:27 +00:00
TEST(CApiTest, TestCudaMemcpyToHostWithSequenceTensors) {
const auto* model_path = SEQUENCE_MODEL_URI_2;
Ort::SessionOptions session_options{};
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, 0));
Ort::Session session{*ort_env, model_path, session_options};
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
std::vector<const char*> input_names{"cond"};
bool input_data[] = {false};
std::vector<int64_t> input_dims{};
ort_inputs.emplace_back(Ort::Value::CreateTensor<bool>(info, input_data, 1U, input_dims.data(), 0));
const char* output_names[] = {"sequence"};
std::vector<Ort::Value> ort_outputs = session.Run(Ort::RunOptions{nullptr}, input_names.data(),
ort_inputs.data(), ort_inputs.size(),
output_names, countof(output_names));
// There is no need to check the contents of the output, we are just checking to see if the
// model runs without crashing
}
Fix race condition in CUDA, ROCm, and TensorRT EP GetKernelRegistry() implementations. (#10200) Make GetKernelRegistry() kernel registry initialization thread-safe.
2022-03-02 01:53:58 +00:00
#endif
Skip Constant Folding for ops producing an optional type output (#11839)
2022-06-30 20:38:35 +00:00
Fix Reduced Ops build pipeline (#12144) Fix ReducedOps build pipeline
2022-07-12 02:02:38 +00:00
// Reduced Ops build doesn't support OptionalHasElement (16) yet
#if !defined(REDUCED_OPS_BUILD) && !defined(DISABLE_OPTIONAL_TYPE)
Skip Constant Folding for ops producing an optional type output (#11839)
2022-06-30 20:38:35 +00:00
TEST(CApiTest, GH_11717) {
Fix unused variable warning from reduced ops build (#12889)
2022-09-08 22:08:56 +00:00
const auto* model_path = TSTR("testdata/gh_issue_11717.onnx");
Skip Constant Folding for ops producing an optional type output (#11839)
2022-06-30 20:38:35 +00:00
Ort::SessionOptions session_options{};
// Just check if the model loads fine without a segmentation fault
// in the default CPU EP
EXPECT_NO_THROW(Ort::Session session(*ort_env, model_path, session_options));
}
#endif
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
Exclude a multi-stream case from reduced ops build (#14351) Exclude a multi-stream case from reduced ops build to unblock [pipeline](https://dev.azure.com/onnxruntime/onnxruntime/_build?definitionId=120&_a=summary). Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-01-19 22:39:25 +00:00
#ifndef REDUCED_OPS_BUILD
Multi-stream execution support (#13495) **Description**: This PR including following works: 1. provide stream and related synchronization abstractions in onnxruntime. 2. enhance onnxruntime's execution planner / executor / memory arena to support execute multiple streams in parallel. 3. deprecate the parallel executor for cpu. 4. deprecate the Fence mechanism. 5. update the cuda / tensorrt EP to support the stream mechanism, support running different request in different cuda stream. **Motivation and Context** - Why is this change required? currently, the execution plan is just a linear list of those primitives, ort will execute them step by step. For any given graph, ORT will serialize it to a fixed execution order. This sequential execution design simplifies most scenarios, but it has the following limitations: 1. it is difficult to enable inter-node parallelization, we have a half-baked parallel executor but it is very difficult to make it work with GPU. 2. The fence mechanism can work with single gpu stream + cpu thread case, but when extend to multiple stream, it is difficult to manage the cross GPU stream synchronizations. 3. our cuda EP rely on the BFCArena to make the memory management work with the GPU async kernels, but current BFCArena is not aware of the streams, so it doesn't behavior correctly when run with multiple streams. This PR enhance our existing execution plan and executor to support multiple stream execution. we use an unified algorithm to mange both single stream and multiple stream scenarios. This PR mainly focus on the infrastructure support for multiple stream execution, that is said, given a valid stream assignment, onnxruntime can execute it correctly. How to generate a good stream assignment for a given model will be in the future PR. Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: cao lei <jslhcl@gmail.com> Co-authored-by: Lei Cao <leca@microsoft.com>
2022-12-15 15:39:29 +00:00
TEST(CApiTest, TestMultiStreamInferenceSimpleSSD) {
Ort::SessionOptions session_options{};
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_DISABLE_ALL);
session_options.AddConfigEntry("session.node_partition_config_file",
"./testdata/multi_stream_models/simplified_ssd_cpu.csv");
Ort::Session session{*ort_env, SIMPLIFIED_SSD_MODEL_URI, session_options};
Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);
std::vector<Ort::Value> ort_inputs;
const char* input_names[] = {"graph_in"};
std::unique_ptr<float[]> input_data = std::make_unique<float[]>(3 * 3 * 300 * 300);
for (int i = 0; i < 3 * 3 * 300 * 300; ++i) {
input_data[i] = 1.f;
}
int64_t input_dims[] = {3, 3, 300, 300};
ort_inputs.emplace_back(Ort::Value::CreateTensor<float>(info, input_data.get(), 3 * 3 * 300 * 300, input_dims, 4U));
const char* output_names[] = {"graph_out"};
std::vector<Ort::Value> ort_outputs = session.Run(Ort::RunOptions{nullptr}, input_names,
ort_inputs.data(), ort_inputs.size(),
output_names, countof(output_names));
ASSERT_TRUE(ort_outputs.size() == 1);
ASSERT_TRUE(ort_outputs[0].IsTensor());
const auto& type_shape_info = ort_outputs[0].GetTensorTypeAndShapeInfo();
std::vector<int64_t> output_dims = type_shape_info.GetShape();
std::vector<int64_t> expected_output_dims = {3, 256, 150, 150};
ASSERT_TRUE(output_dims == expected_output_dims);
Add ability to register custom ops by specifying a function name (#14177) ### Description <!-- Describe your changes. --> Use dlsym/GetProcAddress to lookup a custom ops registration function by name and call it. This will be better on mobile platforms where the custom ops library is linked against, and there isn't necessarily a filesystem that a library path can be loaded from. Alternative is to wire up passing in the address of the function, but that has multiple complications which differ by platform. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Enable using ort and ort-ext packages on mobile platforms. Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
2023-01-12 05:11:34 +00:00
}
Fix issue with schema lookup where there are custom ops using the ONNX domain (#14492) ### Description <!-- Describe your changes. --> Fix issue with schema lookup where there are custom ops using the ONNX domain. Update testing infrastructure to use an explicit domain for custom ops. Using an empty string clashes with the ONNX domain and can cause unexpected issues. It's also a bad example for external users as our docs point to the unit tests. Fix a couple of places using exact matching of the node since version to be slightly more flexible and use a range (which aligns with how the kernel lookup works). ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes a problem that came up when adding support for standalone custom ops in an ORT format model. Separating these changes out to simplify review.
2023-02-02 22:05:18 +00:00
#endif
Single-schema-multi-kernel (#15184) The PR is to allow custom op of different input types to have same op name in a graph. The idea to go over all ops of same name and merge their input/output types into a type-inference function. With the enhancement, custom op node inside a graph can have same op-type given that the input/output types are different. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-04-27 20:39:59 +00:00
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
#if !defined(REDUCED_OPS_BUILD) && !defined(DISABLE_OPTIONAL_TYPE)
TEST(LiteCustomOpTest, CustomFunc) {
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
#if defined(_WIN32)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("custom_op_library.dll"));
#elif defined(__APPLE__)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("libcustom_op_library.dylib"));
#else
session_options.RegisterCustomOpsLibrary(ORT_TSTR("./libcustom_op_library.so"));
#endif
Ort::Session session{*ort_env, TSTR("testdata/fuse_select_filter.onnx"), session_options};
const char* input_names[] = {"vector_1", "vector_2", "alpha", "indices"};
const char* output_names[] = {"vector_filtered"};
float vector_1_value[] = {0.f, 1.f, 2.f, 3.f, 4.f, 5.f, 6.f, 7.f, 8.f, 9.f};
int64_t vector_1_dim[] = {10};
float vector_2_value[] = {0.f, 1.f, 2.f, 3.f, 4.f, 5.f};
int64_t vector_2_dim[] = {6};
int32_t alpha_value[] = {2};
int64_t alpha_dim[] = {1};
int32_t indices_value[] = {0, 1, 2, 3, 4, 5};
int64_t indices_dim[] = {6};
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensors[] = {
Ort::Value::CreateTensor<float>(memory_info, vector_1_value, 10, vector_1_dim, 1),
Ort::Value::CreateTensor<float>(memory_info, vector_2_value, 6, vector_2_dim, 1),
Ort::Value::CreateTensor<int32_t>(memory_info, alpha_value, 1, alpha_dim, 1),
Ort::Value::CreateTensor<int32_t>(memory_info, indices_value, 6, indices_dim, 1)};
Ort::RunOptions run_options;
auto output_tensors = session.Run(run_options, input_names, input_tensors, 4, output_names, 1);
const auto& vector_filterred = output_tensors.at(0);
auto type_shape_info = vector_filterred.GetTensorTypeAndShapeInfo();
const float* floats_output = static_cast<const float*>(vector_filterred.GetTensorRawData());
ASSERT_TRUE(floats_output[0] == 8);
ASSERT_TRUE(floats_output[1] == 16);
}
Versioning for custom op (#18088) Allow custom ops to have versions. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-10-31 23:50:27 +00:00
TEST(LiteCustomOpTest, CustomFuncOpsetMismatch) {
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
#if defined(_WIN32)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("custom_op_library.dll"));
#elif defined(__APPLE__)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("libcustom_op_library.dylib"));
#else
session_options.RegisterCustomOpsLibrary(ORT_TSTR("./libcustom_op_library.so"));
#endif
EXPECT_THROW(Ort::Session(*ort_env, TSTR("testdata/fuse_select_filter_opset_8.onnx"), session_options), std::exception);
}
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
struct Merge {
Merge(const OrtApi* ort_api, const OrtKernelInfo* info) {
int64_t reverse;
ORT_ENFORCE(ort_api->KernelInfoGetAttribute_int64(info, "reverse", &reverse) == nullptr);
reverse_ = reverse != 0;
}
Patch lite custom op API (#17605) A few enhancements: - Support compute returning status; - Support variadic; --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-09-26 21:02:18 +00:00
Ort::Status Compute(const Ort::Custom::Tensor<std::string_view>& strings_in,
std::string_view string_in,
Ort::Custom::Tensor<std::string>* strings_out) {
if (strings_in.NumberOfElement() == 0) {
return Ort::Status("the 1st input must have more than one string!", OrtErrorCode::ORT_INVALID_ARGUMENT);
}
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
std::vector<std::string> string_pool;
for (const auto& s : strings_in.Data()) {
string_pool.emplace_back(s.data(), s.size());
}
string_pool.emplace_back(string_in.data(), string_in.size());
if (reverse_) {
for (auto& str : string_pool) {
std::reverse(str.begin(), str.end());
}
std::reverse(string_pool.begin(), string_pool.end());
}
strings_out->SetStringOutput(string_pool, {static_cast<int64_t>(string_pool.size())});
Patch lite custom op API (#17605) A few enhancements: - Support compute returning status; - Support variadic; --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-09-26 21:02:18 +00:00
return Ort::Status(nullptr);
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
}
Custom op shape inference API (#17737) Add c/cxx API to allow custom ops do shape inference. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-10-13 19:57:42 +00:00
static Ort::Status InferOutputShape(Ort::ShapeInferContext& ctx) {
auto input_count = ctx.GetInputCount();
if (input_count != 2) {
return Ort::Status("input count should be 2", OrtErrorCode::ORT_INVALID_ARGUMENT);
}
Ort::ShapeInferContext::Shape shape_1 = {{-1}};
ctx.SetOutputShape(0, shape_1);
return Ort::Status(nullptr);
}
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
bool reverse_ = false;
};
TEST(LiteCustomOpTest, CustomStruct) {
const auto& ortApi = Ort::GetApi();
Ort::CustomOpDomain v2_domain{"v2"};
std::unique_ptr<Ort::Custom::OrtLiteCustomOp> mrg_op_ptr{Ort::Custom::CreateLiteCustomOp<Merge>("Merge", "CPUExecutionProvider")};
v2_domain.Add(mrg_op_ptr.get());
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.Add(v2_domain);
session_options.SetLogSeverityLevel(0);
Ort::Session session{*ort_env, TSTR("testdata/merge.onnx"), session_options};
const char* input_names[] = {"str_in_1", "str_in_2"};
const char* output_names[] = {"str_out"};
OrtAllocator* allocator = nullptr;
ASSERT_TRUE(!ortApi.GetAllocatorWithDefaultOptions(&allocator));
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
int64_t str_1_dims[] = {2};
int64_t str_2_dims[] = {1};
Ort::Value input_tensors[] = {Ort::Value::CreateTensor(allocator, str_1_dims, 1, ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING),
Ort::Value::CreateTensor(allocator, str_2_dims, 1, ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING)};
const char* str_1_raw[] = {"abc", "de"};
const char* str_2_raw[] = {"fg"};
input_tensors[0].FillStringTensor(str_1_raw, 2);
input_tensors[1].FillStringTensor(str_2_raw, 1);
Ort::RunOptions run_options;
auto output_tensors = session.Run(run_options, input_names, input_tensors, 2, output_names, 1);
const auto& str_out_tensor = output_tensors.at(0);
auto num_chars = str_out_tensor.GetStringTensorDataLength();
std::vector<char> chars(num_chars + 1, '\0');
std::vector<size_t> offsets(3);
str_out_tensor.GetStringTensorContent(static_cast<void*>(chars.data()), num_chars, offsets.data(), offsets.size());
ASSERT_TRUE(strncmp(chars.data(), "gfedcba", 7) == 0);
}
TEST(LiteCustomOpTest, MissingOptional) {
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
#if defined(_WIN32)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("custom_op_library.dll"));
#elif defined(__APPLE__)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("libcustom_op_library.dylib"));
#else
session_options.RegisterCustomOpsLibrary(ORT_TSTR("./libcustom_op_library.so"));
#endif
Ort::Session session(*ort_env, TSTR("testdata/optional_2.onnx"), session_options);
const char* input_names[] = {"float_in_1", "float_in_2"};
const char* output_names[] = {"float_out_1"};
float vector_1_value[] = {0.f, 1.f, 2.f};
int64_t vector_1_dim[] = {3};
float vector_2_value[] = {4.f, 5.f, 6.f, 7.f};
int64_t vector_2_dim[] = {4};
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensors[] = {
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
Ort::Value::CreateTensor<float>(memory_info, vector_1_value, gsl::narrow_cast<size_t>(vector_1_dim[0]),
vector_1_dim, 1),
Ort::Value::CreateTensor<float>(memory_info, vector_2_value, gsl::narrow_cast<size_t>(vector_2_dim[0]),
vector_2_dim, 1)};
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
Ort::RunOptions run_options;
auto output_tensors = session.Run(run_options, input_names, input_tensors, 2, output_names, 1);
ASSERT_TRUE(output_tensors.size() == 1);
}
TEST(LiteCustomOpTest, HasOptional) {
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
#if defined(_WIN32)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("custom_op_library.dll"));
#elif defined(__APPLE__)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("libcustom_op_library.dylib"));
#else
session_options.RegisterCustomOpsLibrary(ORT_TSTR("./libcustom_op_library.so"));
#endif
Ort::Session session(*ort_env, TSTR("testdata/optional_3.onnx"), session_options);
const char* input_names[] = {"float_in_1", "float_in_2", "float_in_3"};
const char* output_names[] = {"float_out_1", "float_out_2"};
float vector_1_value[] = {0.f, 1.f, 2.f};
int64_t vector_1_dim[] = {3};
float vector_2_value[] = {4.f, 5.f, 6.f, 7.f};
int64_t vector_2_dim[] = {4};
float vector_3_value[] = {8.f, 9.f};
int64_t vector_3_dim[] = {2};
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensors[] = {
Fix builds (#16646) - Fix some more `shorten-64-to-32` warnings - Move minimum build.py Python version back to 3.6
2023-07-12 02:21:25 +00:00
Ort::Value::CreateTensor<float>(memory_info, vector_1_value, gsl::narrow_cast<size_t>(vector_1_dim[0]),
vector_1_dim, 1),
Ort::Value::CreateTensor<float>(memory_info, vector_2_value, gsl::narrow_cast<size_t>(vector_2_dim[0]),
vector_2_dim, 1),
Ort::Value::CreateTensor<float>(memory_info, vector_3_value, gsl::narrow_cast<size_t>(vector_3_dim[0]),
vector_3_dim, 1),
Implement lite custom op API (#15778) Implement a set of new APIs for lightweight custom ops registration, to save efforts from schema-composing. A few highlights: - Support build-time type inference; - Support function-as-op for "stateless" ops; - Support structure-as-op for "stateful" ops; - Support varied input/output forms such as span, scalar, and tensors, either optional or non-optional. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-05-04 16:49:17 +00:00
};
Ort::RunOptions run_options;
auto output_tensors = session.Run(run_options, input_names, input_tensors, 3, output_names, 2);
ASSERT_TRUE(output_tensors.size() == 2);
}
Single-schema-multi-kernel (#15184) The PR is to allow custom op of different input types to have same op name in a graph. The idea to go over all ops of same name and merge their input/output types into a type-inference function. With the enhancement, custom op node inside a graph can have same op-type given that the input/output types are different. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-04-27 20:39:59 +00:00
TEST(MultiKernelSingleSchemaTest, valid) {
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
#if defined(_WIN32)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("custom_op_library.dll"));
#elif defined(__APPLE__)
session_options.RegisterCustomOpsLibrary(ORT_TSTR("libcustom_op_library.dylib"));
#else
session_options.RegisterCustomOpsLibrary(ORT_TSTR("./libcustom_op_library.so"));
#endif
Ort::Session session(*ort_env, CUSTOM_OP_SINGLE_SCHEMA_MULTI_KERNEL, session_options);
const char* input_names[] = {"X"};
const char* output_names[] = {"Y", "Z"};
float x_value[] = {0.f, 1.f, 2.f, 3.f, 4.f, 5.f, 6.f, 7.f, 8.f, 9.f};
int64_t x_dim[] = {10};
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensors[1] = {
Ort::Value::CreateTensor<float>(memory_info, x_value, 10, x_dim, 1),
};
RunAsync C/CXX API (#16613) Implement RunAsync API - the session will run in a thread of intra-op thread pool. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-07-16 23:51:40 +00:00
Ort::RunOptions run_options;
auto output_tensors = session.Run(run_options, input_names, input_tensors, 1, output_names, 2);
Single-schema-multi-kernel (#15184) The PR is to allow custom op of different input types to have same op name in a graph. The idea to go over all ops of same name and merge their input/output types into a type-inference function. With the enhancement, custom op node inside a graph can have same op-type given that the input/output types are different. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-04-27 20:39:59 +00:00
ASSERT_TRUE(*output_tensors[1].GetTensorData<int32_t>() == 72);
}
// expect input count mismatch exception
TEST(MultiKernelSingleSchemaTest, InputCountMismatch) {
Ort::CustomOpDomain v2_domain("v2");
MulTopOpFloat mul_top_f32;
MulTopOpDouble mul_top_double;
v2_domain.Add(&mul_top_f32);
v2_domain.Add(&mul_top_double);
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
session_options.Add(v2_domain);
EXPECT_THROW(Ort::Session session(*ort_env, CUSTOM_OP_SINGLE_SCHEMA_MULTI_KERNEL, session_options), std::exception);
}
// expect output count mismatch exception
TEST(MultiKernelSingleSchemaTest, OutputMismatch) {
Ort::CustomOpDomain v2_domain("v2");
MulTopOpFloat mul_top_f32;
MulTopOpInt16 mul_top_int64;
v2_domain.Add(&mul_top_f32);
v2_domain.Add(&mul_top_int64);
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
session_options.Add(v2_domain);
EXPECT_THROW(Ort::Session session(*ort_env, CUSTOM_OP_SINGLE_SCHEMA_MULTI_KERNEL, session_options), std::exception);
}
// expect characteristic mismatch exception
TEST(MultiKernelSingleSchemaTest, CharacterMismatch) {
Ort::CustomOpDomain v2_domain("v2");
MulTopOpFloat mul_top_f32;
MulTopOpFloat16 mul_top_f16;
v2_domain.Add(&mul_top_f32);
v2_domain.Add(&mul_top_f16);
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
session_options.Add(v2_domain);
EXPECT_THROW(Ort::Session session(*ort_env, CUSTOM_OP_SINGLE_SCHEMA_MULTI_KERNEL, session_options), std::exception);
}
TEST(MultiKernelSingleSchemaTest, DuplicateKernel) {
Ort::CustomOpDomain v2_domain("v2");
MulTopOpFloat mul_top_f32_1;
MulTopOpFloat mul_top_f32_2;
MulTopOpInt32 mul_top_i32;
v2_domain.Add(&mul_top_f32_1);
v2_domain.Add(&mul_top_f32_2);
v2_domain.Add(&mul_top_i32);
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
session_options.SetLogSeverityLevel(0);
session_options.Add(v2_domain);
EXPECT_NO_THROW(Ort::Session session(*ort_env, CUSTOM_OP_SINGLE_SCHEMA_MULTI_KERNEL, session_options));
}
#endif
RunAsync C/CXX API (#16613) Implement RunAsync API - the session will run in a thread of intra-op thread pool. --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-07-16 23:51:40 +00:00
static std::thread::id caller_tid = std::this_thread::get_id();
static std::atomic_bool atomic_wait{false};
void CallbackSucceed(void* user_data, OrtValue** outputs, size_t num_outputs, OrtStatusPtr status_ptr) {
auto callee_tid = std::this_thread::get_id();
EXPECT_NE(*(reinterpret_cast<std::thread::id*>(user_data)), callee_tid);
Ort::Status status(status_ptr);
EXPECT_TRUE(status.IsOK());
EXPECT_EQ(num_outputs, 1UL);
Ort::Value output_value(outputs[0]);
EXPECT_EQ(output_value.At<float>({1, 0}), 9.f);
output_value.release();
atomic_wait.store(true);
}
TEST(CApiTest, RunAsync) {
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(2);
Ort::Session session(*ort_env, MODEL_URI, session_options);
const char* input_names[] = {"X"};
float x_value[] = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
int64_t x_dim[] = {3, 2};
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensors[1] = {
Ort::Value::CreateTensor<float>(memory_info, x_value, 6, x_dim, 2),
};
const char* output_names[] = {"Y"};
Ort::RunOptions run_options;
Ort::Value output_values[1] = {Ort::Value{nullptr}};
EXPECT_NO_THROW(session.RunAsync(run_options,
input_names,
input_tensors,
1,
output_names,
output_values,
1,
CallbackSucceed,
&caller_tid));
std::chrono::duration<double, std::milli> dur{100};
// timeout in about 10 secs
for (int i = 0; i < 100 && !atomic_wait.load(); ++i) {
std::this_thread::sleep_for(dur);
}
EXPECT_EQ(atomic_wait.load(), true);
}
void CallbackFail(void*, OrtValue**, size_t, OrtStatusPtr) {
EXPECT_TRUE(false); // the callback is not supposed to be invoked
}
TEST(CApiTest, RunAsyncFail) {
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1); // This will cause RunAsync fail
Ort::Session session(*ort_env, MODEL_URI, session_options);
const char* input_names[] = {"X"};
float x_value[] = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};
int64_t x_dim[] = {3, 2};
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensors[1] = {
Ort::Value::CreateTensor<float>(memory_info, x_value, 6, x_dim, 2),
};
Ort::Value output_values[1] = {Ort::Value{nullptr}};
const char* output_names[] = {"Y"};
Ort::RunOptions run_options;
EXPECT_THROW(session.RunAsync(run_options, input_names, input_tensors, 1, output_names, output_values, 1, CallbackFail, nullptr), std::exception);
Add API for updating TRT EP provider option user compute stream (#16965) Add a generic `UpdateTensorRTProviderOptionsWithValue()` C API to update TensorRT provider options where its data type is pointer that can't be represented by string.
2023-08-04 22:14:43 +00:00
}
Reference in a new issue Copy permalink