This PR updates the ThreadPool API to support multi-loop parallel sections. As with the OpenMP "parallel" construct, this allows per-loop work to be amortized over a series of loops. For ORT, it also promotes locality between successive loops in the sense that iteration X of one loop will tend to run on the same worker thread as iteration X of preceding loops.
The change was developed while optimizing the implementation of a model that performed better with OpenMP. Profiling indicated that OpenMP was providing lower loop entry/exit costs and that, via OpenMP's static scheduling, it was leading to a lower L2 miss rate in the series of parallel loops used in GRU.
The main changes are:
- Addition of ThreadPool::ParallelSection and underlying support in the modified Eigen thread pool.
- In EigenNonBlockingThreadPool.h, refactoring the RunInParallel method to support two variants: one that takes an existing parallel section object created by the caller, and another (used by default) that creates its own parallel section.
- Simplify ThreadPool::LoopCounter (used by worker threads to claim loop iterations), basing it an ID supplied by the underlying Eigen thread pool for affinity in a series of loops.
- Fix a possible perf issue where a loop with iterations scheduled in batches would have more threads than batches available.
- Use of parallel sections in the GRU operator.
- Additional test cases in threadpool_test.h.
- Additional comments at the top of threadpool.h and EigenNonBlockingThreadPool.h.
Add tag types for Ort::Float16_t and Ort:Bfloat16_t structs
that contain uint16_t values for float16 and bfloat16.
These will serve as type dispatching types for C++ API.
They are of uint16_t size and arrays of these types can be used
to create Tensors of the corresponding types.
Make documentation Doxygen compliant.
* Add options for nnapi ep
* Add nnapi flags test
* add comments
* Add flag comments
* Make the flags bitset const
* Fix build break
* Add stub changes to java and c# api
* Fix java related build break
* Fix java build break
* Switch to bit flags instead of bitset
The ROCm EP is designed and implemented based on AMD GPU software stack named ROCm. Here is the link for the details about ROCm: https://rocmdocs.amd.com/en/latest/
ROCm EP was created based on the following things:
1. AMD GPU programming language: HIP
2. AMD GPU HIP language runtime: amdhip64
3. BLAS: rocBLAS, hipBLAS
4. DNN: miOpen
5. Collective Communication library: RCCL
6. cub: hipCub
7. …
Current status:
BERT-L and GPT2 training can be ran on AMD GPU with data parallel.
Next:
1. Make more GPU code be sharable between ROCm EP and CUDA EP since HIP language and HIP runtime API are very close to CUDA.
2. Continue improving the implementation.
3. Continue GPU kernel optimization.
4. Support model parallelism on ROCm EP.
……
The rocm kernels have been removed from this commit and will be in a separate PR. Since the original PR was too big(~180 files), it was suggested to split the PR into two parts, one is rocm-kernels, the other is non rocm kernels.
Co-authored-by: Weixing Zhang <wezhan@microsoft.com>
Co-authored-by: sabreshao <sabre.shao@amd.com>
Co-authored-by: anghostcici <11013544+anghostcici@users.noreply.github.com>
Co-authored-by: Suffian Khan <sukha@microsoft.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* Introduce OpKernelInfo GetAttrAsSpan() for floats and ints attribute proto arrays
and GetAttrsStringRefs() to return a vector of string references.
These new APIs allow kernels not copy attribute arrays especially if they are large
and save on memory.
but refer directly to data that is in AttributeProto.
Modify TfIdfVectorizer to take advantage of the new API.
Signed-off-by: Dmitri Smirnov <dmitrism@microsoft.com>
* Add test for FastGelu + GeluRecompute.
* Fix GeluRecompute for 2 inputs case.
* Fix test for BiasGelu + GeluRecompute.
* Copy all inputs to Gelu, not just 2.
* Move GeluRecompute test to training-specific file.
Description: This change makes three changes to the ThreadPool class to clean up issues identified during performance analysis and optimization. (1) It uses mm_pause intrinsics in spin loops, helping avoid consuming pipeline resources while waiting. (2) It re-organizes the spin-then-steal loop for work distribution to start out spinning as intended, rather than to start out trying to steal. (3) It updates the ThreadPool class's API to be consistent in the use of static methods for public functions. The PR includes minor doc updates and corresponding changes to test cases.
Motivation and Context
The change helps ensure consistency in behavior between the OpenMP and Eigen-based implementations. Unlike the instance methods, the static methods abstract over the different ways in which threading can be implemented; they will map onto the OpenMP or Eigen-based implementations when threading is used. When threading is not used they will run work sequentially.
Introduce sparse_initializers support.
Convert them to dense on model load and prune graph_proto_
so they don't consume space. Convert back to sparse on ORT Format model save.
Implement serializing sparse initializers to OrtFormat.
Fix Model::ToProto() to return original sparse initializers
Set a flag that graph_sync is needed when loading a simple ORT Format model.
otherwise nothing is resolved.
Add ORT Format history to README.md
ifdef MINIMAL build for DenseToSparseTensorInitializer
Allow duplicate initializers to support existing models.
Issue a warning instead of aborting.
* Revert "Remove SparseTensor support from minimal build. (#5114)"
This reverts commit 59ee8ffb17.
Signed-off-by: Dmitri Smirnov <dmitrism@microsoft.com>
Prepacking in subgraph is not supported currently. We see more and more models with subgraph, which has MatMul, MatMulInteger and other ops. Prepacking can speed up those models significantly.
* Change shared providers so that they are shutdown before shared library unload
* Move UnloadSharedProviders declaration into a shared header to avoid bugs.
* Add session option and global thread pool option to set denormal as zero.
* Revert unneccessary changes.
* Add cpuinfo submodule
* Add more comments
* Remove cpuinfo submodule dependency and check only SSE3 support for ftz and daz inspired by Tensorflow
* Preserve API order in C api
* Clean up and utilize SSE3 detection logic from existeing cpuid_info.h
* Keep the same order with header file
* Fix build issue with Linux pipeline, which has old g++ compiler
* Fix broken build on Linux and remove a duplicated unit test
* Remove reformatting at eigen thread pool
* Remove flatbuffers which is not intentionally added
* Revert "Remove flatbuffers which is not intentionally added"
This reverts commit 9f509a9aaaa3c7832d88854c82fd26b234770b7f.
* Remove flatbuffers which is not intentionally added
* Resolve comments
- Put details on APIs
- Add a log for ftz/daz initialization
- Add clang
- Fix typo
* Remove unnecessary header include
* Resolve comments
* add Python API for getProfilingStartTime
* debug for using Python API
* add in C# api
* use uint intead of uint64_t to prevent warning
* typo for GetProfilingStartTimeNs
* remove const
* Update onnxruntime/python/session.py
Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
* remove unnecessary return
* Add Python unit test
* Add C# unit test and refactor Python test
* use ulong in C# for uint64_t in C++
* remove time.monotonic_ns
* syntax: remove public for inner function
* correct the API's order
* getprofilingstarttime after run
* Correct the right order in NativeMethod.cs
* update order
* nit: remove spaces
* Update csharp/src/Microsoft.ML.OnnxRuntime/InferenceSession.cs
Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com>
* use the updated function
* add comment about the precision
* add more comments
* add session.py back
* fix flake8
* remove session.py
* Add comments in C, C#, Python APIs about precision
Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com>
* Add CUDA option to run copy in default stream
This change fixes#4829. Thanks @maherzog for providing the repro!
The bug is caused by memory reuse in BFC arena, where copy and
compute stream in CUDA has a racing condition.
BFC arena is an arena allocator on top of cudaMalloc/Free to
reduce the cost in syncing CPU and GPU when alloc/free. It means
when CPU alloc/free the memory, GPU might not finished previous
work on the memory, so that CPU and GPU could run asynchronously.
This is OK if there's only one stream, where the execution order
in CPU and GPU are consistent. For example, if we have two kernels
A and B, CPU runs allocA->computeA->freeA->allocB->computeB->freeB,
A and B could shares the same memory since computeA and computeB
will not have racing as long as they run in the same GPU compute
stream.
However, if CPU runs allocA->CopyA->freeA->allocB->computeB->freeB,
the order of execution in GPU could have copyA happen after computeB,
if copy and compute happens in different GPU streams.
This change makes copy to run in default compute stream, while adding
an option to fall back to previous behavior if there's perf hit. This
is a short term fix before BFC arena could support multiple streams.
User may use following options to revert to previous behavior:
C API:
struct OrtCUDAProviderOptions cudaProviderOpt;
cudaProviderOpt.do_copy_in_default_stream = false;
C++ API:
CUDAExecutionProviderInfo cudaEPInfo;
cudaEPInfo.do_copy_in_default_stream = false;
C# API:
pending...
Python:
import onnxruntime
onnxruntime.capi._pybind_state.set_do_copy_in_default_stream(False)
* Confirmed the test failes in CI when doing copy in separate stream
Revert the test to get CI pass now
* Fix Windows test
* Address CR
* - Link with libatomic if needed
- Install pip differently so it doesn't clash with the system pip which may involve a wrapper script
- Remove ability to specify offset when Tensor allocates the data. The data prior to offset isn't accessible by anything.
- Fix use of offset in TensorOpTest to work on armv7 where it must be aligned to the type it points to.
- Fix ActivationOpNoInfTest.Softsign to allow for armv7 behavior
- Fix ReductionOpTest.ReduceMean_*keepdims to allow for armv7 floating point inaccuracy
* Address PR comments
* Expose recompute configs to the frontend
* Add frontend test
* Ensure recompute graph transformer is only applied once
Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* Move fbs include from header to cc
* add initial cmake for flatbuffers
* Move most flatbuffers util to ort_flatbuffers
* move code around
* fix
* move test/perf runner to use flatbuffer directly instead of model
* minor update
* Fix build break
* Clean up includes and foward decl
* Fix traning CI build breaks
* Addressed PR comment, replaced some include with forward decls
* Remove ORT_MUST_USE_RESULT temporarily
* Build Recomputation Graph
* Make topological sort to run FW nodes first
* Pattern match start and end of transformer layer
* Topological sort with Priority
* Add logger to Gradient Graph Builder
* Use Logger
* Introduce Execution Order
* add custom logger and global threadpools to C and C++ API
* code cleanup and formatting
* reformat code
* tidy up some more code formatting
* remove comment
* fix API break from merging from master
* renamed API function to CreateEnvWithCustomLoggerAndGlobalThreadPools
* rename log variable and apply clang-format
* Place shape related nodes in CPU
* visit candidates by topological order
* Make CPU node placement a utility function
* skip placing on CPU if the data typs is float16 or bfloat16
* Allow sharing of initializers between sessions.
* Allow sharing of initializers between sessions (2).
* Add test for C#
* Add test for C#; address PR comments
* Address PR comments
Moved AddInitializer logic to internal session options
Added tests for owned buffer
Clarified documentation
Fix bug where memory info and not device was getting compared
* Fix test
* Fix training build
* Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
* Refactor TensorAt
locations* must be const and int64_t since our dims are int64_t
Remove unnecessary copy of locations.
Remove unnecesary casting and C-casting. Simplify implementation.
Add a check for string type.
Make CXX api return T& to fully expose C API in C++, const std::vector& by value as it
covers more ground and eliminate redundant copy.
Eliminate inner loop, compute strides first.
* add GetStartTime() for profiler
* add function in inference_session
* remove qualified name
* add the api in cxx_api.h
* rename starttime to StartTimeNs, expost profiling object
* rename GetProfilingStartTime
* move Ortapis to the right place
* move to the end
* add const for session
* const the right place
* use const auto instead of const auto* for session
* remove const for auto getstarttime
* remove const for auto getstarttime
add unit tests
* nit: update test name and add comments
* Added config flags for VPU Fast Recompile
* clean-up ifdefs
* Add VPU Fast compile config option
Adds an option that enables Fast compilation of models to VPU
hardware specific format.
* Add config option to choose specific device id for inference
Inference of all subgraphs will be scheduled only on this device
even if other devices of the same type are available.
* Add Python API to list available device IDs
* code cleanup
* Add second C/C++ API with settings string parameter
Adds an additional C/C++ API that allows passing multiple
key-value pairs for settings as a single string. Multiple
settings are delimited by '\n' while the key and value
within a setting are delimited by '|'.
* Append 'Ex' to the extended C/C++ API
* Use set_providers Py API to set config options.
Uses Session.set_providers Python API to set EP runtime config
options as key/val pairs
Deprecated older module function definitions for config settings.
Updates documentation.
* avoid globals for py config options where possible
Co-authored-by: intel <you@example.com>
* Remove SparseTensor support from minimal build.
Currently the only valid usage of a SparseTensor is as an attribute of a Constant node. That would have been lifted to a dense tensor initializer when loading the onnx model, so would not exist when saving the ORT format model. Due to that there can be no SparseTensors in an ORT format model.
Co-authored-by: gwang <wanggy@outlook.com>
* Remove serialization of outer scope node arg info in ORT format model. We don't currently need it in a minimal build as only SessionState calls Graph::IsConstantInitializer and it doesn't search outer scope. If we do need it in the future the information can be calculated at runtime (small binary size cost to do so).
Motivation: ORT format model was 32% bigger for a BERT model with multiple levels of subgraph and a lot of nodes due to this. Size is about 5% larger of the original ONNX model with the change. ORT format has type/shape info for all nodes, and this model has 2000 nodes so this seems reasonable.
Added example code to dump ORT format model to json.
Fixed misc bug in python test script around handling float and non-float expected output.
* opset13 cuda kernels for BERT.
* add opset13 SoftmaxCrossEntropyLoss.
* opset13 size.
* fix argmax/min for ut.
* fix ut failure for argmax/min.
* OrtMemTypeCPUInput
Co-authored-by: Vincent Wang <weicwang@microsoft.com>
* Add minimal build option to build.py
Group some of the build settings so binary size reduction options are all together
Make some cmake variable naming more consistent
Replace usage of std::hash with murmurhash3 for kernel. std::hash is implementation dependent so can't be used.
Add initial doco and ONNX to ORT model conversion script
Misc cleanups of minimal build breaks.
* Add SetLanguageProjection C Api and use it in four projections
* static cast enum languageprojection to uint32_t
* resolve comments
* fix typo and line added unintentionally
* revert unecessary change
* reorder c# api
* add TensorAt and CreateAndRegisterAllocator in Csharp to keep the same order as C apis
* Changes to enable saving and loading an ORT format model via the public APIs.
Cleanup session.py to try and make slightly more understandable. More refactoring is needed here.
Couple of bug fixes
* Fix bug in handling NodeArg serialization for optional inputs which has a name and no type info.
* Address PR comments
- tweak SessionOptions config to avoid double lookup
- merge duplicated functionality in python binding around registering an EP with optional options
Fix a couple of build issues.
* Update C API to be consistent with python API
- only load model in InferenceSession ctor if required
- support loading ORT model in minimal build
* Fix nodejs test.
We get an invalid path error from LoadInterOp first now
* Another attempt at fixing nodejs test.
Error message depends on whether ENABLE_LANGUAGE_INTEROP_OPS is defined. Make the output consistent.
The interop implementation looks suspicious given it appears to be internal code that is going via the public api. TBD if that should be fixed.
* Fix couple of build issues.
* Disable test temporarily so PR can be checked in.
Will fix in separate PR that adds final pieces for minimal build as the test is required there.
* Give up on nodejs test and make the match simpler.
Fix init call in TrainingSession python to not pass through sess. it wasn't being used in Session anyway so passing it through just adds confusion.
* Fix call to Session.__init__ in TrainingSession.
Session now initializes Session._sess to None to make it clearer where the 'ownership' of that member is, and that needs to happen before TrainingSession sets it.
* Rename DeviceAllocatorRegistrationInfo to a more generic name; Remove OrtMemType; Simplify CreateAllocator interface.
* - fix builds
- fixed mixed aggregation + constructor calls (which were coded before this PR)
- changed default value of max_mem in API header
- added some validation of values for for arena_extend_strategy
* fix tensorrt and cuda tests
* enable rejecting models based on onnx opset
* enable unreleased opsets in linux and mac CI
* test fixes and more updates
* enable unreleased opsets in CI builds
* enable released opsets in linux cis
* try fix windows ci yml
* yml fixes
* update yml
* yml updates post master merge
* review comments
* bug fix
* Extend C++ API for Map/Sequence Type Info (#3517)
Expose functionality to view type information about sequences/maps
to C++ API.
- Add functions
- `TypeInfo::GetSequenceTypeInfo`
- `SequenceTypeInfo::GetSequenceElementType`
- `TypeInfo::GetMapTypeInfo`
- `MapTypeInfo::GetMapValueType`
- `MapTypeInfo::GetMapKeyType`
- Add structs
- `SequenceTypeInfo`
- `MapTypeInfo`
Co-authored-by: Dudeldu <mustermann.informatik@gmail.com>
Co-authored-by: Jonas-Heinrich <Jonas@JonasHeinrich.com>
* Extend tests to cover new type info functionality for sequences and maps
- two new test case in test_nontensor_types for maps and sequences
Co-authored-by: Jonas-Heinrich <Jonas@JonasHeinrich.com>
* Next round of changes.
Remove inclusion of ONNX schema header
Exclude custom registry related things
Move IsConstantInitializer from graph_utils to Graph as it's needed in a minimal build and graph_utils is excluded.
* Add support for sharing allocators
* Incremental update
* Address some PR comments, add unit tests, add documentation.
* Address PR comments, add tests and some documentation.
* Fix build and test issues
* Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this.
Also fixed the orttraining segfault. The segfault was because in the case of training session,
the CPU exec prov is not available at the time the transformers are applied. Changed it to create
a new one.
* cancel night build on pyop
* add rewriter to rewrite cpu provider
* skip BuildKernelCreateInfo<void>
* refactor variable name and comment
* include ops from csv file
* process multiple eps
* add default function to cuda provider
* rename function and add license header
* fix import
* add doc
* fix typo
* deal with empty kernel entry in cuda
* rename the rewriter file
* add comment into provider file
* add comment and rename function
* log warnings
* refactor extracting logic
* add entry for script to run solo
* add better example
* avoid onnx importing
* fix flake8 alerts
* minor fixes to better comments and doc
* add entries for all domains
* add void entry into contrib providers
* format cuda_contrib_kernels.cc
* format cpu_contrib_kernels.cc
* add all providers
* add default entry to all providers
* include op_kernel header
* cancelling change in providers beyond cpu/cuda
* rename file and switch file format to domain;opset;op1,op2...
* update doc
* restore non-regular ending grammar in cuda_contrib_kernels.cc
* add ort_root as input argument of script
* enable test in ci
* update doc
* update doc
* revert change on linux gnu ci
* switch to set to host ops
* simplify trimming logic
* add domain map to track current model
* allow ort_root to take relative path
* Initial set of changes to start disabling code in the minimal build. Breaking changes into multiple PRs so they're more easily reviewed. Focus on InferenceSession, Model and Graph here. SessionState will be next.
Needs to be integrated with de/serialization code before being testable so changes are all off by default.
Changes are limited to
- #ifdef'ing out code
- moving some things around so there are fewer #ifdef statements
- moving definition of some one-line methods into the header so we don't need to #ifdef out in a .cc as well
- exclude some things in the cmake setup
* Update session state and a few other places.
The core code builds if ORT_MINIMAL_BUILD is specified.
* Add Node::SinceVersion() so that the value is known when loading a graph from the ORT format (OpSchema is not available).
* Fix build warning from returning 'const int'
* adding generic configurations for session options
* fix a build break on linux
* fix training ci build break
* fix training ci build break
* addressed CR comments
* fix traning ci build break
* move config_key from enum to string
* add c# api
* add python api
* fix build break
* move prepacking from 2 new api entries to session options configs
* fix traning ci build break
* add python test, update some comments, move const key definition to avoid build break
* addressed comments
* move definitions of keys to common.h
* move api to version 5
* remove accidental change in build.py
* remove pragma to avoid build break
* addressed CR comments
* fix the python build break, and move location of config keys definition
* small typo changes
* Eliminate redundant subexpressions
Apply local value numbering to merge graph nodes that will always
evaluate to the same value.
* Rename cpp->cc
* Handle optional arguments
* Add test models
* Add more tests with optional arguments
* Fix processing of subgraphs
Also, be resilient to possible mixture of optional and variadic
parameters
* Fix random operators
* Address PR comments
* Minor changes and a test
* Move CSE before constant folding
* Random* operators are always non-deterministic
Even when seed is provided.
* Fix a CSE test
* Reuse the list of non-deterministic operators with constant folding pass
* Address PR comments
* Fix formatting
* Address PR comment
* Minor cleanup / comments
* Fix build failure in Linux
* Reuse existing optimizer/utils file.
Also, check for graph outputs when removing a node.
* Add a test
* Fix compiler warnings
* Fix build in older compilers
* More compatibility with old STL versions
This commit means that when the thread pool is configured to spin, then we spin at the barrier at the end of parallel sections in the main thread, in addition to having workers spin waiting for work.
The change updates Barrier.h to take an additional boolean to select spin/block, and passes this in based on the thread pool configuration.
It adds an additional test case for barriers, although no problems were identified by the test case.
* Gelu Activation Recompute Draft
* Prototype for localized recompute
* Introduce localized_recompute rewriter
* Command line args for enabling recompute
* Add logger to Gradient Graph Builder
* use const when possible
Update TransposeMatMul to support scaling of the matrix product by a constant scalar value (analogous to the GEMM alpha parameter). Rename TransposeMatMul to TransposeScaleMatMul.
Fuse MatMul with surrounding Mul/Div with constant scalar into TransposeScaleMatMul.
While investigating an unrelated issue, I noticed that the thread pool may drop tasks when a burst of 1024+ tasks is submitted by a thread from inside the pool. Today, in general, we execute work synchronously in this case. However, there is a bug where work submitted by a thread already inside the pool will be discarded instead of executed. Currently the only scenario where I can see this occurring is when the parallel executor is used with a model in which such a large number of nodes become eligible to run all at once. This PR fixes the underlying issue and adds a test case for burst-submission of work.
* Add ability to retrieve inferred shapes when executing a kernel.
This ability helps Recv to know its output shapes without doing
actual cummunication. Of course, if the output shapes cannot be
inferred, Recv still needs to do communication to get shapes from
Send.
* Avoid communicating shape information when it can be inferred statically
* Replace unordered_map with thread-safe wrapper.
We don't want to have racing condition and undefined behavior
when using parallel executor.y
* Remove cout
* Add missing file
* Address comments
* Check dim_value. -1 means missing
* lock properly
* Address comments (remove thread-safe map)
* Remove poc header
* Replace Stream with DeferredReleaseCPUPtr
* Add python API for specifying CUDA device id
* Modification for providing session based python api for specifying
device id
* When include header file pybind11/stl.h, conversion between c++
containers and Python list, vector and dict data structure are
automatically enabled.
https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html#
Therefore, refactor the code for better leverage this advantage.
* Make struct CudaDeviceOptions as default cuda device options
* Implement sess.set_providers(list_of_providers, list_of_provider_option_dicts)
But still stay consistent with existing sess.set_providers(list_of_provider)
* Add cuda provider option default setting
* Add support for setting cuda cuda_mem_limit and arena_extend_strategy.
Also resolved the merge conflict on session.py
* Use python ctypes to call cuda library to help python unittest
* Refine the code with reviewer's suggestions
* Add the capability of getting execution provider's configuration
- Once we introduced the capability to set execution provider's
configuration, it makes sense to add capability of getting ep's configuration.
* Modify the code with reviewer's suggestions.
* Using stoull() and stoul() depends on 32/64-bits architecture.
* Rewrite the testcases for testing setting CUDA device id
Note: We need to make sure every ORT process be run on one CUDA device
at a time.
* Make sure old session object is destroyed by python gc before new
session object is being created
* Move testcases to original onnxruntime_test_python.py
* Fix bugs to pass CI build
* Make it pass CI build (cont.)
* Make it pass CI build (cont.)
* support bert partition with shared initializer
* address feedback
* address feedback
* address feedback
* add more test
* remove bert-tiny model
* address feedback
* address function comment
* move CreateNodeArg to graph_utils
* rename function name
* rename function name
* fix windows build
* fix windows type conversion warning
* add function comment
Create N-1 threads in a thread pool when configured with intra-op parallelism of N. This ensures we have N active threads, given that the main thread also runs work. To avoid ambiguity on the value returned, rename ThreadPool::NumThreads method to ThreadPool::DegreeOfParallelism, and make corresponding updates in MLAS and operators.
For the special case where all variadic inputs of a kernel are the same shape (i.e. no broadcasting is required) and there are few enough of them, we perform the entire computation in a single kernel. The general implementation (which was previously used for this special case) handles broadcasting by repeatedly invoking a binary kernel on successive inputs.
* add modern standards to function arguments
* code cleanup
* fix code formatting
* add element access convenience function
* change template type name to match rest of code
* remove new At() convenience function
* add better documentation message
* Update function body initialization
* minor fix
* changes per review comments
* minor fix
* format fix
* add function initialization in mixed precision transformer
* more updates
* more fixes
* Move allocators to SessionState so they're decoupled from ExecutionProviders
- when looking up an allocator it's based on OrtMemoryInfo not the EP so SessionState is a more natural place for that infromation to be stored
- add device based lookup
- simplifies logic for copying feeds/fetches across devices
Cleanup SessionState and SessionStateInitializer
- provide more things to SessionState at construction time so we don't construct and instance and immediately after call a bunch of setters
- simplify SessionStateInitializer
- reduced down to FinalizeSessionState method
As a zero-cost wrapper around the C API, the current state of the C++ API is still pretty low-level and requires programmers to use C-style standards to interact with ONNX.
- Move thread hint vectors from thread-local struct
- Add static_assert that the per-thread state in the thread pool is trivially-destructible
- Rename "thread_data" to "worker_data" (only allocated for workers in the pool, not threads calling into the pool)
Updates the thread pool implementation to make work distribution over the Eigen thread pool more closely resemble techniques used in OpenMP. In particular:
(1) A thread entering a parallel loop works on the iterations itself, rather than requiring a thread switch to/from a thread in the pool, if called from outside the thread pool.
(2) To support this, work items pushed to the thread pool run a loop to claim iterations from a shared counter via atomic-fetch-and-add, as opposed to having work items themselves represent individual batches of iterations. This means that any thread working on the loop can execute any batch of iterations, including having the main thread run through all of the batches itself if the loop turns out to be short-running.
(3) As with OpenMP active scheduling, the worker loop spins waiting for work prior to blocking. This avoids OS blocking / wake-up paths in workloads with series of short-running parallel sections.
* Added GetAvailableProviders to C API
* Fix API version and Windows build error
* Changed function name
* Changed ORT_API_VERSION to 4
* Moved all_providers array to constants.h
* Move check for providers to constants.h
* Changed name of array to avoid warning
* Address review comment
* Added unit test
- Update IAllocator setup to move the OrtMemoryInfo to the base class instead of requiring derived classes to have that as a member and override a virtual method to return it.
- Cleanup CreateAllocator setup to take an argument as to whether to wrap the device allocator in an arena allocator. The choice to do that isn't a property of the underlying device allocator.
- Minor cleanups in the various EPs to adjust to the change to IAllocator and CreateAllocator, and to use the create_arena flag consistently when available.
* Enable static memory planning for pipeline.
1. We fix a bug when resolving symbolic shape for scalars.
2. We pass the original inputs to all pipeline stages so that
the symbolic shapes can be resolved.
* Further Improvements
1. Address comments.
2. Further reduce activation size by ~50% when pipeline is on.
This is done by removing all but one gradient tensor from the last
RecordEvent in the backward pass.
* Address a comment
* Fix Windows build
* Fixes from investigating issue running BERT-Squad model with larger batch sizes. When the batch size gets large enough the initial run will be successful (no memory pattern in use) but the second will fail to allocate the memory pattern block.
The cause of this failure is that we still have the smaller blocks from the first run allocated, as BFCArena has no logic to free those. This essentially results in 2x the memory being required to run the model.
There was inconsistency in BFCArena::Extend which on one path threw an exception if it couldn't do the allocation, and on another just returned false (resulting in Alloc returning a nullptr). Make the behavior consistent by always throwing if BFCArena fails to find a buffer to return. There are a huge number of places in the code where we assume Alloc returns a valid pointer so throwing will result in more correct behavior as a whole. It's also consistent with what happens when CUDA or the standard library fails to allocate memory.
Next, update ExecutionFrame to check for this failure and not insert a memory block entry if it happens. With the existing code if BFCArena Alloc returned a nullptr we happily inserted that in the blocks, delaying detection of the failure to when we attempted to use the block in AllocateMLValueTensorSelfOwnBufferHelper.
Finally update AllocateMLValueTensorSelfOwnBufferHelper to expect a location may not have a block. A log message will be provided when the block allocation fails so it's not necessary to have more on each individual allocation that would have used the block. Falls through to default behavior of doing a normal allocation.
* Add ArmNN Execution Provider
Add a new execution provider targeting Arm architecture based on ArmNN.
Validated on NXP i.MX8QM CPU with ResNet50, MobileNetv2 and VGG models.
reviewed-by: mike.caraman@nxp.com
* Minor fixes
- renamed onnxruntime_ARMNN_RELU_USECPU to onnxruntime_ARMNN_RELU_USE_CPU
- fixed acl typo
* remove extra includes. added exception for ArmNN in test
* fix indentation
* Separated the activation implementation from the cpu and fixed the blockage from the endif
Co-authored-by: Andrei-Alexandru <andrei-alexandru.avram@nxp.com>
* online partition
* fix when multiple consumer nodes is in cut info
* fix windows build
* address feedback
* adding test
* feedback
* address feedback
* add parser for cut edge
* windows build
* Add amd migraphx execution provider to onnx runtime
* rename MiGraphX to MIGraphX
* remove unnecessary changes in migraphx_execution_provider.cc
* add migraphx EP to tests
* add input requests of the batchnorm operator
* add to support an onnx operator PRelu
* update migrapx dockerfile and removed one unused line
* sync submodules with mater branch
* fixed a small bug
* fix various bugs to run msft real models correctly
* some code cleanup
* fix python file format
* fixed a code style issue
* add default provider for migraphx execution provider
Co-authored-by: Shucai Xiao <Shucai.Xiao@amd.com>
* Fold Shape node in constant folding.
* bugfix
* Fix test failure.
* Bugfix for C++ frontend.
* Bugfix for C++ frontend.
Co-authored-by: Vincent Wang <weicwang@microsoft.com>
* add build inbox flag
* remove raw tests and wstring for utf filenames
* enable raw tests
* use ToWideString
* create new utf8 helper
* update string helper to utf8
Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
1. Parallel all the activations ops.
2. Parallel the performance critical path of the LRN op, which makes the ONNX model zoo googlenet model runs 60% faster(latency reduced from 21ms to 13ms).
3. Make the Gemm-Activation fusion support with all the activations ops. Before this change, it only supports LeakyRelu/Relu/Sigmoid/Tanh.
4. Delete onnxruntime/test/framework/op_kernel_test.cc because the file is almost empty.
5. Remove the loggings in KernelRegistry::TryFindKernel, return Status with error message instead.
* Add TrySimpleParallerFor so that there's a path with OpenMP awareness for SimpleParallelFor. Makes it consistent with [Try]BatchParallelFor and [Try]ParallelFor.
Update TopK to check for the number of threads better, and to use TrySimpleParallelFor.
* Update doco to mention TrySimpleParallelFor
* allow switching between eval and training modes dynamically
Co-authored-by: Tixxx <root@525204a066204ea794f942530b05ae7f000000.axlncovkyjne5caro2tmz3zryb.xx.internal.cloudapp.net>
* Added FP16 transformations
* Revert "Added CMAKE_BUILD_TYPE to make building dynamic"
This reverts commit d3e17af1af655cfdc4d2fec33f52055caa525e85.
* Added FP16 transformations for FP16 builds
* Backend logic cleanup
Cleans the backend(intel_graph.*) code in the following ways:-
1. Minimize global usage: Since all the IR graphs need to be
re-generated on every Infer, it is bad practice to rely on globals
for their saving and usage as there would be multiple readers and
writers to the same global variable leading to incorrect usages or
contentions. This change replaces globals with locals where possible.
This change also fixes an existing bug with due to
incorrect global usage.
2. Remove all unused functions.
3. Remove all unused headers and prepocessor directives.
* removed commented out code
* Disabled default optimization for Intel EP
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Fix missed plugins.xml for python bindings
* Fixed the build after latest master changes
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled unsupported ops for accelerators
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Added some more disabled ops
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Added environment variable to enable debugging
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Added more debug statements
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Fixed unsupported ops list for GPU and VPU
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Fixed unsqueeze unit tests
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Added error message to the status
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Overwrite Model proto with shape info from data
Overwrites the shape info of Model proto with the shape from
actual input data. Needed for inferring models with Dynamic
shapes.
* Removed print statement and disabled where op
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled Reshape with Empty initializer
* Added more debug statements for 1P
* Don't allow 1D inputs with symbol for dimension
* Disabled some 3rd phase ops
* Disabled split and added zero dimension check for OutputDefs
* Cleanup zero dimensionality check
* Added different data type check for inputs and initializers
* Added conditions for Mod, Cast and Pad
* Removed unused variable
* Disabled scan and added conditions for squeeze
* Added changes for fixing all C++ unit tests
* Implements Backend Manager class for caching
Backend Manager provides a layer of indirection between EP interface
and OV backend that provides caching services for models with
symbolic dims in input shapes.
* clean up commented blocks
* clang-formatting
* Read I/O type info from ModleProto
Read the tensor element type information from ModelProto object,
as FusedNode is no longer available.
* code cleanup
* clang-formatting
* Added print statement for jenkins
* Disabled some python tests
* Changed the path of convert fp32 to fp16 hpp
* Added conditions for BatchNorm in GetCapability
* Fixed failed tests
* Revert "Added conditions for BatchNorm in GetCapability"
This reverts commit c3c28c3b00d27892c42546b35dacdd807a48ee90.
* Added Intel to onnxruntime backends
* pick up vars set by OV package setupvars.sh
* Added conditions for Identity
* remove a few cout prints
* Added conditions for GPU_FP32 unit tests
* Revert "pick up vars set by OV package setupvars.sh"
This reverts commit 8199e029c03eae21a1a7ef6bfdc93d00e5d0198b.
* Commented out fatal message for protobuf
* Might need to be removed
* Add interface class for current backend
* moved common logic to base class
* simplified cpu backend
* Removed unused headers
* use vectors to save i/o tensors for windows compatibility
* move utils fxns to backend_utils namespace
* rename ov_backend to ibackend
* Factory pattern for backend creation
* rename CPU backend to Basic backend
* renamed to vad-M and added to factory list
* Added conditions for VPU
* Added print statements
* Changed the logic for checking for symbolic shapes
* Modified logic for zero dimension check
* Removed VPU single dimension condition
* Removed comments
* Modified logic in DimensionCheck method
* Remove legacy OpenVINO EP
Remove all the legacy code for OpenVINO EP. UEP code will take its
place going forward.
This change does NOT remove OVEP files in the following areas asa
they will be reused by UEP:-
1. Documentation: All .md files
2. Docker releated files
3. Python bindings
4. Java bindings
5. C# bindings
6. ORT Server
7. CI pipeline setup files
* Rename Intel EP to OpenVINO EP
* Added unique names to the subgraphs
* Removed subgraphs with only constant inputs
* Modified subgraph partitioning algorithm to remove const input subgraphs
* Apply suggestion to onnxruntime/core/providers/openvino/openvino_execution_provider.cc
* Tracking output names to fix the output order bug
* Changed output names to a unordered map
* Modified logic to check for symbolic input shapes
* Fixed a bug in Reshape check
* Added empty model path to Model constructor
* Made necessary changes to cmake to build from the binary package
* Changed INTEL_CVSDK_DIR to INTEL_OPENVINO_DIR
* Enable dyn device selection with C++ API
* Added Round operator to unsupported list
* Modified subgraph partition logic for MYRIAD
* Removed supported ops from the list
* Enable dyn dev selection in Py API's
* Add documentation for dynamic device selection
* Use MYRIAD || HDDL instead of VPU
* Removed temporary cast of Int64 to FP32
* Disabled unit Tests for CPU_FP32 and GPU_FP32
* Removed default "CPU" from unit tests to allow overriding
* Removed ops Concat, Squeeze, Unsqueeze from unsupported list
* Get the device id from info
* Removed overwriting device_id and precision
* Enabled ConvTranspose and EyeLike
* Reordered unsupported ops in alphabetical order
* Fixed syntax error
* Fixed syntax error
* Code clean-up: Handle exceptions, logs and formatting
Code formatted according to ORT coding guidelines.
* remove debug print from pybind code
* updated docs with ops and models
* formatting prints
* Added default values for c and j for openvino
* Overriding the values set for c and j to be 1
* BACKEND_OPENVINO should be empty if openvino is not in build
* Overriding c value with default for perftest
* fix VAD-M device string bug
* Add IE error details to exceptions
* Use IE specific device names in EP
* Add VAD-F (FPGA) device support
* Removed unecessary libraries from whl package
* Code changes for Windows compatibility
* Add VAD-F option to python API
* [revert before merge] cmake changes for RC
* Enable Windows build in CMake
* Unset macro OPTIONAL for windows builds
inference_engine.hpp's include chain defines a macro 'OPTIONAL'
which conflicts with onnx project's headers when using MSVC. So
would need to explictly unset it for MSVC.
* Use a single copy of plugin/IE::Core
Defined as a static member in Backend manager
* Remove restriction of single subgraphs for myriad
* Passed subgraph name to Backend to enhance log statements
* Disabled zero dimension conditions
* Disabled concat to remove zero dims
* Enabled building ngraph as part of ORT
* Removed serializing and added versioning
* Fix CPU_FP32 unit tests
* Removed unecessary condition
* add ngraph.so.0.0 to .whl
* Check for zero dimensions only for inputs and outputs
* Restrict loading only 10 subgraphs on myriad
* Build ngraph.dll within UEP. Doesn't link yet
* Rename Linux included libngraph.so to libovep_ngraph.so
Renames locally built libngraph.so containing ONNX importer to
libovep_ngraph.so in order to avoid linkage conflicts with
libngraph.so supplied by OpenVINO binary installer.
Applies only for Linux builds.
* use output_name cmake properties for lib name
* fix .so name format in lib_name.patch
* CMake code cleanup
* Rename WIN32 included ngraph.dll to ovep_ngraph.dll
To avoid conflict with ngraph.dll distributed by openvino.
* Added myriad config for networks without 4 dimensions
* Loading the 10 max clusters for inference on myriad
* Refactor code and add Batching support
Encapsulate subgraph settings into context structs.
Add batching support for completely supported models.
* Disabled some broken tests
* use input_indexes to avoid batch-checking initializers
* Avoid static initialization order error on WOS
* Added candy to broken tests
* InternalCI changes for 2020.2
* Updated DLDT instructions
* Unsaved changed in install_openvino.sh
* Changes after manual check
* Remove custom ngraph onnx_import build for WOS
ONNX Importer on WOS does not have protobuf issue.
* Remove FP32ToFP16 ngraph pass
This conversion is performed implicitly within IE.
* Surround debug logic by #ifndef NDEBUG
* remove invalid TODO comments
* removed references to ngrpah-ep
* clang-formatting
* remove commented code
* comment edits
* updating copyright year to that of first OpenVINO-EP release
* remove redundant log msg
* Modified operator and topology support
* Update build instructions
* doc formatting
* Fixed clip unit tests
* Revert "Remove FP32ToFP16 ngraph pass"
This reverts commit ec962ca5f315a5658ad980e740196f19de2639c1.
* Applying FP16 transformation only for GPU FP16
* Fixed GPU FP32 python tests
* automatically use full protobuf
* disable onnxrt server for now
* Disabled upsample
* update dockerfile instructions
* Removed MO paths and added ngraph path
* Remove OVEP from ORT Server docs
Will put it back in after validation
* Updated path to Ngraph lib
* Disabled Resize and some other python tests
* Removed unnecesary header files
* Use commit SHA to fetch ngraph repo
* Avoid un-needed file changes due to version update
* Fixed clip tests
* Fixed Pow, max and min onnx tests
* build.md doc typo
* Update cmake patch command for ngraph src
* remove dead cmake code for onnxruntime_USE_OPENVINO_BINARY
* use spaces instead of tab
* remove commented code
* Add info about protobuf version
* edit debug env var and enable for WIN32
* specify only version tag of 2020.2 for dockerbuilds
* remove unnecessary file changes
* Pass empty string as default argument to C# tests
* Use ${OPENVINO_VERSION} to name openvino install directory in CI builds
* Enabled unnecessarily disabled tests
* Fixed ngraph protobuf patch
* Fixed error in protobuf patch
* Revert "Use ${OPENVINO_VERSION} to name openvino install directory in CI builds"
This reverts commit 89e72adb8bf3b9712f5c81c5e13fe68c6c0df002.
* Remove unsetting OPTIONAL macro
This is no longer used in recent ONNX update onnx/onnx@da13be2,
so this unset workaround is no longer necessary.
* Use a null string default argument for C# API
* Set OpenVINO version yml files and pass to CI Docker builds
Git Tag info for DLDT as well as install directory are set
using this value.
This reverts commit 9fa9c20348ed72ae360a95c98e9b074d2f9fafc5.
* Documentation: recommendation and instructions for disabling ORT graph optimizations
* more doc updates
* Reduced the number of models according to CI time constraints
Co-authored-by: ynimmaga <yamini.nimmagadda@intel.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
Co-authored-by: Mikhail Treskin <mikhail.treskin@intel.com>
Co-authored-by: mbencer <mateusz.bencer@intel.com>
Co-authored-by: Aravind <aravindx.gunda@intel.com>
Co-authored-by: suryasidd <48925384+suryasidd@users.noreply.github.com>
Threadpool related changes.
Don't create ORT threadpool if openmp is enabled (except for inter op threadpool).
Created a new static function ThreadPool::NumThreads to account for openmp settings and null threadpool ptr.
Log a warning when using SetIntraOpNumThreads when openmp is enabled.
Added a document for ORT devs.
Fix LSTM to use the new threadpool abstractions.
Rename GetNumCpuCores to GetThreadAffinityMasks and move it to the Env class.
Co-authored-by: Tracy Sharpe <tracysh@microsoft.com>
1. Fix static analysis warnings found by VC++
2. Add a new pipeline for static analysis
3. Merge all the windows CI build into one single yaml file.(Easier to queue them all).
4. Make DNNL build faster by disabling building the tests and examples.
5. Enable custom op unitest.
Fix training modification of Graph SetInputs() and SetOutputs(). Originally there were distinct code paths in Graph based on whether the graph was loaded from a GraphProto or created from scratch. The training modifications made that distinction a bit ambiguous - i.e., even though the Graph is loaded from a GraphProto for training, sometimes we rely on the other code path, e.g., to deduce the graph inputs after modifying it. Consequently, there was some odd behavior when using SetInputs(). For correctness, this change separates the cases where the graph is loaded from a GraphProto and where it is created from scratch.
1. Copy tensorflow's thread pool class to ORT, so that we can get a better implementation of thread pool based parallelfor
2. Copy Eigen's thread pool class to ORT
3. Support thread affinity
4. Remove RNN kernel’s private thread pool
5. Modify pool kernels to use the thread pool when openmp is disabled.
* Add support for sessions to share a global threadpool.
* Fix build issues
* Add tests, fix build issues.
* Added some documentation
* Fix centos issue when threadpools become nullptr due to 1 core.
* Fix mac and x86 build issues
* Address some PR comments
* Disabled test for android, added few more tests and addressed more PR comments.
* const_cast
* port the mimalloc allocator
* hook mimalloc opt into common.h and reduction ops
* repurpose USE_MIMALLOC to only denote subbing in of default allocator with mimalloc and some refactoring
* fix unintended cherry pick diffs
* polish alloctor_mimalloc
* explicitly disable mimalloc where it already had been disabled
* update mimalloc to pull in stl allocator
* switch mimalloc stl allocator to use mimalloc library version
* turn mimalloc on by default (only the stl changes are enabled, the python interacting ones are off already and shall remain so)
* move FastAllocVector into cpu specific code
* separate out defines into arena and stl changes
* the rest of the define renames
* bfc arena allocator
* some typos and rename the bfc arena allocator to fit existing class naming conventions
* adjustments in response to comments
* different template instantiations are friends
Use CUDA 10.1 for Linux build
(Windows change is already in)
Please note, cublas 10.2.1.243 is for CUDA SDK 10.1.243, not CUDA 10.2.x. CUDA 10.2.89 need cublas 10.2.2.89. They match on the last part of the digits.
libcublas10-10.1.0.105 won't work!!!
The cuda docker image by viswamy is already using 10.1, no need to change.
* merge training kernels to master
* merge training kernels to master
* revert two files
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
* merge training kernels to master
Provide alternative std::mutex implementation on Windows. OrtMutex is no longer an alias of std::mutex.
We do it because:
1. This new thing is faster and much much simpler.
2. Static constructors are considered harmful. We should avoid such thing as possible as we can.
* Initial Commit
* Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346)
add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements.
We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything.
* Initial changes for layering
* more snipping to get core into ort
* update build instructions to include --build_shared_lib (#2358)
* update build instructions to include --build_shared_lib
* fix line breaks
* Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368)
* Task 23998197: add winml_lib_core into onnnxruntime.dll
* PR feedback
build break on perf_test
* return proper error when the model path isn't found (#2391)
* LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382)
this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile.
we are going to move this into the L3 so that ryan can start doing intergration testing.
we will pause for a full code review and integration test result prior to going into the L2.
>>>> raw comments from previous commits >>>
* LearningModelSession is cleaned up to use the adapter, and parts of binding are.
* moved everything in the winmladapter
made it all nano-com using, WRL to construct objects in the ORT side.
base interfaces for everythign for winml to call
cleaned up a bunch of winml to use the base interfaces.
* more pieces
* GetData across the abi.
* renamed some namepsace
cleaned up OrtValue
cleaned up Tensor
cleaned up custom ops.
everything *but* learnignmodel should be clean
* make sure it's building. winml.dll is still a monolith.
* model moved over.
everything builds clean.
step !
* weak ref comment
* Layer dev paulm (#2408)
* model moved over.
everything builds clean.
step !
* weak ref comment
* added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects.
fixes model load.
* Layer dev paulm (#2414)
* model moved over.
everything builds clean.
step !
* weak ref comment
* added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects.
fixes model load.
* User/xianz/win ml telemetry (#2410)
* add option to enable winml telemetry
* add option to enable winml telemetry
* clean logs while developping
* clean the log of GUID
* compile onnxruntime_common with winml telemetry
* use option for use_telemetry
* rename option winml_use_telemetry to onnxruntime_use_telemetry
* little change
* fixed some lifetime management.
fixed the debug build.
squeezenet passes using winmlrunner for CPU and GPU
* Layer dev paulm (#2423)
* model moved over.
everything builds clean.
step !
* weak ref comment
* added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects.
fixes model load.
* fixed some lifetime management.
fixed the debug build.
squeezenet passes using winmlrunner for CPU and GPU
* PR feedback.
* Layer dev paulm (#2424)
* model moved over.
everything builds clean.
step !
* weak ref comment
* added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects.
fixes model load.
* fixed some lifetime management.
fixed the debug build.
squeezenet passes using winmlrunner for CPU and GPU
* PR feedback.
* couple of fixes and coded getmutabledata()
* Layer dev paulm (#2425)
* model moved over.
everything builds clean.
step !
* weak ref comment
* added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects.
fixes model load.
* fixed some lifetime management.
fixed the debug build.
squeezenet passes using winmlrunner for CPU and GPU
* PR feedback.
* couple of fixes and coded getmutabledata()
* fixed 2 more heap corruptions
* Layer dev paulm (#2426)
* model moved over.
everything builds clean.
step !
* weak ref comment
* added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects.
fixes model load.
* fixed some lifetime management.
fixed the debug build.
squeezenet passes using winmlrunner for CPU and GPU
* PR feedback.
* couple of fixes and coded getmutabledata()
* fixed 2 more heap corruptions
* Add opset and IR check when loading model (#2413)
* Add opset and IR check.
* Add test case for future opsets.
https://github.com/microsoft/onnxruntime/issues/2371
* fixed map and sequence when passing stl types across the ABI .
found a leak in nvidia driver, but skipped it.
all winmlapitests pass now
* Moved SessionOptions over to the abi
* WinML CI (#2412)
* Pass flags to build/test WinML in CI
* Add initial CMake config for unit tests in WinML
* Set winml_unittests standard to C++17
* Add WinML API tests and port them to googletest
* Install WinML test collateral
* Add LearningModelSessionAPITests ported to googletest
* Fix WinML test files encoding
* Add GPU tests
* Add parameterized test, skip GPU tests
* Enable precompiled header
* Remove unused code and collateral
* Remove brand images
* Add dllload.cpp
* Remove images not used in API tests
* Add LICENSE.md to image collaterals
* Add models with licenses
* Remove FNS Candy tests
* Add API test models
* Add ModelInSubdirectory
* Install collaterals post-build with copy_if_different, split common lib
* fix warnings
* Link to gtest_main
* Register WinML TraceLogging provider on Onnxruntime.dll (#2455)
* Register WinML TraceLogging provider on Onnxruntime.dll
* Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE
* No need for ifdef for TraceLoggingOptionMicrosoftTelemetry
* PR feedback
* Move etw registration into lotus environment constructor and deresgister in lotus environment destructor
* Brianma/cpuwinml (#2466)
* allow building winml cpu without dml.
* Brianma/breaks (#2469)
* fix some more breaks
* learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers
* move dml checks out of winml and into the adapter
* better error handling
* Brianma/fi (#2470)
* learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers
* User/xianz/win ml telemetry (#2410)
* add option to enable winml telemetry
* add option to enable winml telemetry
* clean logs while developping
* clean the log of GUID
* compile onnxruntime_common with winml telemetry
* use option for use_telemetry
* rename option winml_use_telemetry to onnxruntime_use_telemetry
* little change
* Add opset and IR check when loading model (#2413)
* Add opset and IR check.
* Add test case for future opsets.
https://github.com/microsoft/onnxruntime/issues/2371
* WinML CI (#2412)
* Pass flags to build/test WinML in CI
* Add initial CMake config for unit tests in WinML
* Set winml_unittests standard to C++17
* Add WinML API tests and port them to googletest
* Install WinML test collateral
* Add LearningModelSessionAPITests ported to googletest
* Fix WinML test files encoding
* Add GPU tests
* Add parameterized test, skip GPU tests
* Enable precompiled header
* Remove unused code and collateral
* Remove brand images
* Add dllload.cpp
* Remove images not used in API tests
* Add LICENSE.md to image collaterals
* Add models with licenses
* Remove FNS Candy tests
* Add API test models
* Add ModelInSubdirectory
* Install collaterals post-build with copy_if_different, split common lib
* fix warnings
* Link to gtest_main
* fix bad merge
* Checking in a staging checkpoint point so that Ryan can work with me in parrallel
* build break.
* Brianma/testfails (#2473)
* add missing ir version to dictvectorizer-string.onnx
* add missing ir version to relu.onnx
* add missing ir version to zipmap*onnx
* add IR version to manually generated models
* remove an unnecessary ifdef dml
* Brianma/windowsai fi (#2475)
* update dockerfiles/README (#2336)
* Make elementwise op run 4 items per thread (#2335)
Description: Describe your changes.
Make elementwise op run 4 items per thread
unroll for loop to leverage ILP
remove unnessary N==0 check inside elementwise GPU kernel
Motivation and Context
Why is this change required? What problem does it solve?
It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model.
If it fixes an open issue, please link to the issue here.
* Add CUDA GatherElements kernel (#2310)
* Updates
* Update test
* Update
* Updates
* nits
* PR feedback
* Update
* Update
* PR feedback
* PR comments
* Update
* Fix build
* Fix build
* Nits
* Fix
* Layer Normalization Fusion (#2319)
basic layer normalization transform
* Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293)
* Add FastGelu cuda op
* Add AddBiasGelu for experiment
* Revert "Add AddBiasGelu for experiment"
This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b.
* Add bias
* Add unit tests
* update comment
* update script
* fix build error
* update coding style
* update for CR feedback
Enable half2 optimization only when cuda arch >= 7.0
* move _Tanh to common.cuh
* implement CPU contrib OP Attention (#2333)
* Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320)
* Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers.
This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model.
* Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input.
* Use non-const iterators in std::find_if calls to make centos build happy.
* Nuget pipeline changes (#2305)
1. refactor the pipeline, remove some duplicated code
2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool
3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml
4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed
* Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281)
* Set ElementType to String type of node metadata, instead of byte[] (#2348)
* Set ElementType to String type of node metadata, instead of byte[]
* Fix spacing
* Introduce PrimitiveType into a Type System along with an integer constant (#2307)
Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method.
* Fix/test dim value of 0 handling in a couple of places (#2337)
* Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0.
Add unit test
Also add unit test for unary op with dim value of 0
* Exclude ngraph from Where test with 0 dim.
* Openvino EP R3.1 onnxrt server (#2357)
* onnxrt server with OVEP
* onnxrt server with OVEP
* Update Dockerfile.server.openvino
* onnxrt server OVEP fix reviews
* onnxrt server OVEP fix reviews
* Implement cuda nonzero op. (#2056)
Implement cuda nonzero op.
* Direct use python numpy array's memory if already contiguous. (#2355)
* Direct use python numpy array's memory if already contiguous. This
could greatly improve performance for session with large input,
like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved.
* Add test case enforce contiguous/non-contiguos numpy array as inputs.
* Add helper to create output to minimize binary size. (#2365)
Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor.
* fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369)
* fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS
* update
* Add Tracelogging for profiling (#1639)
Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON
* test bidaf with nuphar for avx target (#2370)
increase nuphar test coverage a bit
* Fix a bug in TLS refcount that may destabilized CUDA CI (#2374)
* update output size calculation for resize (#2366)
* change how output size is calculated for resize op
* add tests for ver 10 resize
* Extend OneHot CPU kernel to support more types (#2311)
* Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float
* Skip BERT before the test data fix is picked up
* Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372)
* Add opset 11 version of Split to CUDA ops (#2376)
Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops)
* Layer Norm Fusion Fix (#2379)
* layer norm fusion fix
* Add input shape check in code and unit tests
* Fuse Add + Gelu (#2360)
Implement the transformer to fuse add + gelu
Implement the accurate kernel
* Skip layer norm transform (#2350)
* skip layer normalization transformer
* Another try to stabilize CUDA CI (#2383)
The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check.
* fix BUILD.md typo (#2375)
build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo')
* Fixed compilation with ngraph (#2388)
* Fix reuse logic in allocation planner. (#2393)
* Fix reuse logic in allocation planner.
* PR comments
* Add helpful comments
* Don't allow reuse across string tensors.
* [NupharEP] Multiple optimizations (#2380)
Fuse transpose into MatMul
Implement Pow and constant scalar simplification
Vectorize ReduceMean
Improve symbolic shape inference
Minor updates for better debugging in fused function name
* Avoid using the default logger in the graph lib and optimizers (#2361)
1. Use the session logger if it is available.
2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it.
* Change CUDA implementation of Transpose to support all fixed size tensor types (#2387)
* Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size.
Add support for 8, 16, 32 and 64 bit types.
Add unit tests.
Add method so the implementation can be called directly (will be used by CUDA Scan very soon).
* Disable TensorRT for MLFloat16 and int8 unit tests.
* Address PR comment and add support for calling cublas implementation if type is mlfloat16.
* Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398)
* Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added.
* [NupharEP] force some low/zero cost ops to be inlined (#2409)
* fix cross compile bug (#2415)
* Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417)
* Add Reshape Fusion (#2395)
* Add reshape fusion
* Add some comments
* update comments
* update comment format
* update according to feedback
* update for recent logger change
* fix build error
* (1) Support both input and output edges in find path in graphutils
(2) Add a test case of only one constant initializer of Concat input.
(3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future.
* fix error
* (1) loose constraint on initializer: non constant is allowed for reshape fusion.
(2) Change versions type to vector.
(3) Add logging.
(4) Return false when multiple output edges matched in FindPath. Add comments.
* only allow one direction (input or output) in FindPath
* [NupharEP] Update notebook and docker image (#2416)
Add BERT squad in Nuphar tutorial
Enhance speed comparsion readability
* Fix the issue in matmul_add_fusion (#2407)
Fix the issue in matmul_add_fusion
If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output.
Fix: just remove the shape reset to not fuse it.
Add a negative test case for matmul+add fusion
* feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389)
Updates the `TreeEnsembleRegressor` to allow for `double`, `float`,
`int64`, and `int32` inputs to match the upstream specification.
Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com>
* onnxrt server documentation update (#2396)
* Added support for Pad-2 operator in OpenVINO-EP (#2405)
* Add CUDA If operator. (#2377)
* Add CUDA If operator.
Uses CPU operator for implementation.
By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node.
* Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check.
* Fix the type constraints on CUDA If operator to exclude strings. (#2431)
* add Im2col<uint8_t> (#2438)
* Adjust codegen vectorization width from target (#2439)
* Adjust codegen vectorization width from target
* Add CUDA Scan operator. (#2403)
* Add Scan CUDA op.
Uses CPU implementation for logic.
Added some device specific functors for handling when data needs to be manipulated on a different device.
Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling.
* Fix Windows GPU C API packaging pipeline failure (#2440)
Fix Windows GPU C API packaging pipeline failure (#2440)
* Correctly handle implicit inputs for fused nodes (#2390)
* Correctly handle implicit inputs for fused nodes
Previously, nuphar's partitioning function didn't include
node's implicit inputs into the inputs list of MetaDef, and hence
a crash was triggered in the onnx graph checker.
This commit fixed the issue. Furthermore, it also fixed a related
issue where we didn't add implicit inputs into
graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs.
the issue was that graph_inputs_including_initializers_ populated by
SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain
implicit inputs which were not of any node's initializers in the graph.
Because they were not part of any initializers, these implicit inputs
couldn't be visited by going through all nodes' inputs.
Consequently, they would *not* be added into graph_inputs_excluding_initializers_.
We fixed the issue by first copying the populated graph_inputs_including_initializers_
into graph_inputs_excluding_initalizers_, which then had both initializers and
non-initializers as its initial content. Later, we erase initializers from the
list. In this way, we can ensure all implicit inputs to remain in
graph_inputs_excluding_initializers_.
* refined comments and fixed duplicates
Address CR by revisiting comments in terms of implicit inputs
Also fixed an issue by skipping duplicates while copying inputs
from graph_inputs_including_initializers_.
* address CR
explain why we need to collect nodes' implicit inputs
* don't rely on pointer values for iterating std::set
Previously, openvino relied on iterating a set of NodeArg pointers
to construct inputs and outputs for a fused graph. It could cause
non-determinism. The reason was that although iterating std::set by
itself is stable, pointer values of NodeArgs may vary. Consequently,
we could end up visiting the set's elements in different orders for
different runs for the same test, which resulted in constructing
inputs (and outputs) with different orders to the fused graph.
For example, for the same test, we may have inputs [A, B] in some
runs but inputs[B, A] in others.
Let's use std::string as the key type to avoid such nondeterminism.
This commit also added implicit inputs into meta->inputs while returning
the capability from the openvino provider.
* Fixed another latent issue in openvino's GetCapability function
The issue was that we couldn't simply erase fused_inputs and fused_outputs
while iterating the nodes. For example, an output NodeArg may have multiple
uses, and it's wrong if we erase it from fused_outputs when we encounter only
one of its uses as input.
* Remove DeviceAllocatorRegistry class (#2451)
Remove DeviceAllocatorRegistry class
* CSharp api and test for loading custom op shared library (#2420)
- Added C-API test for loading custom op shared lib.
- Made some changes in C++ api header and C-api implementation to get it working.
- Added C# API and corresponding test for loading custom op shared library.
* Parallel Gelu with ParallelFor (#2399)
Parallel Gelu to get better performance for Gelu
* Clean up build.py (#2446)
* Pull the latest image before running docker build
* Fuse SkipLayerNorm with Bias (#2453)
Fuse SkipLayerNorm with Bias
* Allow more than one invocation of CreateEnv in the same process. (#2467)
* Allow more than one invocation of CreateEnv in the same process.
* Fix centos build
* Symbolic shape inference improvements: (#2460)
* Symbolic shape inference improvements:
- add a mode to guess unknown ops' output rank
- add support for GatherND
- add support for If
- fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data
- add symbol to literal merge when ONNX silently merges dims
- fix a bug in Concat when input dim is 0
- fix a bug in ConstantOfShape that computed dim is not updated
- add support for dynamic shape in ConstantOfShape
- fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0
- add support for dynamic padding in Pad
- add support for dynamic shape in Reshape
- add support for Resize with opset > 10, by treating output dims as dynamic
- fix a bug in Slice when starts/ends are dynamic
- restrict input model to opset 7 and above
- make output model optional to avoid disk write when testing
Run model tests for symbolic shape inference
Reduce 2GB docker image size of nuphar
* add additional test data set for nuget pipeline (#2448)
* add SAS token to download internal test data for nuget pipeline
* update azure endpoint
* fix keyvault download step
* fix variable declaration for secret group
* fix indentation
* fix yaml syntax for variables
* fix setting secrets for script
* fix env synctax
* Fix macos pipeline
* attempt to add secrets to windows download data
* fix mac and win data download
* fix windows data download
* update test data set url and location
* Revert "Brianma/windowsai fi (#2475)"
This reverts commit 5780b864a1.
* Add scenario tests (#2457)
* Add scenario tests
* Remove TODO from model license
* Add winml_api test dependency
* fix model load test. fi from master changed the constructor (#2483)
* make api tests all pass (#2486)
* fix bad merge
* fix bad model merge
* Layer dev paulm (#2492)
* commetns for dml graph transformer
fixed ort value passing using the allocatir info
* fixed and coded maps and sequences across the abi
* Rename ambiguous header (#2489)
* fix one more missing IR version model (#2500)
* add missing IR version to 4 more models used by scenario tests (#2501)
* Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479)
* Support test parameters through CLI arguments
* Add WinML do Windows x86/ARM CI builds
* Code style fixes
* Update googletest
Remove GPUTEST macros everywhere now that GTEST_SKIP is supported
* Refactor main.cpp
* Build scenario tests without DML
* Link scenario tests to DML when it's enabled (#2502)
* Layer dev release pipeline (#2488)
Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP.
* Layer dev paulm (#2506)
* commetns for dml graph transformer
fixed ort value passing using the allocatir info
* fixed and coded maps and sequences across the abi
* cleaned up w4's
cleaned up the model info ABI
delayload directml.dll from winml
* Remove usage of IOBinding in WinML and use C_API Run method (#2504)
* remove usage of iobinding
* Change data structure to use vector of Ort::Values
* Polish bind input / output
* Use C APIrun method
* Update providers on evaluate getresults
* Remove run and IObinding interface from WinMLAdapter
* Remove use of IObinding
* bind unbound outputs code moved to learningmodelbinding
* clean up unneeded istensor adapter function
* Fix comment
* Check if session is closed before binding and clearing
* PR feedback
* Layer dev paulm (#2507)
* commetns for dml graph transformer
fixed ort value passing using the allocatir info
* fixed and coded maps and sequences across the abi
* cleaned up w4's
cleaned up the model info ABI
delayload directml.dll from winml
* cleaned up namepsace aliases.
renamed _winmla to winmla
this was good PR feedback from tiago a while back.
* Make tests dependend on winml_dll (#2509)
* add dml binaries to DirectML package and be more explicit about condition variables (#2520)
* re-enable warnings for winml builds and fix the warnings that were hiding (#2526)
* turn devmode back on for winml builds
* fix some warnings. include protobuf in a way that disables some warnings
* undo protobufhelpers changes and just ignore 4100 errors in pb code
* attempt to isolate protobufhelpers errors
* add template specialization for getting tensor proto data
* Layer dev paulm (#2533)
* commetns for dml graph transformer
fixed ort value passing using the allocatir info
* fixed and coded maps and sequences across the abi
* cleaned up w4's
cleaned up the model info ABI
delayload directml.dll from winml
* cleaned up namepsace aliases.
renamed _winmla to winmla
this was good PR feedback from tiago a while back.
* moved files from inc to lib\api.core
cleaned up some of the cmake
* staged changes
* Spawn child process to run DeviceLostRecovery scenario test (#2530)
* Spawn child process to run DeviceLostRecovery scenario test
* Layer dev paulm (#2536)
ori said yes
* add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542)
* Handle exception thrown from all apis in WinMLAdapter (#2539)
* various changes to unblock windowsai ADO build
* Fix custom ops scenario tests (#2562)
* Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed
* comment typo
* pr comment about calling phoenix singleton
* Make lotus_environment static in winmladapter
* Layer dev paulm (#2567)
* commetns for dml graph transformer
fixed ort value passing using the allocatir info
* fixed and coded maps and sequences across the abi
* cleaned up w4's
cleaned up the model info ABI
delayload directml.dll from winml
* cleaned up namepsace aliases.
renamed _winmla to winmla
this was good PR feedback from tiago a while back.
* moved files from inc to lib\api.core
cleaned up some of the cmake
* staged changes
* making windowsAI azure dev ops work.
* code review comments.
* revert changes
* Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK
* Layer dev dml delayload (#2580)
* Brianma/cpu (#2583)
* don't include dml stuff in cpu builds
* tests that link the image lib also need the telemetry lib now
* Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589)
* Throw Winml_err_invalid_binding if binding gpu resource on cpu device
* PR comments. No need to query executionprovider for is gpu device
* User/xianz/ortthrow (#2596)
* thrown and handle onnxruntime exceptions
* handle exception thrown from ort in winmladapter
* undo changes in error.h
* add message to HRESULT
* User/xianz/ortthrow (#2599)
* thrown and handle onnxruntime exceptions
* handle exception thrown from ort in winmladapter
* undo changes in error.h
* add message to HRESULT
* add status error message
* Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630)
* User/xianz/dedup telemetry (#2631)
* investigate duplication of telemetry in winml and ort
* remove winml telemetry events
* telemetry executionProviderEvent
* remove unneccessary file and refactor code little bit
* Revert back TelemetryEvent, which send up ETW event.
* merge changes from layer_dev to windowsai (#2638)
* Remove underscore from googletest names (#2616)
* Fix leaking memory allocator
Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761
and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198
* Explicitly initialize Ort::Value with nullptr
* Cache WinML adapter
* bad merge
* define private version of dxcore enum that is added in 19H1 SDK. (#2654)
* add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672)
* do not package directml.pdb for redist packages. (#2676)
* Fix leaking operator registry (#2645)
Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916
* User/orilevari/windowsai master merge (#2674)
merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer
* Delete Ort Allocator in LearningModelBinding (#2653)
* Delete OrtAllocator in LearningModelBinding
* PR comments to make Ort::Allocator a smart pointer
* Small comment change
* PR feedback to clean up code
* PR feedback on move semantics
* Clean up std::move
* Fix memory leaks (#2679)
Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109,
https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and
https://microsoft.visualstudio.com/OS/_workitems/edit/24388596
* various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695)
* Windowsai without fi (#2701)
* Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529)
* Setup java ci (#2528)
* Add provision in ORT for session options to be parsed when available via model file (#2449)
* Initial commit
* Fix gitmodules
* Nits
* Nits
* Updates
* Update
* More changes
* Updates
* Update
* Some updates
* More changes
* Update
* Update
* Merge
* Update
* Updates
* More changes
* Update
* Fix nits
* Updates
* Fix warning
* Fix build
* Add comment
* PR feedback
* PR feedback
* Updates
* Updates
* Update
* More changes
* Fix build break
* Comment test for now
* Updates
* Updates
* PR feedback
* Updates
* Nits
* Add tests
* Fix build
* Fix build
* Fix build
* Fix build break
* Fix build
* Nits
* PR feedback
* More change
* Expose GetSessionOptions in pybind logic and add unit test for python
* Fix build
* PR feedback
* PR feedback
* Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535)
This reverts commit 7c7d5a149c.
* Add dynamic shape support in TensorRT execution provider (#2450)
* remove onnx-tensorrt submodule
* add new onnx-tensorrt submodule (experiment) for trt6
* update engine build for trt6
* update compile and compute for tensorrt6.0
* Update tensorrt_execution_provider.cc
* Update tensorrt_execution_provider.cc
* Update tensorrt_execution_provider.cc
* Update tensorrt_execution_provider.cc
* switch to onnx-tensorrt master for TensorRT6'
* Update tensorrt_execution_provider.cc
* Handle dynamic batch size and add memcpy in TensorRT EP
* update test cases
* Update tensorrt_execution_provider.cc
* update onnx-tensorrt submodule
* Update Dockerfile.ubuntu_tensorrt
* Update Dockerfile.ubuntu_tensorrt
* Update run_dockerbuild.sh
* Update run_dockerbuild.sh
* Update install_ubuntu.sh
* Update concat_op_test.cc
* Update tensorrt_execution_provider.cc
* Upgrade TensorRT to version 6.0.1.5
* Update onnxruntime_providers.cmake
* Update CMakeLists.txt
* Update reduction_ops_test.cc
* Update install_ubuntu.sh
* Update Dockerfile.ubuntu_tensorrt
* Update Dockerfile.tensorrt
* Update BUILD.md
* Update run_dockerbuild.sh
* Update install_ubuntu.sh
* Update onnxruntime_providers.cmake
* Update install_ubuntu.sh
* Update install_ubuntu.sh
* Update gemm_test.cc
* Update gather_op_test.cc
* Update CMakeLists.txt
* Removed submodule
* update onnx-tensorrt submodule
* update header file
* Removed submodule
* add submodule onnx-tensorrt kevin's branch shape-test'
* add debugging code
* Update tensorrt_execution_provider.cc
* Update tensorrt_execution_provider.cc
* merge master
* Removed submodule
* update onnx-tensorrt submodule
* add more changes for dynamic shapes
* Update tensorrt_execution_provider.cc
* update for dynamic shape
* update dynamic shape processing
* fix logger issue
* remove submodule onnx-tensorrt
* add submodule onnx-tensorrt
* add env variable min_subgraph_size
* remove redundency
* update document
* use onnxruntime::make_unique
* fix multi-run issue
* remove some tests to save CI build time
* Add dynamic shape test
* Update TensorRT-ExecutionProvider.md
* Add example of running Faster R-CNN model on TensorRT EP
* Add more details on env variables
* update environment variables
* Update tensorrt_basic_test.cc
* Update model tests
* Update tensor_op_test.cc
* remove --use_full_protobuf
* Update build.py
* User/xianz/telemetry (#2458)
* enabme telemetry
* enable telemetry
* set enable telemetry as default
* for debugging
* remove log and set disable telemetry as default back
* delete private file while testing
* resolve comment: mainly add license header, rename macro and update docs
* rewording in privacy.md
* Fix integer overflow in cuda NonMaxSuppression implementation (#2540)
* add test case that should pass but fail
* fix nms
* extract int_max_output_boxes_per_class
* Introduce container type runtime checks and other improvements (#2522)
Rework TensorSeq in a manner consistent with Tensor and SparseTensor
in terms of type system setup.
Reduce templating. Introduce helpers to ensure the same
data type.
Make OrtValue __dtor not virtual.
Introduce ContainerChecker
* Fix C API tests for centos and mac (#2544)
* change c++14 to c++11
* add ld lib path for centos
* enable csharp tests on macos
* fix C API test on MacOS + fix manylinux dotnet install
* fix manylinux dotnet install
* fix lib link
* Add back executable bit to build.py
* Fix a bug handling negative begin pad values in Pad op (#2550)
* Fix bug in Pad op
* Update
* DNNL CMAKE update (#2548)
* Fix android build (#2558)
* Update win-x86-ci.yml (#2557)
Fix build pipeline break
* Re-enable Windows C# tests (#2564)
* disable onnx_test_runner -x invocations for dnnl (#2568)
* Allow sequence length to be symbolic (#2559)
* setup java ci mac (#2570)
* make layernorm fusion to support opset 11 (#2545)
* Fix a warning found in the latest VS release
* Add more check on SkipLayerNorm and BiasGelu fusion (#2574)
* Fix file not found error during docker build. (#2569)
* Add ConvTranspose1D (#2578)
* Ryanunderhill/packagename test (#2582)
* [Nuphar EP] fixes for some object detection models (#2581)
Update notebook tutorial with multi-threaded int8 GEMM from #2517
* EmbedLayerNormalization Fusion Improvement (#2553)
Embedding layer norm fusion improvements - add more checks
* Update version (#2584)
* Temporarily exclude vgg19 test from Python backend test
1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588)
2. Update docker file to decrease the docker image size
* Update docs for Android NNAPI EP (#2586)
* Fix lto bug for protobuf and ubuntu
* add path to build dir before test run (#2590)
* Add missig env variables for mac pipeline test (#2595)
* Fixed an issue in updating realized dims (#2597)
when we update realized dims for scan's output, the sliced axis also
needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis",
because the offset in the symbols are based on Scan sugraph.
Otherwise, we would end up with shape mismatch later.
* Java API for onnxruntime (#2215)
* Add support for opset 11 in reshape fusion (#2592)
Support opset verion 11 in reshape fusion
* Rename automl python tools folder to featurizer_ops. (#2593)
* Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605)
Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer.
* symbolic shape inference: fix warnings in GPT-2 model (#2608)
And revise nuphar perf test on BERT squad
* Dump subgraph ID and fused graph ID (#2607)
* Dump subgraph ID and fused graph ID
Dump subgraph ID and fused graph ID for better debugging
* Remove local static fused_count
added a field global_fused_count_ to NupharExecutionProvider class
* EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613)
Support subgraph of SQuAD model exported from pytorch with dynamic input axes
* Allow providers to be set for InferenceSession at construction (#2606)
* Remove unnecessary parameter in some places in GatherElements implementation (#2612)
* Remove unnecessary parameter in some places
* Update
* Update
* Make sure fenced tensor could not reuse other tensor. (#2561)
Fix random error caused by this.
* Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621)
* fix float16 comparison in initializer (#2629)
* epsilon attribute for layernormalization fusion (#2639)
* removed unnecessary batch file and fix path (#2640)
* Add shape inference to ConvTransposeWithDynamicPads schema (#2632)
* Improve cuda expand() opeator's performance. (#2624)
* Cuda pad optimize when no padding is needed. (#2625)
* Shortcut cuda Pad() when no padding is needed.
* Optimize cuda scatter() on 2D compatible. (#2628)
* Optimize cuda scatter() on 2D compatible.
* Add some comments.
* fix build error for ARM (#2648)
* Improve performance of resize() in Nearest mode (#2626)
Special treatment for 2D, check same size as input image.
And in 2d kernel, template use_expolation.
* Fix memory exception in Layer Norm Fusion (#2644)
* Windows CI changes(#2650)
* Revert "User/orilevari/windowsai master merge (#2674)"
This reverts commit fe26146311.
* Revert "Windowsai without fi (#2701)"
This reverts commit 285d4c85ff.
* Revert "User/orilevari/windowsai master merge (#2674)"
This reverts commit fe26146311.
* Deref unique pointer for session_state
* send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704)
* send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events.
* Add EvalutationStart Event
* add comment
* use correct type for for loop (#2755)
* ARM CI (#2759)
* Set ARM agent pool
* Set CMake generator to VS 2019 in ARM
* Use system-wide CMake instead of custom version
Our custom version is too old for VS 2019
* Use DML and build shared lib in ARM CI
* Restore nuget packages in ARM CI
* Disable DML
* Refactor ARM debug/release builds
* Use system packaged Python version
* Remove hardcoded Python path
* Downgrade Python to 3.7 for build
* Remove explicit CMake path
* Fix invalid JSON in cgmanifest.json (#2760)
* Fix cgmanifest.json generating script (#2770)
* Fix protobuf submodule name
* Workaround pygit2 bug
* Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726)
* Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory
* PR feedback, no need for dll(export) since using def file
* PR comments
* Small comment in gen_def.py
* User/orilevari/32bit comparison warning (#2800)
* use correct type for for loop
* explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276.
* Move winml_provider_factory.h to proper location (#2801)
* Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809)
* Add winml macro wrappers on top of google test macros
* change test methods to disabled
* Add custom winml macros for both taef and google tests
* PR comments
* Filter CPU case for IsFloat16Supported (#2802)
* Merge fixes
* CMake cross-generator fixes (#2790)
* Fix compilation w/ non-VS CMake generators
* Fix custom WINMD target in Ninja
* Remove usage of msbuild .targets file
* Fix linking using DML in Ninja
* Automate SDK kit version choice
* Cleanup DML package install
* Fix SDK version detection
* Fix comment
* Revert unittest linkage changes
* Fix latest SDK detection
* Don't link to non-uapcore libraries
* Remove MessageBoxA reference and unused link libs
* Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829)
* Add winml macro wrappers on top of google test macros
* change test methods to disabled
* Add custom winml macros for both taef and google tests
* PR comments
* Refactor winml api tests
* Move additional gtest specific macro definition into googleTestMacros.h
* Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837)
* Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841)
* Fix warnings that cause build to fail
* Fix test warnings and delayload linking (#2843)
* Ortmemoryinfo struct changed
* mark the camera scenario test as edgecore because it uses d3d11 (#2852)
* User/orilevari/pipeline fi breaks (#2853)
* remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines.
* change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev
* Remove internal libs from tests (#2864)
* Support custom DML in onnxruntime_providers.cmake (#2867)
* Make DML include path global (#2882)
* Make DML include path global
* Add generated cppwinrt headers to winml_lib_common
* Integrate changes to WindowsAI to make ADO Build (#2886)
* Revert "CMake cross-generator fixes (#2790)"
This reverts commit dbe7d97fa1.
* add additional suppress warning in onnx_proto
* ignore /wd4996 warning
* DML execution provider fixes
* Revert "Revert "CMake cross-generator fixes (#2790)""
This reverts commit 1ae7b4bcbc.
* Update func signature of custom op function overloads
* common devicehelpers fixes
* Add pch.h for winml_lib_common
* re-add winml_lib_common_dir/inc to include path for winml_adapter
* User/orilevari/dml redist shared folder (#2890)
* move dml nuget package directory up one level to make it shared between build flavors
* Merge conflict fix
* Revert "Merge conflict fix"
This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c.
* Revert "Merge remote-tracking branch 'origin/master' into windowsai"
This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing
changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6.
* Make winml_test_common free of test macros (#2902)
* Add option to build winml_test_common without googletest specifics
* remove test macros from squeezenet
* comment change
* Make cmake functions to get scenario and api source
* PRcomments about hresult
* Build errors fixed
* Fix cmake variable
* Make winml_google_test_lib to build main.cpp once
* PRcomments
* Don't generate files outside the build root (#2914)
* Don't generate files outside the build root
* Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML
* Add DML depedency on RESTORE_PACKAGES
* User/orilevari/fix yaml merge bugs (#2918)
* Add winml test source parameter into cmake function (#2919)
* Add option to build winml_test_common without googletest specifics
* remove test macros from squeezenet
* comment change
* Make cmake functions to get scenario and api source
* PRcomments about hresult
* Build errors fixed
* Fix cmake variable
* Make winml_google_test_lib to build main.cpp once
* PRcomments
* Add arguments to unittest cmake functions
* remove comment
* Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai""
This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26.
* Fix breaks from merge with ORT master
* Brianma/linux (#2917)
* don't include windows.h in cross-plat header
* add default case for switch statement
* signed/unsigned mismatch fix
Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com>
* User/sheilk/winml adapter c api (#2891)
* Create winml adapter c api
* fix build
* make it build
* move adapter into onnxruntime core/session
* entry point not exported
* minor changes
* make model metadata work
* make tests pass
* implement all the model reflection apis on the adapter c abi
* update the new ort interface to create a lotus ennvironment with a logging sink
* start adding ort env
* move all winml code into adapter folder/lib to isolate it
* ensure a single logging manager at a time
* start refactoring session
* refactor session creation interface
* add cpu and dml session option methods to adapter
* finish session init
* stub out interfaces in ort lib to perform similar mechanics of iinference session
* enable profiling, and enable schema override
* update session register graph transformers
* turn back on custom registry for custom ops
* Add sync api
* add last c api stubs
* should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue
* remove ep adapter header
* Implement DML execution provider functions from adapter (#2846)
* Implement DML execution provider functions from adapter
* Use functions in OnnxruntimeEngine.cpp
* make map/sequence type_infos freeable, and start implementing ivalue
* make it build again
* implement value methods
* implement remaining methods
* remove com adapter abi
* check dml session
* cache the allocator on ivalue
* check if resource is cpu/gpu when access its mutable data
* update tensor
* mismatched parentheses
* fix tensor base and binding obj
* it evaluates tensors! sometimes...
* minor fixes
* enable gpu evals
* wrapper all existing winml adapter apis with API_IMPL to try catch (#2854)
* update winml... tensor strings are broken, need to template tensorbase to do different things for strings
* make tensor strings work with 2 copies in/2 copies out
* Fix tensor string and allocator bug
* make maps work again... needs some fixes still
* Make it build!
* enable map inputs
* map outputs
* unbound outputs for sequences and maps
* User/xianz/merge windowsai (#2883)
* Packaging pipeline changes for VS 2019 (#2711)
* Tiny fix to codegen
* Simplify cache implementation and avoid static variables that may carry over between models
* Extend DML kernels (#2641)
* Additional DML operators
* Check unsupported attributes and inputs
* Address PR comments
* Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range
* Fix test failures
* Build fix
* PR comments
* Update Nuphar tutorial notebook (#2721)
1. Reflect int8 GEMV improvements for multi-threading from #2696
2. Add notes on multi-threading control using OpenMP
3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2
4. Add rnn_benchmark example to resolve#1993
* Add schema for new Qops (#2611)
* Add schema for new Qops
* adding shape inference + qlinearaveragepool
* plus review comments
* plus review comments
* updates per review comments
* plus review comments
* [server] Add supposed for model_name and model_version as cli parameter (#2708)
* remove 64bit warning message from python validation. (#2727)
* MLAS: ARM64 build fix (#2734)
fix bad usage of vreinterpret to cast vector element types
* Fix broken python docs links (#2740)
* Fix build on Mac OS (#2731)
mac os ld doesn't support --while-archive, correct option is -all_load
* fix ngraph wheel (#2737)
* fix ngraph wheel
1.1.0 onnxruntime_ngraph wheel doesn't work
* remove libdnnl.so in nGraph Libs
* make it easy to compare
* Split onnxruntime server to a separated folder (#2744)
* Fix build for Python 3.8 (#2747)
* Fix build for Python 3.8
* Update protobuf to 3.11.2 (#1928)
Update protobuf to 3.11.2 (#1928)
* Change default optimization level to All (from Basic) (#2745)
* change default optimization level to All (from Basic)
* fix test
* fix c# test
* Update numpy to 1.18 (#2758)
* Update numpy to 1.18
* Pipeline changes for python 3.8 (#2753)
1. Pipeline changes for python 3.8
2. Fix a regression in setup.py which was just introduced in the previous commit.
Please notice, we still haven't made python 3.8 + Windows + CUDA work.
* Add basic stacktrace output for posix debug builds. (#2749)
* [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772)
* Revert "Change default optimization level to All (from Basic) (#2745)"
This reverts commit 56bb503c2f.
* Fix typo in error message (#2736)
* Rename MKL-DNN to DNNL to fix broken link (#2730)
* Fix nightly build version number issue
* Pass BUILD_BUILDNUMBER to linux docker
* Disable featurizers in python packages
* Import more featurizers (#2781)
Make kernels non-template. Add input constraint for learnt data.
Add min_max_scalar_transformer, robust_scalar_transformer,
inputation_marker_transfomer, label_encoder_transformer,
missing_dummies_transformer along with tests.
Advance Featurizers library commit.
* Implement a more stable softmax (#2715)
* Implement a more stable SoftMax
e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough.
A math transform as below is leveraged to get a stable softmax:
e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max))
And for convenience, force max to 0.f if all xi are negative
* Contributing: Fix a typo (#2784)
* ACL EP GEMM improvements (#2780)
When it is posible we use a fully connected layer instead of the gemm implementation.
This will let the library use the best implementation based on the input data.
* ACL EP convolution improvements (#2774)
Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05.
Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead.
* Add script for release Nuget validation (#2719)
* Initial commit
* Nits
* Disable a test temporarily
* Change working directory
* Test
* Add download python step
* Test update
* More changes
* Fix space issue
* Fix
* Verify nuget signing
* Fix
* Spaces
* PR feedback
* Nit
* Fix
* Fix
* Remove temporary changes
* add uint8 support to where op (#2792)
* Improve bert optimization script: (#2712)
(1) Move input int64=>int32 conversion to embed layer fusion.
(2) Output epsilon attribute for LayerNormalization fusion.
* add session creation time cost. (#2798)
* ML.NET team needs featurizers within a package (#2789)
Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines.
* Initialize max of softmax with lowest of float (#2786)
* MLAS: update SGEMM threading parameters (#2808)
* add interface to copy batch tensors. (#2807)
* add interface to copy batch tensors.
* onnxruntime
* speed up Windows TRT CI (#2811)
* don't run cuda tests if building with tensorrt
* remove unnecessary build options for win trt ci
* refactor win gpu tensorrt ci yml
* --numpy_version=1.17
* update
* update
* azcopy and cuda path
* Update test data (#2356)
* Add timeseries imputer transformer featurizer kernel (#2813)
Make kernels non-template. Add input constraint for learnt data.
Fixup tests.
Add two more featurizers along with tests. Tests fail.
min_max_scalar_transformer
robust_scalar_transformer
Fix tests serialized stream by prepending version bytes.
Add inputation_marker_transfomer and the test.
Fix up float/double type designations.
Added label_encoder_transformer along with a test.
string_throw case is broken at the momement.
Fix labelencodertransfomer_test.cc string_throw case
Rename maxabsscalertransformer_test.cc
Add MissingDummiesTransformer along with the test.
Update manifest.
Add TimeSeriesImputerTransformer definition, implementation and tests
* Fix memory leak in TRT (#2815)
* fix memory leak issue
* revert EP_FAIL on enueueV2
* Add manifest missing comma
* Run static code analyzer on most of our code (#2817)
* Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809)
* Add winml macro wrappers on top of google test macros
* change test methods to disabled
* Add custom winml macros for both taef and google tests
* PR comments
* update quantization doc (#2783)
* update documentation for quantization script
* plus some spell corrections
* Filter CPU case for IsFloat16Supported (#2802)
* update default optimization level + fix gemm_activation fusion (#2791)
* update defualt optimization level + fix gemm_activation fusion
* fix typo
* add unit test and incorporate review comments
* fix test comment
* Fix dnnl wheel package name (#2823)
* Append '-dnnl' to whl package name when --use_dnnl
* Update build.py
* Update Ubuntu & TensorRT version in README (#2820)
Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to
https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09
* Merge fixes
* Add OneHotEncoder and HashOneHotEncoder kernels. (#2830)
Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test.
Add OneHotEncoder kernel test.
Add HashOneHotVectorizerTransformer unit test.
This does not link due to multiple definitions of functions
that are included into header from a CPP file.
* Upgrade gtest to the latest version (#2827)
WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version.
However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work.
The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213
* Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833)
* Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768)
* Remove allocator type from the key comparison in ExecutionProviders.
Remove usage of DummyArena as it's no longer necessary.
* Fix x86 tests where arena allocator is disabled.
Make initialization of OrtMemoryInfo clearer by adding Invalid enum value.
* Make OrtValueNameIdxMap::MaxIdx more intuitive.
* Convert ExternalProject Featurizers into git submodule (#2834)
Add git submodule for Featurizer library.
Update cmake to build for git submodule.
* add domain check for nodes + update documentation (#2831)
* Fix cgmanifest.json generating script (#2770)
* Fix protobuf submodule name
* Workaround pygit2 bug
* User/orilevari/32bit comparison warning (#2800)
* use correct type for for loop
* explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276.
* CMake cross-generator fixes (#2790)
* Fix compilation w/ non-VS CMake generators
* Fix custom WINMD target in Ninja
* Remove usage of msbuild .targets file
* Fix linking using DML in Ninja
* Automate SDK kit version choice
* Cleanup DML package install
* Fix SDK version detection
* Fix comment
* Revert unittest linkage changes
* Fix latest SDK detection
* Don't link to non-uapcore libraries
* Remove MessageBoxA reference and unused link libs
* Fix Linux CUDA nuget packaging pipeline break
* Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829)
* Add winml macro wrappers on top of google test macros
* change test methods to disabled
* Add custom winml macros for both taef and google tests
* PR comments
* Refactor winml api tests
* Move additional gtest specific macro definition into googleTestMacros.h
* Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837)
* Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841)
* update optimization doc for BERT related fusions (#2819)
* Add bert related transformers to doc
* Add execution provider and comment for bert optimizations
* Add comment about accuracy impact of approximation
* Fix warnings that cause build to fail
* MLAS: enable threading for quantized GEMMs (#2844)
* Fix test warnings and delayload linking (#2843)
* Ortmemoryinfo struct changed
* mark the camera scenario test as edgecore because it uses d3d11 (#2852)
* User/orilevari/pipeline fi breaks (#2853)
* remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines.
* change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev
* Remove internal libs from tests (#2864)
* Support custom DML in onnxruntime_providers.cmake (#2867)
* remove old winmladapter cpp
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: KeDengMS <kedeng@microsoft.com>
Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com>
Co-authored-by: Ashwini Khade <askhade@microsoft.com>
Co-authored-by: Andrey <andrey.lompart@gmail.com>
Co-authored-by: George Wu <jywu@microsoft.com>
Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com>
Co-authored-by: Faith Xu <txsafx@gmail.com>
Co-authored-by: zhanyi-ms <zhanyi@microsoft.com>
Co-authored-by: Changyoung Koh <gkcy1019@gmail.com>
Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>
Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com>
Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com>
Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>
Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com>
Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com>
Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com>
Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com>
Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
Co-authored-by: Ke Zhang <kezhan@microsoft.com>
Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com>
Co-authored-by: Ryan Lai <ryalai96@gmail.com>
Co-authored-by: Ori Levari <ori.levari@microsoft.com>
Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com>
Co-authored-by: Qing <cwq1913@gmail.com>
Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com>
* move sequence implementation into ort lib... still commented out... need to turn back on...
* begin sequence implementation
* make maps and sequences work
* fix broken tests
* remove dead code
* misc cleanup
* CR feedback
* User/xianz/winml adapter c api (#2869)
* wrapper all existing winml adapter apis with API_IMPL to try catch
* Return HR or Throw for WinML adapter APIs if failed
* undo macro wrapper for two places
* Wrap error macros around ort apis, too.
* address CR feedback #2
* add more api throw/return macros
* Revert changes no longer needed
* revert changes to cxx api
* format winml lib.ort and winml adapter
* remove static pheonix singleton
Co-authored-by: Ryan Lai <ryalai96@gmail.com>
Co-authored-by: Xiang Zhang <xianz@microsoft.com>
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: KeDengMS <kedeng@microsoft.com>
Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com>
Co-authored-by: Ashwini Khade <askhade@microsoft.com>
Co-authored-by: Andrey <andrey.lompart@gmail.com>
Co-authored-by: George Wu <jywu@microsoft.com>
Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com>
Co-authored-by: Faith Xu <txsafx@gmail.com>
Co-authored-by: zhanyi-ms <zhanyi@microsoft.com>
Co-authored-by: Changyoung Koh <gkcy1019@gmail.com>
Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>
Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com>
Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com>
Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>
Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com>
Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com>
Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com>
Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com>
Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
Co-authored-by: Ke Zhang <kezhan@microsoft.com>
Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com>
Co-authored-by: Ori Levari <ori.levari@microsoft.com>
Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com>
Co-authored-by: Qing <cwq1913@gmail.com>
Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com>
* missing use_dml check in winml_adapter_session (#2930)
* --use_dnnl flag was mangled in merge (#2931)
* use dml macro not wrapping custom registry code (#2934)
* Disable LNK4199 winml_dll to enable cuda builds (#2936)
* Disable LNK4199 in winml_dll
* linkler->linker
* LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937)
* Disable LNK4199 in winml_dll
* linkler->linker
* Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16
* Publish build symbols (#2939)
* Publish build symbols
* Don't upload PDBs for .exe files
* Make x86 build (#2943)
* fix last remaining size_t/int64_t warnings->errors (#2948)
* TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952)
* fix tensor string allcoator
* clean up default allocator usage for strings in winml lib/api.ort
Co-authored-by: Ryan Lai <ryalai96@gmail.com>
* Handle tensor shape of zero (#2954)
Co-authored-by: Ryan Lai <ryalai96@gmail.com>
* CR feedback (#2970)
* CR feedback
* fix weird formatting on privacy readme
* Add 'All rights reserved.' everywhere
* readd all rights reserved to winml_provider_factory.h
* remove extra space in comment
* remove extra whitespace
* fixes post master merge
* remove winml from nuget gpu pipeline
* set IR VERSION on generated_model in rnn_benchmark (#2972)
* Fix slice conformance failures (#2908)
Co-authored-by: Adrian Tsai <adtsai@microsoft.com>
Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com>
Co-authored-by: Ryan Lai <ryalai96@gmail.com>
Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com>
Co-authored-by: Xiang Zhang <xianz@microsoft.com>
Co-authored-by: Dwayne Robinson <fdwr@hotmail.com>
Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com>
Co-authored-by: Ori Levari <ori.levari@microsoft.com>
Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com>
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: KeDengMS <kedeng@microsoft.com>
Co-authored-by: Ashwini Khade <askhade@microsoft.com>
Co-authored-by: Andrey <andrey.lompart@gmail.com>
Co-authored-by: George Wu <jywu@microsoft.com>
Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com>
Co-authored-by: Faith Xu <txsafx@gmail.com>
Co-authored-by: zhanyi-ms <zhanyi@microsoft.com>
Co-authored-by: Changyoung Koh <gkcy1019@gmail.com>
Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>
Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com>
Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com>
Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>
Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com>
Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com>
Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com>
Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com>
Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
Co-authored-by: Ke Zhang <kezhan@microsoft.com>
Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com>
Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com>
Co-authored-by: Qing <cwq1913@gmail.com>
Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
1. Add support for vstest.
2. Add support for vcpkg. To use it:
```bat
vcpkg install zlib:x64-windows benchmark:x64-windows gtest:x64-windows protobuf:x64-windows pybind11:x64-windows re2:x64-windows
mkdir build
cmake ..\cmake -DCMAKE_BUILD_TYPE=Debug -A x64 -T host=x64 -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -Donnxruntime_PREFER_SYSTEM_LIB=ON
```
3. New cmake option: onnxruntime_PREFER_SYSTEM_LIB, which allows user using the preinstall libs instead of the things in onnxruntime submodule.
4. New cmake option: onnxruntime_ENABLE_MEMLEAK_CHECKER, which allows user turn on/off the memory leak checker by @RyanUnderhill in Windows Debug Build. The checker doesn't work with vstest.
4. Fix the post merge pipeline(Mainly for test coverage report).
5. Ignore the compile warning from the Featurizer library code
6. Apply "/utf-8" VC compile flag to our code. Without this, you can't build onnxruntime on Chinese Windows.
7. Remove the SingleUnitTestProject cmake option because it's deprecated more than one year and nobody is using it.
8. Move opaque api tests to onnxruntime_test_all
9. Enable "/W4" on CUDA ep's C++ code(Not the *.cu files), and fix some warnings, add some extra checks.
10. Delete the onnxruntime::test::TestEnvironment class.
11. Add a DLLmain for onnxruntime.dll.
12. Allow dynamic link to libprotobuf
* Documentation for API Breaking Changes
* Add version 2 of the API, plus update documentation
* Add a static assert to ensure version 1 of API we shipped does not change in size.
1. Enable warning "4503" # Decorated name length exceeded.
2. Enable warning "4146" # unary minus operator applied to unsigned type.
3. Enable float64 support for the Softmax operator
4. Enable compliance checks for Windows x86 32bits build
5. Use TryBatchParallelFor to replace some fallback code in mlas pooling.cc
6. Fix Android CI pipeline.
* use correct type for for loop
* explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276.
* Remove allocator type from the key comparison in ExecutionProviders.
Remove usage of DummyArena as it's no longer necessary.
* Fix x86 tests where arena allocator is disabled.
Make initialization of OrtMemoryInfo clearer by adding Invalid enum value.
* Make OrtValueNameIdxMap::MaxIdx more intuitive.
Advance commit to 4df80d5865a9d4e97f6d0b9304d4316115a04d9e
Add generated code for the commit before editing.
Import more featurizers.
Rename Automl ops domain to mlfeaturizers.
Rename conditional compilation macro.
Move and rename files getting rid of automl
Rename --use_automl build switch to --use_featurizers
Rename CMake option accordingly. Rename automl CMake targets.
Adjust CI and packaging pipeline switches.
Rename namespace automl to featurizers.
Rework TensorSeq in a manner consistent with Tensor and SparseTensor
in terms of type system setup.
Reduce templating. Introduce helpers to ensure the same
data type.
Make OrtValue __dtor not virtual.
Introduce ContainerChecker
* enabme telemetry
* enable telemetry
* set enable telemetry as default
* for debugging
* remove log and set disable telemetry as default back
* delete private file while testing
* resolve comment: mainly add license header, rename macro and update docs
* rewording in privacy.md
* Optimize CPU Transpose for one axis moving either inwards or outwards. We have optimizations for NCHW <-> NHWC in CUDA but not CPU. This provides a more generic optimization to the CPU implementation.
Tested performance in both directions with data sizes of 8, 16, 32 and 64 bits, size of axis being moved of 3, 16 and 32, and number of elements to move of 100x100, 300x300 and 1000x1000.
Across all tests the average improvement even with the overhead of python was 2.5x. No cases were slower. Some were 6x faster.
Binary size increase in RelWithDebInfo build is ~5K.
NOTE: See PR comments for details of performance comparison with Eigen. Eigen is slightly faster but increases binary size by 55K just for support of rank 4 input. Binary size would be further increased to support different ranks.
- Added C-API test for loading custom op shared lib.
- Made some changes in C++ api header and C-api implementation to get it working.
- Added C# API and corresponding test for loading custom op shared library.
* Guard unused parameter
Guard unused parameter for Linux Arm and other cases.
* Add ACL (Arm Compute Library) execution provider
Add a new execution provider targeting Arm architecture based on Arm Compute Library.
Validated on NXP i.MX8QM CPU with ResNet50, MobileNetv2 and VGG models.
All unit tests are passing.
Comparative performance improvements for ResNet50v1 model obtained with
onnxruntime_perf_test:
A72 2xA72 A53 4xA53
ACL vs CPU 16% 9% 21% 13%
Usage documentation available in ACL-ExecutionProvider.
* Fix eigen unused parameter
Fix eigen unused parameter error for Arm cross-compilation.
* Initial draft
* updates per review
* fix link
* plus one more link fix
* small changes to the optimizer documentation
* some more changes
* done
* update C_API with doc link
* Refine optimizers
* Address PR comments
* Changes from PR comments and discussion.
* Fixed signed/unsigned mismatch
* Address PR comments
* Address PR comments
* Fix linux build
* Fix issue with mkldnn logic.
* Turn off optimizers by default for operator unit tests.
* Handle edge case of graph with no nodes in partitioner so all execution providers don't need to.
* Comment out change to turn off optimizers for unit tests. Add details on what needs to be done to re-enable.
This change adds a new execution provider powered by [DirectML](https://aka.ms/DirectML).
DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning on Windows. DirectML provides GPU acceleration for common machine learning tasks across a broad range of supported hardware and drivers.
The DirectML execution provider is capable of greatly improving evaluation time of models using commodity GPU hardware, without sacrificing broad hardware support or requiring vendor-specific extensions to be installed.
**Note** that the DML EP code was moved verbatim from the existing WindowsAI project, which is why it doesn't yet conform to the onnxruntime coding style. This is something that can be fixed later; we would like to keep formatting/whitespace changes to a minimum for the time being to make it easier to port fixes from WindowsAI to ORT during this transition.
Summary of changes:
* Initial commit of DML EP files under onnxruntime/core/providers/dml
* Add cmake entries for building the DML EP and for pulling down the DirectML redist using nuget
* Add a submodule dependency on the Windows Implementation Library (WIL)
* Add docs under docs/execution_providers/DirectML-ExecutionProvider.md
* Add support for DML EP to provider tests and perf tests
* Add support for DML EP to fns_candy_style_transfer sample
* Add entries to the C ABI for instantiating the DML EP
* Introduce execution mode for clarity and extensibility; Change Python APIs accordingly; Replace DisableSequentialExecution API with EnableParallelExecution for clarity.
* Fix cuda build
* Modify the test slightly
* Make C and C# APIs consistent with Python.
* Add ability to get symbolic dimension info for graph inputs and outputs.
WIP to get initial feedback.
* Fix linxu build error.
Update C# API and add unit test
* Clarify the two different ways Tensor shape and type info is created. One is from concrete values and one is from a type proto where symbolic dimensions may exist. Doing so allows a change to default to empty strings for the symbolic dimensions if not provided.
* Mention OrtCreateSessionFromArray in C API doc
* fix seq of tensors
* changes on 9/30
* All tests passing
* Add SequenceAt op
* Fix shared_lib non_tensor_types test
* Address some PR comments
* Address PR comments
* Add support in python bindings to accept seq(tensor)
* Change data type from vector<Tensor> to TensorSeq
* Change data type from vector<Tensor> to TensorSeq
* Added some documentation
* Added missing test model
* Fix Linux build
* Fix Mac build
* Fix Mac build
* Add Unique operator.
* Enable onnx tests. Disable one with incorrect expected output and add unit test to validate ORT behavior. Need onnx update to fix (will address that separately but don't want to block this checkin on that change).
Remove gsl subodule and replace with a local copy of gsl-lite
Refactor for onnxruntime::make_unique
gsl::span size and index are now size_t
Remove lambda auto argument type detection.
Remove constexpr from fail_fast in gsl due to Linux not being happy.
Comment out std::stream support due to MacOS std lib broken.
Move make_unique into include/core/common so it is accessible for server builds.
Relax requirements for onnxruntime/test/providers/cpu/ml/write_scores_test.cc
due to x86 build.
Add ONNXRUNTIME_ROOT to Server Lib includes so gsl is recognized
* Don't return shape for non-const initializer in InferenceContextImpl::getInputType
Don't return initializer for non-const initializer in InferenceContextImpl::getInputData
Update graph_utils to support these scenarios
- fix GetConstantInitializer to make sure a name is for an outer scope value before checking a parent graph, as local name could shadow an outer scope initializer.
* Mention OrtCreateSessionFromArray in C API doc
* Add C API for free dim override
* Add C API for free dim override, fix missing API mention in InferenceTest.cs, fix confusing print statement in perf_test.
* Remaining C#files
* fix c# build
* Run the tests in blame mode. This option is helpful in isolating a problematic test causing the test host to crash.
* fix order
Description: Refine threading control options and move inter op thread pool to session state.
Added thread_utils.h/cc to centralize the decision around the thread pool size under various conditions.
Motivation and Context
Currently the thread pool size of the parallel executor is hardcoded to 32 for some reason. This PR makes the options to configure the thread pool sizes clearer.
* Fix symbolic shape inference for faster_rcnn, mask_rcnn, yolov3
Force merge when --auto_merge, on symbolic dims which sympy cannot simplify
Add symbolic inference for Resize opset 10
Add support for step != 1 in Slice
Add support for computed dim in TopK
Bug fixes in passing symbolic dims from subgraph
Fix an outdate comment in Nuphar provider header
* Bump onnx to latest
Update onnx.in.proto with changes for SparseTensor.
* add temp skip tests
* remove passed tests from skip list
* skip more tests for new ops in opset 11
* skip crashing tests
* update handling of new attribute types sparse tensor and sparse tensors
* advance onnx commit and remove skip cpu_flaky_tests
* temporarily skip yolo3 model test due to resize opset10 shape inference regression
* update proto for onnxruntime server
* advance onnx commit further
C/C++ Opage APIs
Add new virtual interfaces for NonTensorType
Implement entry points.
Add shared header for the data container.
Add export symbols.
Add serialization/deserialization.
Implement model with Opaque types.
Rework opqaue_api_test as a standalone executable.
* Mention OrtCreateSessionFromArray in C API doc
* Add GetDataTransfer() interface in the EP.
* Check return status of RegisterDataTransfer
* Address PR comments
* Rework the feed/fetch copy setup so that it can be calculated upfront by the control flow nodes. Also simplifies how it all works.
Update the control flow nodes to do the calculation prior to graph execution.
* Implement Nuphar execution provider
Nuphar execution provider is a TVM-based compilation provider. It has shown great speedups for RNN models using Scan.
This PR is mainly for a preview of the shared codegen library for other TVM-based providers.
* Fix submodules
* Fix TVM submodule
* Update Nuphar to latest and resolve confliction
* Remove stale files caused by merge -X theirs
* Revert heap buffer change to not introduce onnxruntime_framework into onnxruntime_perf_test
* Fix bad merge
* Merge from Nuphar
* Fix warning treated as error, revert some unnecessary changes
* Revert some more test changes
* Some more test revert or comments to make review easier
New tests could be added later
* One more revert of unnecessary changes
* More change revert. Test could be added back later.
* Mention OrtCreateSessionFromArray in C API doc
* Don't create the default allocator every single time. Rename API accordingly.
* Don't create the default allocator every single time. Rename API accordingly.
* updates...
* updates...
* PR comments
* fix typo in license header
* fix build
1.Let mlas use session thread pool
2.Remove onnxruntime_USE_MLAS cmake option
3. Remove the win32 thread pool code inside mlas
mlas will:
1.use ort thread pool if it get passed in
2.use openmp if the threadpool parameter is nullptr
3.run single threaded if the threadpool parameter is nullptr and openmp is disabled.
Added Sample Featurizer and Infrastructure
Make featurizers and unit tests compile and run with GTest.
Create definitions for the first featurizer kernel.
Add new operator domain.
Create datetime_transformer kernel and build.
Move OPAQUE types definitions for featurizers kerneles out to a separate cc.
Register them with the type system.
Provide unit tests for new AutoML DateTimeTransformer kernel.
Make necessary adjustments to the test infrastructure to make it run
with new types.
- Added python script for generating markdown doc from the registered opkernels.
- Made some conditional changes in the pybind to expose necessary python API
- Added some missing type-constraints in the op kernel registrations
* Mention OrtCreateSessionFromArray in C API doc
* review changes
* use enum for graph optimization level
* Use explicit values for enums
* updates...
* Add friendly enum for graph optimization levels in C, C# and Python APIs.
* Fix linux build
* Fix build breakage due to master merge
* PR comments
* Mention OrtCreateSessionFromArray in C API doc
* Fix perf test executable due to removal of certain C APIs
* fix linux build
* Avoid duplication
* Fix mem leak
* remove memory copy between CUDA and TRT
* add info to RegisterExecutionProvider input
* use new IDeviceAllocator for trt allocator
* remove SetDefaultInputsMemoryType from TRT EP
* remove onnx-tensorrt 5.0
* add submodule onnx-tensorrt branch 5.1
* remove redundancy
* Update transformer_memcpy.cc
* Update tensorrt_execution_provider.cc
* switch to TensorRT 5.1.5.0
* update python binding
* disable failed test case on TensorRT
* Update activation_op_test.cc
* upgrade to TensorRT container 19.06
* update according to feedback
* add comments
* remove tensorrt allocator and use cuda(gpu) allocator
* update onnx-tensorrt submodule
* change ci build cuda directory name
* A few performance improvements:
- Make the iteration in NonZero more efficient by using a raw pointer and simplifying the increment logic
- add another unit test to check the new logic works with 3 dimensional tensor
- gains about 2% for ssd_mobilenet
- Avoid floating point operations on each iteration on Concat
- about 0.5% for ssd_mobilenet and ssd_resnet34
- Put common case first in ExecutionFrame::AllocateAsPerAllocationPlan to avoid unnecessary call to IsSparseTensor
- about 0.05% for ssd_mobilenet
- Minor tweak to put some ctors in the TensorShape header so they can be inlined more easily
* If there is an outer scope value that matches a subgraph input, don't create an implicit input from the outer scope value.
Minor unrelated change for issue noticed while debugging: Use unordered_set for implicit inputs so we don't add them multiple times.
* Add unit test based on onnx issue.
* Add string attribute interface for C API.
* Add string attribute interface for C++ API accordingly.
* Update comment to say that string is also valid
* Use INFO instead of WARNING for an unused graph input.
* Drop severity of unused initializer as well
* Update to output a warning level message if removing an initializer that is never used, and an info level message if removing an initializer that optimization has made redundant.
* Now that we check for a constant initializer in an ancestor graph we also need to be able to retrieve and replace that initializer.
Add helpers to do so.
Update optimizers to use the new helpers.
Fix bug in UnsqueezeElimination where it wasn't checking if the initializer it was replacing was constant.
This change integrates the NCHWc support recently added to MLAS into ONNX Runtime. When using "-o 3" optimizations, then the runtime will do a NCHWc layout optimization pass to convert standard ONNX operators such as Conv/MaxPool to the com.microsoft.nchwc domain with weights and biases reordered for speed.
Description:
Disallow overriding an initializer via a graph input if the IR version is < 4. This enforces an implicit assumption that initializers should be treated as constant, and allows constant folding to be done on a model with an older IR version.
Separate constant and overridable initializers so that it's clear which ones constant folding can utilize.
Update Graph to not add all initializers to the graph inputs when the graph is manually created (i.e. not loaded from a GraphProto) and the IR version is >= 4.
Motivation and Context
In order to do constant folding we need to know which initializers can be treated as constant and which are overridable. All initializers were required to have a matching graph input prior to IR version 4, technically making all of them overridable. The intention however was for them to be treated as constants, and this change enforces that intent.
The benefit of doing so is that constant folding will work for models with IR version < 4. The cost is that if someone is actually overriding an initializer they will need to update the IR version of their model to version 4 in order to keep doing so. The belief is that this is a very small subset of usage (e.g. models involving feeding in a truncated sequence) and the cost to update that small subset is warranted by the benefit of constant folding being able to be enabled on all older models without them needing an IR version update.
* init
* Update DNNLibrary
* Update DNNLibrary, set compiler flags, it compiles now
* Add more missing flags, add test
* Update DNNLibrary
* Update Compile method, fix allocator and some other bugs
* Update DNNLibrary
* Implement CopyTensor
* Not delete state explicitly since it is managed by unique_ptr
* Add the missing files when SingleUnitTestProjct is ON
* misc changes
* Fix wrong name in provider factory
* Add my own test
* Update the code of add node into graph, and add the missing initializer into graph
* Fix the bug that re-build the graph produces extra output
* Update DNNLibrary
* Transpose nchw (ONNX) -> nhwc (NNAPI)
* Add license
* Add GetSupportedNodes method (implement it later)
* Rename onnxruntime_nnapi_test->onnxruntime_nnapi_squeezenet_test
* Update squeezenet_test.cpp after rebase master
* Remove squeezenet_test.cpp since it is almost same with the c++ sample
* Update DNNLibrary for GetSupportedNodes
* Update GetSupportedNodes
* Revert "Remove squeezenet_test.cpp since it is almost same with the c++ sample"
This reverts commit a97575fd9ff49e50ba1dc8d8154790d8cd86c48d.
* Update DNNLibrary
* Fix multiple outputs bug
* Remove GetKernelRegistry
* Revert "Revert "Remove squeezenet_test.cpp since it is almost same with the c++ sample""
This reverts commit 2a0670e9cbf10ea654111ce39e198a4be0ddd838.
* Set default memory type of NNAPI EP
* Add CPUOutput allocator
* Update DNNLibrary for multiple outputs
* Fix bug of nhwc->nchw
* Remove GetExecutionHandle()
* Initial commit for OpenVINO Execution Provider
OpenVINO Execution Provider provides the interface for ONNX Runtime
applications to access Intel's hardware accelerators using Intel's
OpenVINO Toolkit.
* Fixed bug in GetCapability to disable custom ops
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Added OPENVINO ci pipeline
Added new pipeline for openvino provider,
made changes to support the docker build and
onnxruntime build with openvino.
Signed-off-by: Luis Daniel Castellanos <luis.daniel.castellanos@intel.com>
* Enabled all unit tests for OpenVINO EP
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Fixed syntax issue in run_docker_build.sh file
* Added missing default OPENVINO_VERSION
Default value for OPENVINO_VERSION env was
missing causing the build to fail
* Added install Model Optimizer deps step
* Fixed python unit tests and some tests from onnx_backend_test_series
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Fixed indentation bug
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled some of the python backend tests for OpenVINO
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled some model tests
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Remove Duplicate checks for openvino in build.py
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Modified GetCapability for FP16
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled GPU FP32 tests that are not supported
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Convert modelProto to string and use it in compile
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Pass byte-array input args to MO
* Serialized ModelProto passed in-memory to MO
ModelOptimizer python module receives the serialized ModelProto
in-memory.
Uses appropriate ONNX function to load the serialized bytes.
* Make Py_Finalize compatible with older python versions
Also, remove pFunc unassigned variable possibility.
* Fallback if input dims of Matmul is greater than 2
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* fixup: Device #define syntax
* Updated the documentation
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Enable dynamic dim value
* removed commented out code
* Added Dockerfile for openvino EP
Updated instructions on dockerfiles/README.md file
Signed-off-by: Luis Daniel Castellanos <luis.daniel.castellanos@intel.com>
* Disabled fp16_inception_v1 test
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Code formatting with clang-format
Uses style from the .clang-format file in root directory.
* fixup: docker tag and build error fixes
* Heuristics to automatically detect batching
Distributes slices from batch into parallel infer-request objects.
* Handle disabled tests in GetCapability
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled average pool and max pool if ceil_mode is 1
Also dilations are not supported if they are greater than 1
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled Unsqueeze int32 test
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* changes to fix output results bug
* Disabled a few C++ unit tests for MYRIAD FP16
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Manually revert '9fe162bb Enable dynamic dim value'
Reverts compile time setting of dynamic shape
Reverting manually due to significantly huge auto-revert conflicts.
* Fixed unused variable warning
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled Mul test for GPU_FP16 due to accuracy issue
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* VPU documentation update
* Disabled inception_v1 for MYRIAD and HDDL
*Also disabled few C++ accuracy tests for HDDL
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* updates from upstream
* use the new CustomOpApis for I/O interfacing
* Pass initializers as subgraph meta-def inputs in GetCapability()
Requirement due to API changes introduced with PR# 1019.
* Remove obsolete functions
* Save indexes of graph inputs from fused_node info
Both inputs and initializers are passed as data inputs to the
infer function. To identify only inputs among them, save thier
index info from fused_node in Compile function.
* Documentation changes to enable VPU
* Fix VPU related changes in documentation
* Fix minor changes in documentation
* Fix VPU related changes in documentation
* Use Node.In/OutputDefs() to track graph inputs and outputs.
Don't use graph_viewer's GetInputs() or
GetInputsIncludingInitializers().
* Permit "SAME_UPPER" auto_pad attribute from MaxPool
* Disabled fp16_tiny_yolov2 in onnx model tests
* Updated documentation to include configuration guides for myriad and hddl
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Use 8 Infer requests only for VAD-R
* disable debug prints
* Clang-format source files
* Updated BUILD.md with OpenVINO R5 links
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled same upper python tests
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Update test exclusion syntax
* Change path of install_onnx.sh
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disable tiny_yolov2 in broken tests
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Revert "Change path of install_onnx.sh"
This reverts commit ba9db165f3be430f2aff1ef413299ed04637196a.
This change is only required for Intel internal CI pipeline until
the settings are matched with the upstream's CI pipeline.
* Added debug statements for debugging CI error
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Add --build_wheel to linux openvino pipeline
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Added -v option to onnx_test_runner for debugging
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Removed path change patch
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Added -c 1 to onnx_test_runner
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Refactor MO python invocation in separate function
Cleans up Model Optimizer python invocation check and conversion
logic. Invokes MO only once in GetCapability() and passes the
IR strings (xml and bin) to the Compiler as meta-def attributes.
* Add comments
* code cleanup and comments
* Code cleanup for GetCapability
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Removed unnecessary files
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Revert "Added -v option to onnx_test_runner for debugging"
This reverts commit d1dd70938a94d648df1a1dbbc2e48d0b97e49ec8.
* Revert "Added debug statements for debugging CI error"
This reverts commit b86d41afed2aa29c3508155d6f9c8d3a7263cc60.
* incorporate Status Code changes
* ComputeFunc returns Status::OK() on success
* Use test names to disable tests for MYRIAD and VAD-R
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Rename local identifiers from CNNNetwork to OpenVINO network
CNNNetwork is an OpenVINO's API class that represents more than
just convolutional neural networks (CNNs). Renaming helps to avoid
confusion that the API's only support CNN type models.
* Added error message if building on windows
* Removed duplicate option in Cmake
* Removed unnecessary parameters in activation_opt_test
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Refactor Map search and access logic for efficiently and cleanliness.
* use C++ style casts
* Use os.path.join for python directory path operations
* use C++ style casts
* EP classes should use onnxruntime namespace
* Clean up fixes from PR comments
* Don't explicitly shutdown Py interpreter
* Remove debug print statements
Prints will be re-enabled later with a logging mechanism with
debug/verbose printing options.
* Decrement ref counts for used pyObjects
* Restore build instructions for other compilers
Content under the "Using other compilers" section has been
accidentally deleted by a previous commit. Restoring back that
content from the latest upstream repo.
* CMake code cleanup
Code clean up, commenting and formatting of CMake code.
* Don't pass the unused device_info parameter to OpenVINOGraph ctor.
* Add support for multiple I/O data types
Adds support for the following tensor data types for graph inputs
and outputs:
1) float
2) float16
3) int32
4) int16
5) int8
6) uint16
7) uint8
* cleanup setup.py module list definition
* Deduce index of input using tracked input index map
Ignores initializers in case they are ordered before inputs.
* Removed debug statement in MO code
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* PR feedback
* Removed per_sample_tolerance for openvino
* Removed unnecessary disabled tests
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Removed debug function
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled tiny_yolo_v2 due to accuracy issues
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Changed the disabled reason for broken tests
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Disabled Reshape with no input
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Python formatting with Autopep8
* Minor fix for MYRIAD devices
* Added zero dimension check
*Removed setting batch size for the network
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Set the threshold to larger value for MNIST
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Removed setting higher threshold in provider_test_utils
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
* Check for --use_openvino in python wheel setup.py
Add openvino modules to the setup script for building the wheel
package only for --use_openvino a build option.
* Removed nullptr checks for GetNode()
Signed-off-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
Add ability to set the session and run logger severity via SessionOptions and RunOptions
Inherit severity from the next logger up if logger severity isn't specified in SessionOptions or RunOptions
Expose ability to set default logger severity in python bindings.
* refactoring the ep codes.
* remove unnecessary lock.
* fix the comment to claim KernelRegistryManager is not thread safe.
* clarify that APIs to add custom op in inferencesession is not thread safe.
More C++ API improvements and cleanup
Add templates to tensor creation
Add run method that allows preallocated outputs
Simplify CreateTensor<T> to multiply by sizeof(T)
Convert io_types code
Optimize away vector copies in Session::Run
* Change function signature
* Convert compute to use custom op style APIs
* Remove dead CustomOp function
* Use CustomOp API in TensorRT EP
* Switch to new API in ngraph
* More C++ API improvements and conversions
* Mark more constructors as explicit
* Fix CSharp function name changes
* Change more test cases to use C++ API
* allow users to set graph inputs and outputs fully.
* update
* update the comments of the APIs
* update
* remove commented-out codes.
* fix test failures.
* fix comments.
* adding more check to throw not support exception right now.
Memory pattern doesn't work for parallel executor by design. Enabling Memory Pattern for parallel executor logs warning and make the perf bad.
Add option to enable/disable memory pattern back.
Introduce a quick pre-filtering of rules based on the node op types they are targeting.
The goal is to avoid evaluating all rules for all nodes. Instead, for each node, we will only be evaluating the rules associated with its op type.
* constant node should not be put into graph inputs any more.
* simplify graph input/output set logic.
* refactor comments.
* remove adding initializers as graph inputs when creating graph from scratch.
Some changes that reduce the size of the release onnxruntime.dll by 170KB:
Change the ONNX_OPERATOR_KERNEL macros to not create a unique virtual class per kernel create lambda, but instead use a generic class with the raw function address supplied at BuildCreateKernelInfo time.
Changed the exceution providers to use a table driven approach to calling the BuildCreateKernelInfo functions instead of a massive function with construct/call/delete sequences.
The CreateFunc in data_types.h didn't need to be a std::function, eliminating more lambda virtual classes.
N.B. To accommodate MSVC 14.11 toolchain (used for CUDA builds), the operator+() syntax cannot be used to retrieve the raw function address. The older toolchain can't resolve between cdecl/vectorcall and gives up. An explicit cast is needed to help the compiler along.
* Adding a custom op interface to the C API to remove shared library dependency.
* Remove old custom op test
* Rework how custom ops handle inputs/outputs to enable custom op output shape calculation in the compute method
* Add a nicer C++ API for custom ops and switch the tests to use it.
Generalize node removal method in graph_utils. This is a higher-level method that keeps the graph consistent so that no Resolve is needed after the removal of a node.
The new method supports the removal of nodes with a single input (be it an incoming node or an initializer) and a single output (but allowing multiple output edges of that output). It also takes into account the case that one of the output edges is fed to a subgraph.
Also updated the rewrite rules to use this new, less restrictive method, and improved the rules' conditions. Introduced a GraphEdge struct to simplify various methods in graph_utils.
* fix graph transformers and refactor tests
* fix merge master
* Set default optimization level to Level1
* fix build warnings for Linux
* try root cause tensorrt test failures
* try root cause tensorrt test failure
* Test level2 transformers with all CI builds
* remove ConvActivation fusion transformer
* change default level back to level1
* remove providers from apply api
* more changes
* Convert unsqueeze elimination to rewrite rule
* Simplify the way we register predefined transformers and rules in the inference session (all details are now moved to the graph transformer utils)
* Some reorganization and renaming of methods in graph_utils
* Updates in graph transformers test
* Update in edge removal to not perform unnecessary check of node args that led to race conditions when updating the graph
* Improve documentation for rewrite rules
* Remove top-down rule-based transformer (given we currently have only one type of rule-based transformer)
* Adding a custom op interface to the C API to remove shared library dependency.
* Fixup const issues
* Renaming to make things a little simpler
* Add a comment
* Test protobuf-lite
* Test protobuf-lite
* Test protobuf-lite
* Optimize protobuf usage for LITE_RUNTIME to reduce the binary size of
onnxruntime.dll. More details can be found here https://developers.google.com/protocol-buffers/docs/proto.
The reduction is significant. For commit id: 4873b452151bafe49da332aaeab639ef0318fc1ca28d728, the size
reduced by ~700K; from 4873728 to 4172800.
* Add LITE_RUNTIME flag in in.proto files
* Fix merge conflict.
* Address PR comments
* Forgot to add 2 files + fix linux and gpu build errors.
* Fix build errors + test failures
* Fix cuda tests
* Fix tensor rt build
* Use full protobuf for trt
* Address PR comments
* Print tensor shape proto as text string for easier debugging
* Check usage of node output as implicit input in any subgraphs.
* Add logic to check/update subgraphs when removing a node.
Fix some issues with Graph
- Include local outer scope variables when validating. Required if calling Resolve on a subgraph
- Include outer scope variables in the value info so the type information is captured. Also required to Resolve a subgraph but will detect a type mismatch (previously we threw the type information away).
- Fix GraphNodes iterator so it can be used with std::find_if. Needed to be assignable so the end_ value can't be const.
* graph transformers update
* some updates
* plus changes
* more updates
* fixes per review comments
* enable tests
* adding more tests
* more changes
* update api in inference sesion
* changes per review
* Linux CI fix
* fix linux CI failure
* fix MAC CI failure
* more updates
* add more documentation and add level param to register transformer
* Add AllocKind::kShare to allow copying the MLValue for a pre-existing value to a graph output when an Identity node is involved. Ideally we can make this handling for an Identity node more general purpose, however the current logic to free an MLValue during execution doesn't take into account a re-use point also needing a free. Due to that, limit the scope and start with a somewhat ugly hardcoded approach.
Migrate some changes from PR497
The existing Loop unit tests exercise the new code. Also manually stepped through the problematic model to verify the unnecessary copy was avoided.
* Fix build error
* Fix missing switch case in debug output of allocation plan
* Limit optimization to Loop
* updated cmake files for trt
* added trt execution provider
* added trt basic test
* removed trt_path action attribute
* Add files via upload
* Update build.py
* Update trt_allocator.h
* fixed issues found by reviewers
* changed cast operator
* added comment for custom kernel implementation
* changed auto to auto&
* changed to function compile APIs for TRT execution provider
* changed to function compile APIs for TRT execution provider
* added new DType DInt64
* adapted to the changes of onnxruntime_c_api
* removed trt kernel (use function compile instead)
* updated onnx-tensorrt submodule
* set default memory type to TRT fused kernel
* resolve merge conflict
* fixed the issue that USE_CUDA conflicts with USE_TRT
* construct graph by adding nodes in topological order
* made changes for Windows
* change buffers type
* bypass HasImplementationOf check for TRT XP because TRT kernel is not registered
* added domain to version info in rebuilt model proto
* added trt to test option list
* added DomainToVersionMap() to GraphViewer
* removed Copy()
* fixed broken code
* format the code to clang format
* used local reference to the frequently used values
* fixed a couple of issues according to reviewers feedback
* fixed a couple of issues according to reviewers feedback
* added python binding for TRT and enable use_cuda when use_trt is on
* fixed a redefinition issue
* changed shared_ptr to unique_ptr on trt engines, and made a few changes required by reviewers
* enabled trtexecution provider for unit tests
* renamed trt to tensorrt
* added tesorrt to python binding
* update submodule onnx and onnx-tensorrt
* made a couple of minor changes based on reviewer's feedback
* added CUDA_CHECK
* removed test code
* fixed broken code after merge
* updated onnx-tensorrt submodule
* added post processing to align trt inputs/outputs with graph inputs/outputs
* updated onnx submodule
* added CUDA fallback for TensorRT and fixed TensorRT cmake issue
* added ci pipeline for tensorrt and removed some redundent code from trt xp
* fixed syntax issue
* updated onnx-tensorrt submodule
* fix trt build problem by: (#602)
1. Add additional /wd for debug build
2. Add io.h for additional targets
3. Bring back mb version of getopt
* Update install_ubuntu.sh
* Update linux-gpu-tensorrt-ci-pipeline.yml
* Update linux-gpu-tensorrt-ci-pipeline.yml
* Update run_build.sh
* Update run_build.sh
* Update run_build.sh
* Update run_build.sh
* fixed the issue that GetKernelRegistry returns nullptr
* merged master to this branch
* moved some data types to private
* fixed tensorrt CI pipeline issue
* customized test data for TensorRT pipeline
* added onnx-tensorrt in json file and fixed an issue in ci script
* added comments
* Prototype version that demonstrates it can work
* Switched to OrtValue and removed the OrtCustomOpTensor code.
* Support multiple outputs and reading of attributes
* Add custom domain handling to custom ops
* Update documentation
* more wording changes
1. Support the new external data extension in ONNX 1.4 onnx/onnx#678
2. Enable onnxruntime_perf_test in Mac Build
3. move path_lib.h from onnx_test_runner source dir to onnxruntime_framework
4. Enable memory planner for string tensors
5. Make memory planner always enabled, to simplify model loading logic
6. Delete some duplicated code between onnxruntime_perf_test and onnx_test_runner
7. Delete win_getopt_mb lib.
8. Remove the dependency on Pathcch lib, which is only available on Windows 8 and newer.
* Initial commit
* Adding shrink tests
* Fix formatting in shrink_test.cc
* Fix broken build
* More changes
* PR feedback and formatting
* Place files in the right location corresponding to def file location in onnx
* Exclude shrink model test in test_series.py
* Remove shrink from exclusion list in main.cc
* Adding test to exclusion list
* More tests
* Formatting
* PR feedback
* PR feedback
* More changes
* PR feedback
* More changes
* Fix broken build
* Fix nit
* Fix nit
* Break dependency on SessionState for ExecutionFrame and OpKernelContext so optimizers can execute a node with a minimal setup.
- Create IExecutionFrame
- split out core logic and interface from extended logic used in full Graph execution (that uses allocation plan and memory pattern planner)
- Update NodeIndexInfo to allow contruction from a subset of nodes
- split out logic from GraphNodes into a re-usable template so it can be used with a vector of const Node* as well as a vector of unique_ptr<Node>
- Remove SessionState from OpKernelContext
- Misc cleanups
- move AllocPlanPerValue out of SequentialExecutionPlan as it's used in a more generic manner that isn't specific to a sequential execution plan
NOTE: I manually tested the new paths, especially NodeIndexInfo. There will shortly be optimizers added that use the new infrastucture so they'll get test coverage as part of those changes.
* Fix linux build issue.
Handle graph with no nodes in NodeIndexInfo.
* Add ability to enable caching to the C API, and update the internals to pass the feed names and MLValue instances in vectors so the order is deterministic (so cache entry matching works as expected).
* Address PR comment and don't use 'bool'
* Remove meaningless C# test around duplicate input.
We _could_ check input names for duplicates (previously we did this via the usage of unordered_map), but the system will gracefully handle with the duplicate anyway (will just use the last value provided for the input name).
Based on that, I don't think the cost of checking for duplicates is worth it.
* Fix c-style cast in test_run_options.
* cleaned up the additional header in C-api
* ensure test failure surfaces in the build pipeline
* sanitized runtest.bat
* cleanup unneeded headers
* formatting and typos
* Various optimizations to reduce the setup and execution cost.
Cache information about the feeds and fetches, and any device copies required to execute the graph so we minimize checking for later calls to ExecuteGraph using the same input/output.
- enable use of caching in Loop and Scan
- make use of caching optional for InferenceSession::Run
- handle calls to Run with different feeds and fetches to support scenarios where there may be a truncated sequence in some calls
Take the feed names and MLValue instances as vectors so the order is deterministic.
Add unit tests
Update onnxruntime_perf_test to enable caching.
* Couple of tweaks.
Fix shared library unit test failure.
Attempt to workaround MacOS build failure due to VC++ bug around including reaching scope values in a lambda automatically.
* Rework order of init in Run so we get nice error messages about invalid feed/output names.
* Refine logic around copying MLValue using execution provider so common code can be used. Simplify the logic due to this change.
Split the paths for executing with/without cached info so we can be more const correct with how FeedsFetchesManager is passed in. This makes it clearer when a shared instance can be used due to it being const.
Cache the FeedsFetchesManager instances in the control flow nodes. They can be re-used across calls to Compute.
* Removed unused local variable to fix some builds.
* Fix build issue by cleaning up some more unused params.
* Check names when using cache entry from SessionState. Add unit test.
* support non-tensor types
* support non-tensor types.
* support non-tensor types.
* fix compilation issues
* fix compilation issues
* fix compilation issues
* add test cases
* test cases
* add test cases
* try to fix string test case
* working now
* use allocator (broken)
* string test broken after using allocator
* full working example
* Fix PR comments
* Create a project for graph optimizer.
Move optimizer related code to the folder optimizer.
* Fix build failures.
* rebase and fix build failures.
* fix build failure.
* fix build failure with cuda path.
* fix python build failure.
* Move two transformers(memcpy and insert_cast) from framework to optimizer.
* rebase.
* SessionState should not depend on optimizer.
* Add Recurse method to GraphTransformer.
Move GraphTransformer::Apply to ApplyImpl and make private.
Add non-virtual GraphTransformer::Apply method to handle calling Graph::Resolve in a more consistent manner.
Create MemcpyTransformer GraphTransformer to handle memcpy operations on subgraphs in a more standard way.
* Checkpoint
* Make the subgraph insert less verbose
* Add graph nesting level to transformer ApplyImpl
Tweak cast transformer to recurse nicely and avoid unnecessary Resolve calls by splitting out the duplicate removal into a separate transformer.
Decouple memcpy transformer from ExecutionProviders and minimise what's in the header.
* Recurse into subgraphs inside GraphPartitioner
* Update a couple of new transformers
* Check Recurse return value.
* Cleanup some memory management in inference session by moving some things into SessionState
* Add deleted flag to rewrite rules so we stop processing nodes that are removed.
Remove some (most likely) unnecessary Resolve calls. As we always call Resolve for a graph modified by a transformer there's generally no need for the transformer to do it.
* Minor cleanups.
* Add some extra usage information to the comments in GraphTransformer.
* Address PR comments
* Separate out the NodeArg index information from ExecutionFrame so it is only calculated once.
* Skip copy to/from device if only CPU execution provider is registered.
Cleanups.
* Address PR comments.
Clean up a few areas.
* Fix Linux build error
Rewrite rule that eliminates slice operators when they are redundant (i.e., when they preserve the whole input).
Re-implementation of the identity elimination rule using the latest Graph API.
* refactor kernel registry to make it a little bit more readable.
* update
* update cudaexecutionprovider
* fix build break
* fix comments
* fix build break
* Make OrtAllocator not be reference counted
* Make the allocator interface more type safe
* Fix build break
* Build break fix
* Build break fix
* Mistake in previous build fix.
* Fix review comments + build break
* Missed the export symbols
* C specific error, need 'struct' keyword in one case.
* Function calling OrtReleaseObject instead of OrtReleaseEnv
* merge function compile interface
* fix build error
* fix linux build break
* fix static cast issue; fix clang style
* fix argument change
* use alignment allocation;fix comments in pr
* fix linux break
* apply clang format
* rename according to comments in pr
* rename according to pr comments;remove useless file
* remove the need_compile flag
* avoid passing whole session state
* More API changes, remove 'Inference' from function names. Remove enum values. Make Status match other types.
* Switch to bool instead of int, and remove stdbool
Applies to all public headers and macros, plus many internal ones. There are still some internal things with OnnxRuntime in the name, but this fixes all public functions & macros.
* Rename ONNXRUNTIME_API_STATUSCALL to ONNXRUNTIME_API_CALL since it's just for calling convention, there's no status return value implied.
* Missed a function
* remove input edges while removing a node.
* put comments on how to remove a node without using resolve.
* fix test failure
* fix test failure.
* fix format
* try to guess and fix mac ci failure.
* change add remove edge api.
* update IR
* update comments
* fix comments.
* Checkpoint.
* Add doco to graph.h and graph_base.h.
Change NodeConstIterator to return a reference to clearly advertise no nullptr's are going to be returned as it's only iterating valid Nodes.
Fix some code analysis warnings.
* Make a couple of APIs return a reference instead of a pointer as they never return nullptr.
* More doco and some minor naming cleanups.
* Cleanups
Couple more consistency changes.
* Fix CUDA test file
* Fix invalid line.
