* Move allocators to SessionState so they're decoupled from ExecutionProviders
- when looking up an allocator it's based on OrtMemoryInfo not the EP so SessionState is a more natural place for that infromation to be stored
- add device based lookup
- simplifies logic for copying feeds/fetches across devices
Cleanup SessionState and SessionStateInitializer
- provide more things to SessionState at construction time so we don't construct and instance and immediately after call a bunch of setters
- simplify SessionStateInitializer
- reduced down to FinalizeSessionState method
* Make quantization support GPT2 past state
* Make OpTester to be able to generate reference outputs with a model. With it, there is no need to compute outputs manually, which are impossible for some cases.
* cast support for layernormfusion
* cast support for layernormfusion
* bug fix
* fix build
* bug fix
* fix test
* minor refactor
* on comments
* on comments
* on comments
* on comments
Co-authored-by: Ethan Tao <ettao@microsoft.com>
* Move nnapi dnnlib to subfolder
* dnnlib compile settings
* add nnapi buildin build.py
* add onnxruntime_USE_NNAPI_BUILTIN
* compile using onnxruntime_USE_NNAPI_BUILTIN
* remove dnnlib from built in code
* Group onnxruntime_USE_NNAPI_BUILTIN sources
* add file stubs
* java 32bit compile error
* built in nnapi support 5-26
* init working version
* initializer support
* fix crash on free execution
* add dynamic input support
* bug fixes for dynamic input shape, add mul support, working on conv and batchnorm
* Add batchnormalization, add overflow check for int64 attributes
* add global average/max pool and reshape
* minor changes
* minor changes
* add skip relu and options to use different type of memory
* small bug fix for in operator relu
* bug fix for nnapi
* add transpose support, minor bug fix
* Add transpose support
* minor bug fixes, depthwise conv weight fix
* fixed the bug where the onnx model input has mismatch order than the nnapi model input
* add helper to add scalar operand
* add separated opbuilder to handle single operator
* add cast operator
* fixed reshape, moved some logs to verbose
* Add softmax and identity support, change shaper calling signature, and add support for int32 output
* changed the way to execute the NNAPI
* move NNMemory and InputOutputInfo into Model class
* add limited support for input dynamic shape
* add gemm support, fixed crash when allocating big array on stack
* add abs/exp/floor/log/sigmoid/neg/sin/sqrt/tanh support
* better dynamic input shape support;
* add more check for IsOpSupportedImpl, refactored some code
* some code style fix, switch to safeint
* Move opbuilders to a map with single instance, minor bug fixes
* add GetUniqueName for new temp tensors
* change from throw std to ort_throw
* build settings change and 3rd party notice update
* add readme for nnapi_lib, move to ort log, add comments to public functions, clean the code
* add android log sink and more logging changes, add new string for NnApiErrorDescription
* add nnapi execution options/fp16 relax
* fix a dnnlibrary build break
* addressed review comments
* address review comments, changed adding output for subgraph in NnapiExecutionProvider::GetCapability, minor issue fixes
* formatting in build.py
* more formatting fix in build.py, return fail status instead of throw in compute_func
* moved android_log_sink to platform folder, minor coding style changes
* addressed review comments
As a zero-cost wrapper around the C API, the current state of the C++ API is still pretty low-level and requires programmers to use C-style standards to interact with ONNX.
1. Increase job timeout, while we are investigating why the tests take much longer
2. Upgrade the linux docker image to manylinux2010, by request from Tianlei. (We had an offline discussion with Pranav and Tracy)
3. Remove the installation of "devtoolset-7" in the CUDA image. It was added for CUDA 10.0, it is not needed for CUDA 10.1. We have moved to CUDA 10.1.
* Add amd migraphx execution provider to onnx runtime
* rename MiGraphX to MIGraphX
* add migraphx EP to tests
* support multiple program output
* disable more tests
* backup changes related to program multiple outputs
* remove logging code
* remove unnecessary changes in migraphx_execution_provider.cc
* add migraphx EP to tests
* add input requests of the batchnorm operator
* add to support an onnx operator PRelu
* update migrapx dockerfile and removed one unused line
* chagnes related to support dynamic input shape
* fix build error
* code backup
* code backup
* version that has 106 models run correctly
* code backup
* code backup
* remove unnecessary print info
* code backup
* code backup
* code backup
* code backup
* code backup
* code backup
* changes corresponding to migraphx change
* fix merge conflict
* minor code cleanup
* code cleanup
* remove unnecessary code
* remove unnecessary code
* add to support more constant folding analysis
* more constant folding checking for shape input
* add env var to control whether fp16 is enabled. Modify docker file to use ROCM3.3
* fix function name to avoid build error
* add build and execution instruction for migraphx execution provider
* added more build instructions
* fixed a small format error
* a minor change
* fix review comments
* another minor change
* additional refinement of the documents
* additional changes
* remove unnecessary changes in the dockfile
* additional changes for the dockerfile
* code change backup
* fix errors related to a few unit tests
* fix a build error related to api change
* fix unit test errors by either disabling the test or fix related isssues
* remove unnecessary log info
* sync submodule tvm with master
* remove unnecessary changes
* remove an unnecessary code line
* refine documents for addition example
- Move thread hint vectors from thread-local struct
- Add static_assert that the per-thread state in the thread pool is trivially-destructible
- Rename "thread_data" to "worker_data" (only allocated for workers in the pool, not threads calling into the pool)
* fix python ep default ordering. cpu provider should be last.
* add comment.
* add test case to ensure no regressions for get_all_providers().
* expand on get_all_providers() api documentation
* Rename partition_optimizer -> deepspeed_zero
* Use ZeROConfig in orttraining_pybind_state.cc
* deepspeed_zero -> deepspeed_zero_stage for clarity
* Expose as deepspeed_zero_stage in pybind
* Avoid signed/unsigned warning on loops
* Report sizes when distributed world configuration is inconsistent
* Add DistributedRunContextTest for pipeline stage configuration
* Changed the scheduler for VAD-M to bypass scheduler and modified logic
* Added extra configuration step to documentation for VAD-M
* Removed cout statement
* Fixed documentation
* Removed softmax restriction
* Added VPU config setting for graphs with dynamic shape
* Set VPU config only for MYRIAD
* Added log statement
* ReduceMean/Sum gradient without shape dependency.
* optimize expand and use it to replace add.
* Adjust test.
Co-authored-by: Vincent Wang <weicwang@microsoft.com>
Updates the thread pool implementation to make work distribution over the Eigen thread pool more closely resemble techniques used in OpenMP. In particular:
(1) A thread entering a parallel loop works on the iterations itself, rather than requiring a thread switch to/from a thread in the pool, if called from outside the thread pool.
(2) To support this, work items pushed to the thread pool run a loop to claim iterations from a shared counter via atomic-fetch-and-add, as opposed to having work items themselves represent individual batches of iterations. This means that any thread working on the loop can execute any batch of iterations, including having the main thread run through all of the batches itself if the loop turns out to be short-running.
(3) As with OpenMP active scheduling, the worker loop spins waiting for work prior to blocking. This avoids OS blocking / wake-up paths in workloads with series of short-running parallel sections.
* Added GetAvailableProviders to C API
* Fix API version and Windows build error
* Changed function name
* Changed ORT_API_VERSION to 4
* Moved all_providers array to constants.h
* Move check for providers to constants.h
* Changed name of array to avoid warning
* Address review comment
* Added unit test
- Update IAllocator setup to move the OrtMemoryInfo to the base class instead of requiring derived classes to have that as a member and override a virtual method to return it.
- Cleanup CreateAllocator setup to take an argument as to whether to wrap the device allocator in an arena allocator. The choice to do that isn't a property of the underlying device allocator.
- Minor cleanups in the various EPs to adjust to the change to IAllocator and CreateAllocator, and to use the create_arena flag consistently when available.
* Add build option to disable traditional ML ops from the binary.
* Fix python tests by splitting tests for ML ops to a separate file. Exclude ML tests from onnx_test_runner and C# tests. Exclude ML op sources.
* Update Edge pkg pipelines with new MLops env variable and fix C# packaging pipeline tests to skip ML ops.
According to profiling in #4267, getting the allocator can account for a large fraction of overhead when accessing a kernel output, due to STL container operations. The allocator isn't used when (i) we're not creating a fence, and (ii) we have a memory pattern and a pre-allocated buffer, so we can avoid this overhead.
* support position_ids input
* support fp16 conversion for gpt2 past state
* output results to csv file
* Remove the useless check that output of matmul is in cuda
