* Simplified version of WebAssembly support to keep most of existing data structures and add cmake using Ninja and emcmake
* Clean up CMakeLists.txt and add an example to create and compute a kernel
* Load a model from bytes and remove graph building steps
* Add all cpu and contrib ops with mlas library
* WebAssembly build with Onnxruntime C/CXX API
* Use protobuf cmakefile directory instead of adding every necessary source file
* Fix invalid output at example
* add missing files
* Change an example to use Teams model and support ort mobile format
* add API for javascript
* fix input releasing in _ort_run()
* update API
* Let onnxruntime cmake build WebAssembly with option '--wasm'
* allow one-step building for wasm
* Make build script working on Linux and MacOS
* Fix broken build from Windows command
* Enable unit test on building WebAssembly
* Resolve comments
* update build flags
* wasm conv improvement from: 1) GemmV; 2) Depthwise direct convolution 3x3; 3) Direct convolution 3x3
* Cleaned mlas unittest.
* use glob
* update comments
* Update baseline due to loss scale fix (#6948)
* fix stream sync issue (#6954)
* Enable type reduction in EyeLike, Mod, random.cc CPU kernels. (#6960)
* Update EyeLike CPU kernel.
* Update Mod CPU kernel.
* Update Multinomial CPU kernel.
* Slight improvement to Pad CPU kernel binary size.
* Update RandomNormal[Like], RandomUniform[Like] CPU kernels.
* Fix warning from setting multiple MSVC warning level options. (#6917)
Fix warning from setting multiple MSVC warning level options. Replace an existing /Wn flag instead of always appending a new one.
* MLAS: quantized GEMM update (#6916)
Various updates to the int8_t GEMMs:
1) Add ARM64 udot kernel to take advantage of dot product instructions available in newer cores. Some models run 4x faster than the stock implementation we used before.
2) Refactor the x64 kernels to share common code for AVX2(u8u8/u8s8/avxvnni) vs AVX512(u8u8/u8s8/avx512vnni) to reduce binary size.
3) Extend kernels to support per-column zero points for matrix B. This is not currently wired to an operator.
* Implement QLinearAveragePool with unit tests. (#6896)
Implement QLinearAveragePool with unit tests.
* Attention fusion detect num_heads and hidden_size automatically (#6920)
* fixed type to experimental session constructor (#6950)
* fixed type to experimental session constructor
Co-authored-by: David Medine <david.medine@brainproducts.com>
* Update onnxruntime_perf_test.exe to accept free dimension overrides (#6962)
Co-authored-by: Ori Levari <orlevari@microsoft.com>
* Fix possible fd leak in NNAPI (#6966)
* Release buffers for prepacked tensors (#6820)
Unsolved problems:
1. One test failure was caused by a bug in Cudnn rnn kernels, when they can allocate a buffer and partially initialize it, the garbage data near tail of the buffer caused problem in some of the hardware. To attack this problem in a broader sense, should we add code in our allocators, and during a memory fuzzing test, fill an allocated buffer with garbage before returning to the caller?
2. Prepacking is used more widely than we know. For instance, Cudnn rnn kernels also cache their weights. They mix several weight tensors together into a single buffer, and never touch the original weight tensor anymore. This is the same idea with pre-pack, but they didn't override the virtual function, and they never tried to release those weight tensors, leading to memory waste. It also seems to me that there are some other kernels have similar behavior. Wonder how much memory we can save if we try to cleanup those too.
3. Turning off memory pattern planning does increase memory fragmentation, leading to out of memory error in some training test cases. Perhaps we can revisit the idea of pushing kernels-creation stage earlier, and then during initializer deserialization, we only avoid tracing those that will be prepacked.
* Enable type reduction for Range, ReverseSequence, ScatterND, Split, and Unique CPU kernels. (#6963)
* add CI
* fix test in ci
* fix flags for nsync in wasm build
* add copyright banner
* fix wasm source glob
* add missing exports
* resolve comments
* Perf gain by make packb wide to 4 from 16 on GEMM for WASM.
Remove no need direct conv in previous perf tuning.
* fix buildbreak introduced from latest master merge
* fix buildbreak in mlasi.h
* resolve all comments except MLAS
* rewrite packb related 3 functions for WASM_SCALAR seperately rather than using #ifdef in each.
and other changes according to PR feedback in mlas.
* More complete scalar path in sgemm from Tracy.
* Fix edge case handling in depthwise conv2d kernel 3x3. where:
*) support input W==1 and H==1
*) recalc in accurate pad_right and pad_bottom
*) support hidden pad_right == 2 or pad_bottom == 2 when W == 1 or H==1 and no pad left/top
* Add more test coverage for conv depthwise from Tracy.
Fix one typo according to PR.
* resolve comments
* replace typedef by using
* do not use throw in OrtRun()
* output error message
Co-authored-by: Sunghoon <35605090+hanbitmyths@users.noreply.github.com>
Co-authored-by: Lei Zhang <zhang.huanning@hotmail.com>
Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>
Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com>
Co-authored-by: David Medine <david.eric.medine@gmail.com>
Co-authored-by: David Medine <david.medine@brainproducts.com>
Co-authored-by: Ori Levari <ori.levari@microsoft.com>
Co-authored-by: Ori Levari <orlevari@microsoft.com>
Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com>
Co-authored-by: Chen Fu <chenfucs@gmail.com>
* Enabled rocm support for graph transformations
* Support for external Hip allocator
* Added const_cast to reinterpret_cast to fix compiler issue
* Another crack at fixing the compile error
* More compilation fixes
* Added compilation flags to load_inline extension
* Added ROCM, ROCM_PINNED constants
* Changes to address PR comments
* Changed gpu identifier from ROCM to CUDA
* Added HIP compilation flag for torch inline functions
* Fixed a typo in header allocator string formatting
* Fix for runtime error with external_cuda_allocator
* Removed cuda/rocm specific code paths for allocators
* More name changes to generic gpu from rocm/cuda
* Removed duplicate allocator creation
* Rename cuda_external_ config options as gpu_external_
* Rename hip_mem_limit to gpu_mem_limit
* Rename cuda_mem_limit to gpu_mem_limit
* Integrate openvino-ep-2021.3
* operators type
* changed the myriad as it is case sensitive
* logging information for openvino-ep-2021.3
* Unit test fix
* Resize operator added for myriad
* Fixed python tests for CPU and GPU
* data commit for loop tile and gatherelements failure
* adding checks for Where
* fixing gatherelements and loop tests
* disabling instance normalization test for now as there seems to be a
myriad bug, putting loop in ops supported only because all the tests
fail
* gather elements op test taking care of warning message
* condition needs to be an intializers
* Disabled python test for Myriad
* Disable compilation warning for MSVC windows compiler
* softmax_test, threedimaxis0 and 1 test give accuracy mismatch
tensoroptest disables test gives accuracy mismatch
gather test gives accuracy mismatch
* Updated with ov version 2021.3
* Updated with ov version 2021.3
* Updated README
* Disabling python tests for cpu
* Disabling python tests with accuracy mismatch on cpu
* Added fix for Linux CI Pipeline failure
-> Disabled tests that were throwing segfault
Co-authored-by: sfatimar <sahar.fatima@intel/com>
Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com>
Co-authored-by: Aravind <aravindx.gunda@intel.com>
* Add function body to SoftmaxGrad schema
* Add type context and cleanup
* Add test case with symbolic dimensions
* Add opset specification to function
* handle opset dependence
* Exclude from minimal build
* Code refactor
* Modify code to tackle OOM when calibrating on larget dataset
* Fix mismatch issue when setting keepdims on ReduceMin/ReduceMax
* Add COCO val 2017 annotation
* Fix mismatch issue when setting keepdims on ReduceMin/ReduceMax
* Fix bug of "No module named:onnxruntime.quantization.CalTableFlatBuffers"
* Check and install flatbuffers module
* Add script to donwload coco dataset image and refactor example
* Fix bug of "No module
named:onnxruntime.quantization.CalTableFlatBuffers"
* Add CalTableFaltBuffers as module
* Remove annotation, user can download by themselves.
* Uncommet code
* Add back instances_val2017.json
* Make sure flatbuffers installed when ORT is installed
* Refactor code to call coco api
* Enable FP16 for example
Changes include:
* Revert Event Pool changes
* Add copyright and revert unrelated changes
* Add DLPack as submodule and remove to_dlpack and from_dlpack from public API
* Update golden numbers for DHP Parallel tests
* Update ORTTrainer unit test numbers
* Rollback to DLPack v0.3
* Disable flaky test
* Update third party notices and CG manifest file
* Minor refactoring of ORTValue API
1. Remove openmp related packaging pipelines and build jobs.
2. Set continueOnError to true for the TSAUpload tasks. Their service is unstable recently.
3. Update Ubuntu 16 docker images to Ubuntu 18, in prepare for getting C++17 support
4. Cherry-pick the changes in 1.7.1 to the master: updating CFLAGS/CXXFLAGS to strip out debug symbols
Various updates to the int8_t GEMMs:
1) Add ARM64 udot kernel to take advantage of dot product instructions available in newer cores. Some models run 4x faster than the stock implementation we used before.
2) Refactor the x64 kernels to share common code for AVX2(u8u8/u8s8/avxvnni) vs AVX512(u8u8/u8s8/avx512vnni) to reduce binary size.
3) Extend kernels to support per-column zero points for matrix B. This is not currently wired to an operator.
* Added code for Relugrad with GPU support.
Signed-off-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com>
* Add GPU support for DNNL ConvGrad
Signed-off-by: George Nash <george.nash@intel.com>
* Add GPU support for DNNL MaxPoolGrad
Updates to MaxPool for training with GPU
Update oneDNN to version 1.8.1
Signed-off-by: George Nash <george.nash@intel.com>
* Fixed issues found durring code review
- error in code comment
- using auto when the direct type would have been better
- removed ternary operators that were returning bool values
Signed-off-by: George Nash <george.nash@intel.com>
Co-authored-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com>
Implement an alternate workaround for the LLVM x86 problem described in PR #5088. That change made the x86 assembly files build with the GNU assembler by using -fno-integrated-as
* Fixed issue in python cmake to update wheel package
* Fixes python cmake issue for OV EP
Added post build step for libonnxruntime_providers_openvino
that copies the updated libonnxruntime_providers_openvino.so file
to /onnxruntime/capi directory every time this target is rebuilt.
Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>
* Removed post_build step from onnxruntime_python.cmake
Now that we have added the post build step to copy
onnxruntime_providers_openvino.so and providers_shared.so
to /onnxruntime/capi directory in onnxruntime_providers.cmake file.
so removing the duplication of the same from here.
Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>
* Fixed python cmake issue for OpenVINO-EP
->Fixed issue for both Linux and windows
Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>
Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com>
Add providers for CoreML, ROCM, NNAPI, ArmNN
Adding the structs for OrtCUDAProviderOptions and OrtOpenVINOProviderOptions
Updating NNAPI flags.
Adding the new CoreML flag.
Adding hooks to the build system to tell Java about the new providers.
* If unit tests are manually excluded via `--cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=OFF` (e.g. testing changes to binary size where you want to keep the build time as quick as possible) it should still be possible to create the python bindings.
Update CMakeLists.txt to decouple the inclusion of onnxruntime_python.cmake from unit tests being enabled.
Update onnxruntime_python.cmake so it works when unit tests are disabled. Also skip copying of test py files when unit tests are disabled.
* Add robust dependency check for Python package
* Add version_info.py to .gitignore
* Fix Linux build
* Fix Windows CPU build
* Fix Windows 32-bit build
* Minor tweak
* Generate version_info.py earlier in onnxruntime_python.cmake
* Print a user-friendly message if cuDNN is not found in
* Relax version requirements for CUDA 11 - only the major version has to match
* Fix PATH environment variable to include CUDA 11 in 'Python packaging pipeline' (Windows/GPU)
* Fix the build with cuDNN 7
* Remove support from custom ops from the base minimal build as they contribute too much binary growth to an Android build.
Add ability to explicitly enable custom op support in a minimal build.
Change one minimal build CI to test adding custom op support (unit tests are run in that build to validate)
* model building
* fix build
* winml adapter model building api
* model building
* make build
* make build again
* add model building with audio op
* inplace and inorder fft
* add ifft
* works!
* cleanup
* add comments
* switch to iterative rather than recursive and use parallelization
* batched parallelization
* fft->dft
* cleanup
* window functions
* add melweightmatrix op
* updates to make spectrogram test work
* push latest
* add onesided
* cleanup
* Clean up building apis and fix mel
* cleanup
* cleanup
* naive stft
* fix test output
* middle c complete
* 3 tones
* cleanup
* signal def new line
* Add save functionality
* Perf improvements, 10x improvement
* cleanup
* use bitreverse lookup table for performance
* implement constant initializers for tensors
* small changes
* add matmul tests
* merge issues
* support add attribute
* add tests for double data type windowfunctions and minor cleanup
* stft onesided/and not tests
* cleanup
* cleanup
* clean up
* cleanup
* remove threading attribute
* forward declare orttypeinfo
* warnings
* fwd declare
* fix warnings
* 1 more warning
* remove saving to e drive...
* cleanup and fix stft test
* add opset picker
* small additions
* add onnxruntime tests
* add signed/unsigned
* fix warning
* fix warning
* finish onnxruntime tests
* make windows namespace build succeed
* add experimental flag
* add experimental api into nuget package
* add experimental api build flag and add to windows ai nuget package
* turn experimental for tests
* add minimum opset version to new experimental domain
* api cleanup
* disable ms experimental ops test when --ms_experimental is not enabled
* add macro behind flag
* remove unused x
* pr feedback
Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
hipFFT is transitioning to a separate repository (away from being
included in rocFFT). During this transition, using the hipFFT version
of hipfft.h won't produce a deprecation warning.
* Generate error when an explicit stream argument is not provided in the <<<...>>> kernel launch syntax
Co-authored-by: Weixing Zhang <wezhan@microsoft.com>
* Support to allow user to specify compute stream per session
Create computation cuda stream explicitly rather than use default legacy stream or per-thread default stream.
remove some redudant cudaStreamSynchronize
fix gpt2 model test failures
don't use default stream in nccl either.
add stream schronization in OnRunEnd()
using cub::DeviceScan::InclusiveSum which can be called with stream specified.
fix topK failure due to latest rebase
fix tensorrt
support user specified stream
add user_stream support in tensorrt EP
use same stream for both tensort and CUDA EP.
fix ScatterND
specify stream for adasum and p2p kernels.
fix loop
fix CApiTest.custom_op_handler
fix CApiTest.varied_input_custom_op_handler
change for cudaMemcpyFromSymbol
improve provider options for user specified compute stream
* add changes for ROCM EP
* fix GatherGrad UT for ROCM EP
* clean code and fix NonMaxSuppression
* use default stream for ROCM now
* fix CApiTest.custom_op_handler:OrtFormatCustomOpTests.ConvertOnnxModelToOrt
* fix tensorrt ut: CApiTest.io_binding_cuda
Co-authored-by: Weixing Zhang <wezhan@microsoft.com>
