* TopKGrad CPU kernel
* use Scatter for GatherElementsGrad and TopKGrad.
* rollback convgrad change.
Co-authored-by: Vincent Wang <weicwang@microsoft.com>
* Add type inference for BroadcastGradientArgs
This change enables the ONNX shape and type inference to work on a function body containing a BroadcastGradientArgs op. Without this change, the dummy inference function is used, and no types are inferred for the output here:
531e6dd459/onnx/shape_inference/implementation.cc (L467-L469)
* Handle optional outputs.
* Created shared version of InferenceSession wrapper class and update relevant tests to use it.
Include domain in the ops counting helper so it's more general and we don't need to duplicate it in the nchwc tests. Update tests to include domain in key being checked.
* Fix some training tests
* Fix prefixing of contrib op names in test
* Add CUDA option to run copy in default stream
This change fixes#4829. Thanks @maherzog for providing the repro!
The bug is caused by memory reuse in BFC arena, where copy and
compute stream in CUDA has a racing condition.
BFC arena is an arena allocator on top of cudaMalloc/Free to
reduce the cost in syncing CPU and GPU when alloc/free. It means
when CPU alloc/free the memory, GPU might not finished previous
work on the memory, so that CPU and GPU could run asynchronously.
This is OK if there's only one stream, where the execution order
in CPU and GPU are consistent. For example, if we have two kernels
A and B, CPU runs allocA->computeA->freeA->allocB->computeB->freeB,
A and B could shares the same memory since computeA and computeB
will not have racing as long as they run in the same GPU compute
stream.
However, if CPU runs allocA->CopyA->freeA->allocB->computeB->freeB,
the order of execution in GPU could have copyA happen after computeB,
if copy and compute happens in different GPU streams.
This change makes copy to run in default compute stream, while adding
an option to fall back to previous behavior if there's perf hit. This
is a short term fix before BFC arena could support multiple streams.
User may use following options to revert to previous behavior:
C API:
struct OrtCUDAProviderOptions cudaProviderOpt;
cudaProviderOpt.do_copy_in_default_stream = false;
C++ API:
CUDAExecutionProviderInfo cudaEPInfo;
cudaEPInfo.do_copy_in_default_stream = false;
C# API:
pending...
Python:
import onnxruntime
onnxruntime.capi._pybind_state.set_do_copy_in_default_stream(False)
* Confirmed the test failes in CI when doing copy in separate stream
Revert the test to get CI pass now
* Fix Windows test
* Address CR
* Update MaxBatchSize and include recompute mode
* Minor fix for frontend test
Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
**Description**: Add missing gradient registration for the `Exp` op.
**Motivation and Context**
* Adding support for training a model that uses the `Exp` op.
Co-authored-by: Derek Murray <demurra@microsoft.com>
* t5 layer norm changes
* add t5 layer norm kernel
* use template for t5 layer norm
* template definition changes
* no build error
* add CPU cuda kernel
* first unit test
* other forward unit tests
* add T5LayerNormGrad
* Add c++ transform and test for T5 LN
* fix and some debug prints
* fix cuda error
* rename from t5 to simplified
* PR comments
* revert change on invertible LM code path
* remove duplicate forward computation
* add GradientCheckerTest.SimplifiedLayerNormGrad
* change back macro
* Fix SimplifiedLayerNorm Gradient
* merge with Sherlockss changes
* changed cuda kernel
* reapply cpu kernel changes
Co-authored-by: Jingyan Wang <jingywa@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: aishwarya bhandare <aibhanda@microsoft.com>
Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* Avoid inserting other CUDA calls in-between NCCL Send's and Recv's
* Add a comment
* Place CUDA EP on the right device
* Fix a warning
* Address a comment
* add tensor board, remove torch.distributed.lanuch because ort nccl depends on MPI. Use MPI to launch parallel training.
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* Expose recompute configs to the frontend
* Add frontend test
* Ensure recompute graph transformer is only applied once
Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* Move fbs include from header to cc
* add initial cmake for flatbuffers
* Move most flatbuffers util to ort_flatbuffers
* move code around
* fix
* move test/perf runner to use flatbuffer directly instead of model
* minor update
* Fix build break
* Clean up includes and foward decl
* Fix traning CI build breaks
* Addressed PR comment, replaced some include with forward decls
* Remove ORT_MUST_USE_RESULT temporarily
* Build Recomputation Graph
* Make topological sort to run FW nodes first
* Pattern match start and end of transformer layer
* Topological sort with Priority
* Add logger to Gradient Graph Builder
* Use Logger
* Introduce Execution Order
* bug fix transformer
* fuse cpu kernel for transposescalematmul and matmul
* fuse transpose_scale_matmul cpu kernel with matmul
* fix test
* Add FusedMatMul Contrib Op
* fix test
* fix typo
* plus more updates per review
* Rework broadcasting setup to decrease binary size. Push all the type specific down and separate out the broadcasting/parallelization.
Reductions:
element_wise_ops: 521.0KB -> 268.8KB
where: 25.8 KB -> 17.3 KB
qlinear_binary_op: 28.1 -> 12.8
* Allow sharing of initializers between sessions.
* Allow sharing of initializers between sessions (2).
* Add test for C#
* Add test for C#; address PR comments
* Address PR comments
Moved AddInitializer logic to internal session options
Added tests for owned buffer
Clarified documentation
Fix bug where memory info and not device was getting compared
* Fix test
* Fix training build
* Add ver 5 end marker and ver 6 starter, add scenario and usage examples.
* bias softmax kernel
* bias softmax kernel
* remove debug comments
* remove debug comment
* windows build doesnt handle unary minus on unsigned type
* int64 => int treated as error
* only support cuda
* add bias softmax fusion tests
* PR comments
* more PR comments
* use MLTypeCallDispatcher
* break function into pieces
* add loop unroll and add to list for inference as well
* use std::min and move operator==
* revert std::min (doesnt work ci pipeline) and fix int to size_t error
* pr comments
* fixes for windows ci
* fix for windows ci
* pr comments on consistency
* p_model_
* fix formatting and add anonymous namespace
Co-authored-by: suffian khan <sukha@OrtTrainingDev1.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* add option, feature to orttrainer and test
* address comments
* minor fixes
* further address comments
* minor changes
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* Remove SparseTensor support from minimal build.
Currently the only valid usage of a SparseTensor is as an attribute of a Constant node. That would have been lifted to a dense tensor initializer when loading the onnx model, so would not exist when saving the ORT format model. Due to that there can be no SparseTensors in an ORT format model.
Co-authored-by: gwang <wanggy@outlook.com>
* Prototype NCCL P2P
* Clean code
* Fix NCCL path and some minor bugs
* Add path
* Fix path
* Try fix path
* Add missed files
* Address some comments
* Clean code
* Rename files
* Add MPI path back and fix a path
* Put MPI path under USE_NCCL flag
* not to build Send and Recv when MPI is not installed
