* Prototype NCCL P2P
* Clean code
* Fix NCCL path and some minor bugs
* Add path
* Fix path
* Try fix path
* Add missed files
* Address some comments
* Clean code
* Rename files
* Add MPI path back and fix a path
* Put MPI path under USE_NCCL flag
* not to build Send and Recv when MPI is not installed
* match new/old api numbers
* new golden numbers for Roberta and MC
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* opset13 cuda kernels for BERT.
* add opset13 SoftmaxCrossEntropyLoss.
* opset13 size.
* fix argmax/min for ut.
* fix ut failure for argmax/min.
* OrtMemTypeCPUInput
Co-authored-by: Vincent Wang <weicwang@microsoft.com>
This PR includes:
* Previous CODEOWNERS was encompassing more files than just training files
* Polynomial optimizer config is missing part of its docstring
* add deterministic path for reduce l2
* add unit tests
* memset zero size off by one
* eliminate windows warning as error
Co-authored-by: suffian khan <sukha@OrtTrainingDev1.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* Changes to enable saving and loading an ORT format model via the public APIs.
Cleanup session.py to try and make slightly more understandable. More refactoring is needed here.
Couple of bug fixes
* Fix bug in handling NodeArg serialization for optional inputs which has a name and no type info.
* Address PR comments
- tweak SessionOptions config to avoid double lookup
- merge duplicated functionality in python binding around registering an EP with optional options
Fix a couple of build issues.
* Update C API to be consistent with python API
- only load model in InferenceSession ctor if required
- support loading ORT model in minimal build
* Fix nodejs test.
We get an invalid path error from LoadInterOp first now
* Another attempt at fixing nodejs test.
Error message depends on whether ENABLE_LANGUAGE_INTEROP_OPS is defined. Make the output consistent.
The interop implementation looks suspicious given it appears to be internal code that is going via the public api. TBD if that should be fixed.
* Fix couple of build issues.
* Disable test temporarily so PR can be checked in.
Will fix in separate PR that adds final pieces for minimal build as the test is required there.
* Give up on nodejs test and make the match simpler.
Fix init call in TrainingSession python to not pass through sess. it wasn't being used in Session anyway so passing it through just adds confusion.
* Fix call to Session.__init__ in TrainingSession.
Session now initializes Session._sess to None to make it clearer where the 'ownership' of that member is, and that needs to happen before TrainingSession sets it.
This PR also includes:
* More LossScaler tests
* Minor LossScaler improvement
* Check model after extra post processing
* Improve basic training tests to include all optimizers
* Set rtol=1e-7 tolerance for Legacy vs Experimental frontend API tests
* Increase number of training tests for Legacy vs Experimental tests
* Minor refactoring on existing tests
* Fix Checkpoint API for Gradient Accumulation / fp16 scenarios
https://github.com/microsoft/onnxruntime/pull/4639 changed the default
behavior by removing optimizer state from state_dict/checkpoint APIs.
The reason for the previous change was to allow models trained on ORT to
be used for inference on PyTorch, which is an important feature.
Due to the change aforementioned, when resuming training from a checkpoint,
the optimizer would start with random weights, leading to a bad performance.
This behavior would also cause reproducibility issues, as the optimizer
wouldnt be able to resume from its previous state.
This PR adds a boolean flag to state_dict/save_xheckpoint API that
when True (default) it saves both model and optimizer state.
When False, only the model state is kept.
Disabling this test until it's intermittent failure is root caused, this is a function and does not have a dedicated op by itself. However, this op is not used in known model to the best of my knowledge to disabling this test for the sanity of CI until the investigation is over is probably reasonable.
* regsiter part of opset13 cpu kernels; fix a bug in func impl; adjust reshapefusion order
* remove useless function
Co-authored-by: Cheng Tang <chenta@microsoft.com>
* Optimized MatMulGrad for dB when B's shape is 2D
* Refactor for ConstantScalarNode
Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* matching multiple choice between new and old apis
* update according to reviewer's comments
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* Copy samples to build folder and load models from there. Fix CI
* This PR also includes a fix to path validation for save_as_onnx API
* Add torchtext to CI for GPU training
* Remove new frontend tests from CI
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>
* Add support for sharing allocators
* Incremental update
* Address some PR comments, add unit tests, add documentation.
* Address PR comments, add tests and some documentation.
* Fix build and test issues
* Remove RegisterAllocator API restoring the OrtAllocator interface changes. Changed docs to reflect this.
Also fixed the orttraining segfault. The segfault was because in the case of training session,
the CPU exec prov is not available at the time the transformers are applied. Changed it to create
a new one.
* create branch for debug
* move unit test
* more changes
* move relu to activations_grad*
* Fix ReluGrad Domain and opset version
* added unit test, CudaKernelTest.Relu_basic doesn't work yet
* remove CudaKernelTest.Relu_basic
* PR comment
* add unit test ReluGradTest_Basic
Co-authored-by: Jingyan Wang <jingywa@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
This PR also includes:
* Remove defaults from named tuples to support python 3.6
* Allows model which takes dicts as input
* Adapts BERT finetuning example to run on the new frontend
* Match numbers for BERT fine tuning model
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>
* Port legacy checkpoint API into new front-end
This PR also fixes:
* Warnings on ORTTrainer for improper tensor copies
* Inaccurate LRScheduler tests using wrong LR
* Stale DeepSpeed documentation
* Minor code refactoring for Toy BERT tests
* Move experimental state_dict() and load_state_dict() into checkpoint ns
* adding generic configurations for session options
* fix a build break on linux
* fix training ci build break
* fix training ci build break
* addressed CR comments
* fix traning ci build break
* move config_key from enum to string
* add c# api
* add python api
* fix build break
* move prepacking from 2 new api entries to session options configs
* fix traning ci build break
* add python test, update some comments, move const key definition to avoid build break
* addressed comments
* move definitions of keys to common.h
* move api to version 5
* remove accidental change in build.py
* remove pragma to avoid build break
* addressed CR comments
* fix the python build break, and move location of config keys definition
* small typo changes
`LossScaler.update()` was not being properly called due to the incorrect TrainStepInfo.all_finite assignment.
Additionally to this fix, _ORTTrainerModelDesc.is_finite was renamed to _ORTTrainerModelDesc.all_finite to make it more uniform with TrainStepInfo
