Thiago Crepaldi
83c3da3fc0
Fix code-base after breaking API changes
2020-03-31 17:59:20 -07:00
Weixing Zhang
1bbc421884
Don't cast to fp16 in LayernormGrad ( #3328 )
...
Co-authored-by: Weixing Zhang <wezhan@microsoft.com>
2020-03-28 19:07:32 -07:00
Sherlock
ffb2a3359e
Implement WhereGrad ( #3343 )
2020-03-27 19:10:40 -07:00
Tixxx
49e6043d07
support Huggingface's adamw ( #3318 )
...
* add weight decay mode to support both pytorch and huggingface's adamw
2020-03-27 08:04:27 -07:00
ytaous
131c65d23d
Addressing PR comments ( #3334 )
...
* PR comments
* PR comments
* PR comments
* error out bad shape
Co-authored-by: Ethan Tao <ettao@microsoft.com>
2020-03-26 18:43:30 -07:00
Xueyun Zhu
0a6ec0df56
Merge pull request #3285 from microsoft/xuzhu/merge_from_master
...
Merge from master to ort_training
2020-03-26 12:10:13 -07:00
Sherlock
d143b41b81
Expose frozen_weights in PyTorch Frontend ( #3317 )
2020-03-26 11:26:54 -07:00
Wei-Sheng Chin
b38fc0d541
Add bias correction in Adam & Lamb for C++ frontend & python frontend ( #3301 )
2020-03-25 09:46:44 -07:00
Xueyun Zhu
e9877850a4
fix python error
2020-03-25 01:59:37 +00:00
Bowen Bao
6474801ceb
Update ort_trainer.py with lazy onnx export ( #3244 )
...
* Delay onnx export to avoid extra info
* handle cases where onnx model is provided at initialization
* address comments
* fix rebase error
2020-03-24 13:34:15 -07:00
Li-Wen Chang
98c28060b0
Aggregated Send/Recv ( #3232 )
...
* Aggregated Send/Recv
* fix typos
* CR refine
* CR refine
* CR refine
* Add scalar check.
* typo
* reformat
* CR refine
* Forgot to swap order in the implementation after spec changed
* CR refine
* Cr refine
* add Send's input type checking
2020-03-24 10:20:11 -07:00
KeDengMS
d15c74e713
Implement pipeline event generator ( #3206 )
...
Implement pipeline event generator with OneFWOneBW schedule in timeline. Each stage of pipeline contains FW and BW of a subset of the model and are scheduled in one worker thread for each microbatch.
2020-03-23 17:32:54 -07:00
Xueyun Zhu
8f7bd51f7a
fix pybind issue introduced by merge
2020-03-23 23:23:34 +00:00
Tixxx
7f610caca0
Make gradient clipping configurable. ( #3243 )
...
* Make gradient clipping configurable.
add control flag to c++ and python frontend
2020-03-23 12:21:48 -07:00
Xueyun Zhu
9dbc50c438
fix build break
2020-03-21 02:16:00 +00:00
liqunfu
d521efd904
refactor frontend ( #3235 )
...
* refactor frontend
* remove training python files from inferencing build
* update according to reviewer's comments
* merge pybind_state.cc
* refactor pybind_state.cc
* code clean up
* missed a forward declaration in ort_pybind_state.cc
* passed pytest
* move training_session.py into a subfolder per reviewer's comment
* add copyright
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
2020-03-19 20:59:41 -07:00
edgchen1
d9f628cb1d
Remove orttraining/tools/scripts/profile directory. ( #3268 )
2020-03-19 14:13:05 -07:00
edgchen1
61e8a24340
Address PR comments ( #3255 )
...
* Added comment for ntfw_remove().
* Rewrite WindowsEnv::DeleteFolder(), some other clean up.
2020-03-18 17:57:57 -07:00
edgchen1
c5576d70a6
Fix build issues ( #3214 )
...
* Fixed issues with Python and inference-only build.
* Handle ImportError for training imports.
* fix windows build
* fix compile error
* fix centos build
* fix windows build
* fix compile error
* Use SafeInt for allocation calculation, fix typo.
Co-authored-by: Ethan Tao <ettao@microsoft.com>
2020-03-17 16:10:23 -07:00
Sherlock
4b2c8e884e
Udpate License Header ( #3212 )
2020-03-16 10:24:31 -07:00
Jesse Benson
3a7539e071
Update bert-base convergence values
2020-03-13 23:03:34 -07:00
Jesse Benson
dc11b82956
Tweak the dropout calculation.
2020-03-13 23:03:34 -07:00
Ethan Tao
2f1e997e5b
Merged PR 5686: fix P100/fp16 issues
...
1. misaligned address in atomic_add()
2. GatherNDGradKernel to use atomic_add
3. enable/add UTs for GatherNDGrad and reduction_ops using half
- __CUDA_ARCH__ won't take effect on .cc code, leverage HasCudaEnvironment() instead
4. verified convergence graph and perf test
- p100 is much slower than v100 on fp16
- fp16/128 need to reduce batch size from 66 to 64 to avoid OOM issue
5. verify convergence test on Dev3/v100
TBD - broken UTs related to MatmulIntegerOpTest (works on v100/windows, though)
2020-03-12 16:51:45 -07:00
Ke Deng
75025461e2
Initial implementation of graph cut and pipeline
...
This is a draft of graph cut and wait/record to demonstrate cut and Wait/Record design. You may find sub models and profiling json under onnxruntime/test if you run "onnxruntime_test_all --gtest_filter=GradientGraphBuilderTest.TrainingSession_WithPipeline"
2020-03-12 16:51:45 -07:00
Edward Chen
80dd62a240
Enable CI for training.
2020-03-11 14:41:32 -07:00
Edward Chen
e542cfd0e0
Introduce training changes.
2020-03-11 14:39:03 -07:00
