saymrwulf/pytorch
1
0
Fork 0
mirror of https://github.com/saymrwulf/pytorch.git synced 2026-05-14 20:57:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
pytorch/docs
History
fduwjj 953aa6d90e [TP] Enable more generic attn in Tensor Parallelism (#100508) 
To make TP more generic for Attention module, we come up with this new col/rowwise parallel style.

Basically, the idea behind is that:
We only do DTensor op for Col/Rowwise sharded part. For the rest of ATen ops, we will leave it to Tensor ops.

And we set this behavior as default for Colwise and Rowwise parallel style. If people want to customize it, they can always pass in different prepare_input or prepare_output

Pull Request resolved: https://github.com/pytorch/pytorch/pull/100508
Approved by: https://github.com/wanchaol
2023-05-07 18:15:49 +00:00
..
caffe2 [BE] Remove dependency on six and future (#94709) 2023-02-14 09:14:14 +00:00
cpp Replace master with main in links and docs/conf.py (#100176) 2023-05-02 18:20:32 +00:00
source [TP] Enable more generic attn in Tensor Parallelism (#100508) 2023-05-07 18:15:49 +00:00
.gitignore
libtorch.rst Replace master with main in links and docs/conf.py (#100176) 2023-05-02 18:20:32 +00:00
make.bat
Makefile
README.md
requirements.txt

README.md

Please see the Writing documentation section of CONTRIBUTING.md for details on both writing and building the docs.