Haoci Zhang 8fe5b93667 support zb1p and zb2p algorithms (#130752) ... Previously, we have proved that ZB2P is not truly zero bubble when num_local_stages exceed 4 and so only ZB1P was supported. We did a few tweaks to the ZB2P to really make it zero bubble. Algorithm and proof is attached. [zero_bubble.pdf](https://github.com/user-attachments/files/16238738/zero_bubble.pdf) Pull Request resolved: https://github.com/pytorch/pytorch/pull/130752 Approved by: https://github.com/H-Huang		2024-07-24 17:58:46 +00:00
..
_static	Clean up distributed/CONTRIBUTING.md (#128450)	2024-06-22 02:41:22 +00:00
_templates
community	Add Alban and Piotr into Core Maintainers (#130903)	2024-07-20 16:02:42 +00:00
elastic	DOC: add docstring to construct_and_record_rdzv_event() (#128189)	2024-06-10 22:17:33 +00:00
notes	Add torch.serialization.safe_globals context manager (#127939)	2024-07-12 20:38:43 +00:00
rpc
scripts	[BE][Easy][1/19] enforce style for empty lines in import segments (#129752)	2024-07-16 00:42:56 +00:00
amp.rst	add xpu for amp (#127276)	2024-06-20 21:49:35 +00:00
autograd.rst
backends.rst	[cuDNN][SDPA] Remove TORCH_CUDNN_SDPA_ENABLED=1, enable cuDNN SDPA by default on H100 and 2nd on other archs >= sm80 (#125343)	2024-06-30 19:22:16 +00:00
benchmark_utils.rst	Adding Compare in torch.utils.benchmark documentation (#125009)	2024-05-03 00:50:54 +00:00
bottleneck.rst
checkpoint.rst	[checkpoint] Clean up selective activation checkpoint and make public (#125795)	2024-06-18 18:18:50 +00:00
complex_numbers.rst
cond.rst
conf.py	[BE][Easy][1/19] enforce style for empty lines in import segments (#129752)	2024-07-16 00:42:56 +00:00
config_mod.rst
cpp_extension.rst
cpp_index.rst
cpu.rst
cuda._sanitizer.rst
cuda.rst	Revert "Add wrappers for synchronous GPUDirect Storage APIs (#130633)"	2024-07-23 17:19:34 +00:00
cuda.tunable.rst	[ROCm] TunableOp improvements (#124362)	2024-06-03 22:30:11 +00:00
cuda_environment_variables.rst
cudnn_persistent_rnn.rst
cudnn_rnn_determinism.rst
data.rst
ddp_comm_hooks.rst
debugging_environment_variables.rst
deploy.rst
deterministic.rst
distributed.algorithms.join.rst
distributed.checkpoint.rst	[DCP] Provides default AsyncStager (#124939)	2024-05-02 19:48:54 +00:00
distributed.elastic.rst	Reapply "distributed debug handlers (#126601)" (#127805)	2024-06-04 19:44:30 +00:00
distributed.optim.rst
distributed.pipelining.rst	support zb1p and zb2p algorithms (#130752)	2024-07-24 17:58:46 +00:00
distributed.rst	Retire torch.distributed.pipeline (#127354)	2024-06-07 08:11:58 +00:00
distributed.tensor.parallel.rst
distributions.rst
dlpack.rst
docutils.conf
export.ir_spec.rst
export.rst	[export] minor typo fix (#129543)	2024-06-26 18:35:31 +00:00
fft.rst
fsdp.rst
func.api.rst
func.batch_norm.rst
func.migrating.rst
func.rst
func.ux_limitations.rst
func.whirlwind_tour.rst
future_mod.rst
futures.rst
fx.experimental.rst	Traceable wrapper subclass support for deferred runtime asserts (#126198)	2024-05-21 01:21:46 +00:00
fx.rst	Implement Graph Transform Observer (#127427)	2024-06-02 06:49:47 +00:00
hub.rst
index.rst	Retire torch.distributed.pipeline (#127354)	2024-06-07 08:11:58 +00:00
jit.rst
jit_builtin_functions.rst
jit_language_reference.rst
jit_language_reference_v2.rst
jit_python_reference.rst
jit_unsupported.rst
jit_utils.rst
library.rst	[custom ops] Add register_vmap for custom ops (#130589)	2024-07-23 17:48:38 +00:00
linalg.rst
logging.rst
masked.rst
math-quantizer-equation.png
meta.rst
miscellaneous_environment_variables.rst	[RFC] Add support for device extension autoloading (#127074)	2024-07-09 06:14:13 +00:00
mobile_optimizer.rst
model_zoo.rst
module_tracker.rst	Add module tracker (#125352)	2024-05-04 18:33:35 +00:00
monitor.rst
mps.rst	Add support in Python API for the recommended max working set size. (#128289)	2024-06-12 16:03:57 +00:00
mps_environment_variables.rst	[MPS] Add mps profiler env vars to docs (#129552)	2024-07-04 06:44:48 +00:00
mtia.rst	[torch.mtia] Noop set_rng_state and get_rng_state APIs (#130873)	2024-07-24 01:52:21 +00:00
multiprocessing.rst
name_inference.rst
named_tensor.rst
nested.rst
nn.attention.bias.rst
nn.attention.flex_attention.rst	Revert "Added and_masks and or_masks utilities (#131073)"	2024-07-22 13:44:55 +00:00
nn.attention.rst	Make FlexAttention API public (#130755)	2024-07-16 16:21:25 +00:00
nn.functional.rst
nn.init.rst
nn.rst
onnx.rst
onnx_dynamo.rst
onnx_dynamo_onnxruntime_backend.rst
onnx_torchscript.rst
onnx_torchscript_supported_aten_ops.rst
optim.rst	[MPS] Fused SGD optimizer (#129350)	2024-06-27 04:37:14 +00:00
package.rst
profiler.rst
quantization-accuracy-debugging.rst
quantization-backend-configuration.rst
quantization-support.rst	Add numeric_debugger top level APIs (#130643)	2024-07-18 20:54:18 +00:00
quantization.rst
random.rst
rpc.rst
signal.rst
size.rst
sparse.rst	Define 'zero-preserving unary functions' in docs (#130804)	2024-07-18 13:30:29 +00:00
special.rst
storage.rst
tensor_attributes.rst	Refine the logic of device construction when only device index is given (#129119)	2024-07-15 14:34:29 +00:00
tensor_view.rst
tensorboard.rst
tensors.rst	add xpu to torch.tensors (#127280)	2024-06-11 18:13:01 +00:00
testing.rst
threading_environment_variables.rst
torch.ao.ns._numeric_suite.rst
torch.ao.ns._numeric_suite_fx.rst
torch.compiler.rst	add xpu to torch.compile (#127279)	2024-06-13 21:15:09 +00:00
torch.compiler_aot_inductor.rst	[AOTI] docs: add suggestion to turn on freezing on CPU (#128010)	2024-06-07 08:57:02 +00:00
torch.compiler_api.rst
torch.compiler_best_practices_for_backends.rst
torch.compiler_cudagraph_trees.rst	[CUDAGraph] add more docs for cudagraph trees (#127963)	2024-06-18 02:07:07 +00:00
torch.compiler_custom_backends.rst	Fix a link in the compiler backend doc (#126079)	2024-05-21 20:16:04 +00:00
torch.compiler_dynamic_shapes.rst
torch.compiler_dynamo_deepdive.rst	Stop immediately specializing common constants 0/1 for plain int (#128327)	2024-07-03 16:41:51 +00:00
torch.compiler_dynamo_overview.rst	Rename TorchDynamo -> Dyanamo in the dynamo tutorial doc (#123431)	2024-05-07 05:07:00 +00:00
torch.compiler_fake_tensor.rst	[Docs] Fix fake tensor doc (#131205)	2024-07-19 17:59:45 +00:00
torch.compiler_faq.rst	Fixed broken link and removed unfinished sentence from issue #126367 (#127938)	2024-06-05 07:37:32 +00:00
torch.compiler_fine_grain_apis.rst
torch.compiler_get_started.rst	add xpu to torch.compile (#127279)	2024-06-13 21:15:09 +00:00
torch.compiler_inductor_profiling.rst
torch.compiler_ir.rst
torch.compiler_nn_module.rst
torch.compiler_performance_dashboard.rst
torch.compiler_profiling_torch_compile.rst
torch.compiler_transformations.rst
torch.compiler_troubleshooting.rst	Add force_disable_caches to the docs (#126184)	2024-05-15 07:16:08 +00:00
torch.overrides.rst
torch.rst	Introduce the concept of Accelerators to PyTorch doc (#129363)	2024-07-15 14:24:46 +00:00
torch_cuda_memory.rst
torch_environment_variables.rst	[Docs][MPS] Add mps environment variable table (#129008)	2024-06-20 03:30:35 +00:00
torch_nccl_environment_variables.rst	[c10d][doc] add a doc page for NCCL ENVs (#128235)	2024-06-09 16:08:38 +00:00
type_info.rst
utils.rst
xpu.rst