mirror of
https://github.com/saymrwulf/pytorch.git
synced 2026-05-14 20:57:59 +00:00
371f587c92
This is to minimize network flakiness when running lint jobs. I create a new Docker image for linter and install all linter dependencies there. After that, all linter jobs are converted to use Nova generic Linux job https://github.com/pytorch/test-infra/blob/main/.github/workflows/linux_job.yml with the new image. For the future task: I encounter this issue with the current mypy version we are using and Python 3.11 https://github.com/python/mypy/issues/13627. Fixing this requires upgrading mypy to a newer version, but that can be done separately (require formatting/fixing `*.py` files with the newer mypy version) `collect_env` linter job is currently not included here as it needs older Python versions (3.5). It could also be converted to use the same mechanism (with another Docker image, probably). This one rarely fails though. ### Testing BEFORE https://github.com/pytorch/pytorch/actions/runs/4130366955 took a total of ~14m AFTER https://github.com/pytorch/pytorch/actions/runs/4130712385 also takes a total of ~14m Pull Request resolved: https://github.com/pytorch/pytorch/pull/94255 Approved by: https://github.com/ZainRizvi |
||
|---|---|---|
| .. | ||
| amd_build | ||
| autograd | ||
| bazel_tools | ||
| build_defs | ||
| code_analyzer | ||
| code_coverage | ||
| config | ||
| coverage_plugins_package | ||
| dynamo | ||
| fast_nvcc | ||
| gdb | ||
| iwyu | ||
| jit | ||
| linter | ||
| lite_interpreter | ||
| lldb | ||
| onnx | ||
| pyi | ||
| rules | ||
| setup_helpers | ||
| shared | ||
| stats | ||
| test | ||
| testing | ||
| __init__.py | ||
| bazel.bzl | ||
| BUCK.bzl | ||
| BUCK.oss | ||
| build_libtorch.py | ||
| build_pytorch_libs.py | ||
| download_mnist.py | ||
| extract_scripts.py | ||
| gen_flatbuffers.sh | ||
| gen_vulkan_spv.py | ||
| generate_torch_version.py | ||
| generated_dirs.txt | ||
| git_add_generated_dirs.sh | ||
| git_reset_generated_dirs.sh | ||
| nightly.py | ||
| nvcc_fix_deps.py | ||
| pytorch.version | ||
| README.md | ||
| render_junit.py | ||
| substitute.py | ||
| update_masked_docs.py | ||
| vscode_settings.py | ||
This folder contains a number of scripts which are used as
part of the PyTorch build process. This directory also doubles
as a Python module hierarchy (thus the __init__.py).
Overview
Modern infrastructure:
- autograd - Code generation for autograd. This includes definitions of all our derivatives.
- jit - Code generation for JIT
- shared - Generic infrastructure that scripts in
tools may find useful.
- module_loader.py - Makes it easier to import arbitrary Python files in a script, without having to add them to the PYTHONPATH first.
Build system pieces:
- setup_helpers - Helper code for searching for third-party dependencies on the user system.
- build_pytorch_libs.py - cross-platform script that builds all of the constituent libraries of PyTorch, but not the PyTorch Python extension itself.
- build_libtorch.py - Script for building libtorch, a standalone C++ library without Python support. This build script is tested in CI.
- fast_nvcc - Mostly-transparent wrapper over nvcc that
parallelizes compilation when used to build CUDA files for multiple
architectures at once.
- fast_nvcc.py - Python script, entrypoint to the fast nvcc wrapper.
Developer tools which you might find useful:
- git_add_generated_dirs.sh and git_reset_generated_dirs.sh - Use this to force add generated files to your Git index, so that you can conveniently run diffs on them when working on code-generation. (See also generated_dirs.txt which specifies the list of directories with generated files.)
Important if you want to run on AMD GPU:
- amd_build - HIPify scripts, for transpiling CUDA
into AMD HIP. Right now, PyTorch and Caffe2 share logic for how to
do this transpilation, but have separate entry-points for transpiling
either PyTorch or Caffe2 code.
- build_amd.py - Top-level entry point for HIPifying our codebase.
Tools which are only situationally useful:
- docker - Dockerfile for running (but not developing) PyTorch, using the official conda binary distribution. Context: https://github.com/pytorch/pytorch/issues/1619
- download_mnist.py - Download the MNIST dataset; this is necessary if you want to run the C++ API tests.