Fixes various friction points with the documentation for onboarding new users and remove instructions that were no longer valid
Changes include:
- Listing prerequisites earlier, so that devs can ensure they're met before encountering error messages
- Removing linter invocations that are no longer valid
- Modifying instructions to install mkl packages to only apply to x86 based CPUs
[skip ci]
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78682
Approved by: https://github.com/seemethere, https://github.com/janeyx99, https://github.com/malfet
Hi. While looking at the contributing doc for pytorch in master. I saw a typo about installing `ccache` via `conda`. This tiny PR fixed it.
Hope this could help someone copy-paste the command faster!
Pull Request resolved: https://github.com/pytorch/pytorch/pull/70284
Approved by: https://github.com/janeyx99
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/69041
`TH_CONCAT_{N}` is still being used by THP so I've moved that into
it's own header but all the compiled code is gone.
Test Plan: Imported from OSS
Reviewed By: anjali411
Differential Revision: D32872477
Pulled By: ngimel
fbshipit-source-id: 06c82d8f96dbcee0715be407c61dfc7d7e8be47a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62550
I noticed that running the build twice in a row resulted in ~80 CUDA files being
rebuilt. Running `ninja -d explain` shows
```
ninja explain: TH/generic/THStorage.h is dirty
ninja explain: TH/generic/THStorageCopy.h is dirty
ninja explain: THC/generic/THCStorage.h is dirty
ninja explain: THC/generic/THCStorageCopy.h is dirty
ninja explain: TH/generic/THTensor.h is dirty
ninja explain: THC/generic/THCTensor.h is dirty
ninja explain: THC/generic/THCTensorCopy.h is dirty
ninja explain: THC/generic/THCTensorMath.h is dirty
ninja explain: THC/generic/THCTensorMathMagma.h is dirty
ninja explain: THC/generic/THCTensorMathPairwise.h is dirty
ninja explain: THC/generic/THCTensorScatterGather.h is dirty
```
considering `ninja` is working relative to the `build` folder, these files don't
actually exist. I traced this back to the output of `nvcc -MD` containing
paths relative to the include directory, instead of being absolute.
This adds a little script to launch the compiler then resolve any relative paths
in the `.d` file before `ninja` looks at it. To use it, I run the build with
```
export CMAKE_CUDA_COMPILER_LAUNCHER="python;`pwd`/tools/nvcc_fix_deps.py;ccache"
```
There are some possible pit-falls here. The same relative path might work for
two include directories, and the compiler could pick a different one. Or,
the compiler might have additional implicit include directories that are needed
to resolve the path. However, this has worked perfectly in my testing and it's
completely opt-in so should be fine.
Test Plan: Imported from OSS
Reviewed By: ejguan
Differential Revision: D31503351
Pulled By: malfet
fbshipit-source-id: b184c4526679d976b93829b5715cafcb1c7db2ae
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/64598
This adds a filter option rather than an all-or-nothing so it's easier to iterate on a specific job.
```bash
python tools/testing/explicit_ci_jobs.py --filter-gha '*generated-linux-*gcc5.4*'
```
See #64600 for an example usage
NB: If you regenerate the worfklows you will need to re-run that command to re-delete everything.
Test Plan: Imported from OSS
Reviewed By: janeyx99
Differential Revision: D30788850
Pulled By: driazati
fbshipit-source-id: a32c266bbd876c396665bceef9a0a961b4586564
Summary:
This PR is created to replace https://github.com/pytorch/pytorch/pull/53180 PR stack, which has all the review discussions. Reason for needing a replacement is due to a messy Sandcastle issue.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/64234
Reviewed By: gmagogsfm
Differential Revision: D30656444
Pulled By: ansley
fbshipit-source-id: 77536c8bcc88162e2c72636026ca3c16891d669a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63385
Correcting a mistake for the pytorch uninstall, and
adding an extra note for Darwin.
Test Plan: Imported from OSS
Reviewed By: janeyx99, heitorschueroff
Differential Revision: D30530234
fbshipit-source-id: e0f88a1725eeadabfb4b28c1da11e369ee878ab4
Summary:
1. In fact, Visual Studio isn't supported as CMAKE generator
2. I was asked many times why there's error as 'Could NOT find OpenMP'
3. Add Newly added Best Practices link in contributing.md
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63843
Reviewed By: seemethere, heitorschueroff
Differential Revision: D30514095
Pulled By: janeyx99
fbshipit-source-id: 76715a1d8c049122546e5a7778cafe54e4dfd5d6
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62549
When building CUDA files with native CMake support, it will respect the
`CMAKE_CUDA_COMPILER_LAUNCHER` setting. So, there's no need for symlinks.
Test Plan: Imported from OSS
Reviewed By: bdhirsh
Differential Revision: D30498488
Pulled By: malfet
fbshipit-source-id: 71c2ae9d4570cfac2a64d777bc95cda3764332a0
Summary:
This option was added in https://github.com/pytorch/pytorch/issues/61940 and fits with this section's theme of improving build times.
I've also changed it to a `cmake_dependent_option` instead of `FATAL_ERROR`ing for older CMake versions.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62827
Reviewed By: astaff
Differential Revision: D30342102
Pulled By: malfet
fbshipit-source-id: 3095b44b7085aee8a884ec95cba9f8998d4442e7
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63228
It is a quick summary and links to a page on the Developer Wiki that has
more detail.
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D30347109
Pulled By: zou3519
fbshipit-source-id: a6242986d275e5279ca3f61ade2294a132d268c4
Summary:
This PR enables installing our custom MacOS clang-tidy binaries. It also updates related documentation.
The binaries are produced by [this CI job](https://github.com/pytorch/test-infra/blob/master/.github/workflows/clang-tidy-macos.yml), and are published to S3.
This PR does not handle versioning of the downloaded binaries as this is being worked on separately. See https://github.com/pytorch/test-infra/issues/73
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62214
Test Plan:
On a MacOS machine, run
```bash
python3 -m tools.linter.install.clang_tidy
.clang-tidy-bin/clang-tidy --checks="*" --list-checks | grep "misc-max-tokens"
```
Reviewed By: janeyx99, mruberry
Differential Revision: D29917728
Pulled By: 1ntEgr8
fbshipit-source-id: 98d0d8b7a57bdebf0ebcdc83228ef391e8c6629e
Summary:
Now expecttest is an independent library but `CONTRIBUTING.md` and `requirements.txt` do not mention the need of the library.
Related: https://github.com/pytorch/pytorch/pull/60658
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61163
Reviewed By: heitorschueroff
Differential Revision: D29660296
Pulled By: ezyang
fbshipit-source-id: e2e86d42526c83bec7cdf7221e19fe83d9686103
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61119
This change spilts the clang-tidy CI job into smaller steps and uses a
refactored version of the clang_tidy.py script.
The new folder structure is as follows:
```
tools/linter/clang_tidy
|_ __main__py
|_ requirements.txt
|_ run.py
|_ setup.sh
```
`__main__.py`
This script will run `tools/linter/clang_tidy/setup.sh` if a `build`
directory doesn't exist, mimicing what used to be done as a separate
step in the CI job.
After that, it will invoke `clang-tidy` with default arguments being
declared in the script itself (as opposed to declaring them in
lint.yml).
The reasoning behind this approach is two-fold:
- Make it easier to run `clang-tidy` locally using this script
- De-duplicate the option passing
`requirements.txt`
Contains a list of additional python dependencies needed by the
`clang-tidy` script.
`setup.sh`
If a build directory doesn't exist, this command will run the necessary
codegen and build commands for running `clang-tidy`
Example usage:
```
python3 tools/linter/clang_tidy --parallel
```
Notice that we don't have to put the `.py` at the end of `clang_tidy`.
Test Plan:
Run the following command:
```
python3 tools/linter/clang_tidy --paths torch/csrc/fx --parallel
```
Reviewed By: walterddr, janeyx99
Differential Revision: D29568582
Pulled By: 1ntEgr8
fbshipit-source-id: cd6d11c5cb8ba9f1344a87c35647a1cd8dd45b04
Summary:
People don't need to self host these anymore since we do it automatically in PRs
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61080
Reviewed By: VitalyFedyunin, janeyx99
Differential Revision: D29506465
Pulled By: driazati
fbshipit-source-id: 45875cb229f8cc565a9a1405f52cef198ee0e687
Summary:
Changes including:
- introduced `linter/`, `testing/`, `stats/` folders in `tools/`
- move appropriate scripts into these folders
- change grepped references in the pytorch/pytorch repo
Next step
- introduce `build/` folder for build scripts
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60473
Test Plan:
- CI (this is important b/c pytorch/test-infra also rely on some script reference.
- tools/tests/
Reviewed By: albanD
Differential Revision: D29352716
Pulled By: walterddr
fbshipit-source-id: bad40b5ce130b35dfd9e59b8af34f9025f3285fd
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58001
Adds a script so that devs can generate a commit (at the base of a stack) that removes all CI jobs but the set that they care about. See CONTRIBUTING.md changes for usage
Test Plan: Imported from OSS
Reviewed By: mruberry
Differential Revision: D28359910
Pulled By: driazati
fbshipit-source-id: 2741570f2bab2c28f4a9d7aef727b1b2399d0ce1
Summary:
Why:
To keep VS version always updated in README
1. update VS version link in CI. It's more convenient for my PR robot to update the version in README once the VS in CI is updated. and permlink isn't stable.
2. Move `building on legacy code` to development tips. The table is big and it looks the REAMD not updated at the first sight.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56686
Reviewed By: janeyx99
Differential Revision: D28272060
Pulled By: samestep
fbshipit-source-id: 4bb879ea2914cc8bcd68343a9ed230418e1f9268
Summary:
This queries the local git repo for changed files (any changed files, not just committed ones) and sends them to mypy/flake8 instead of the default (which is the whole repo, defined the .flake8 and mypy.ini files). This brings a good speedup (from 15 seconds with no cache to < 1 second from my local testing on this PR).
```bash
make quicklint -j 6
```
It should be noted that the results of this aren’t exactly what’s in the CI, since mypy and flake8 ignore the `include` and `exclude` parts of their config when an explicit list of files is passed in.
](https://our.intern.facebook.com/intern/diff/27901577/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56559
Pulled By: driazati
Reviewed By: malfet
Differential Revision: D27901577
fbshipit-source-id: 99f351cdfe5aba007948aea2b8a78f683c5d8583
Summary:
This pulls out shell scripts from an action and runs them locally as a first pass at https://github.com/pytorch/pytorch/issues/55847. A helper script extracts specific steps in some order and runs them:
```bash
$ time -p make lint -j 5 # run lint with 5 CPUs
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'flake8-py3' \
--step 'Run flake8'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'mypy' \
--step 'Run mypy'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'quick-checks' \
--step 'Ensure no trailing spaces' \
--step 'Ensure no tabs' \
--step 'Ensure no non-breaking spaces' \
--step 'Ensure canonical include' \
--step 'Ensure no unqualified noqa' \
--step 'Ensure no direct cub include' \
--step 'Ensure correct trailing newlines'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'cmakelint' \
--step 'Run cmakelint'
quick-checks: Ensure no direct cub include
quick-checks: Ensure canonical include
quick-checks: Ensure no unqualified noqa
quick-checks: Ensure no non-breaking spaces
quick-checks: Ensure no tabs
quick-checks: Ensure correct trailing newlines
cmakelint: Run cmakelint
quick-checks: Ensure no trailing spaces
mypy: Run mypy
Success: no issues found in 1316 source files
Success: no issues found in 56 source files
flake8-py3: Run flake8
./test.py:1:1: F401 'torch' imported but unused
real 13.89
user 199.63
sys 6.08
```
Mypy/flake8 are by far the slowest, but that's mostly just because they're wasting a bunch of work linting the entire repo.
In followup, we could/should:
* Improve ergonomics (i.e. no output unless there are errors)
* Speed up lint by only linting files changes between origin and HEAD
* Add clang-tidy
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56439
Reviewed By: samestep
Differential Revision: D27888027
Pulled By: driazati
fbshipit-source-id: d6f2a59a45e9d725566688bdac8e909210175996
Summary:
This PR includes:
- A formatting change to make katex installation instructions more visible for Facebook employees.
- A short tip about how to start a lightweight HTTP server on a remote machine to browse the doc build artifacts.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56018
Reviewed By: H-Huang
Differential Revision: D27765157
Pulled By: cbalioglu
fbshipit-source-id: 67663de0ba7b742e0deb5358d1e45eea9edd840f
Summary:
PRs https://github.com/pytorch/pytorch/issues/53652 and https://github.com/pytorch/pytorch/issues/54693 attempted to increase the consistency of our choice of commit (head vs merge) for CI on PRs, and have so far been unsuccessful. This PR takes a less ambitious approach to the problem by clarifying the choice in one specific way (see the following paragraph) and documenting it in `CONTRIBUTING.md`.
In addition to documentation, this PR also removes the current behavior of our GHA jobs that checkout the PR tip instead of the merge commit. At first glance, this behavior seems to increase consistency (by eliminating the special-case for `ghstack` PRs), but in reality, it actually just means that for non-`ghstack` PRs, the question "Which commit is used in CI?" has *two* answers instead of just one; see the description of https://github.com/pytorch/pytorch/issues/53652 for more details.
Once merged, this PR will unblock other PRs that add modify our GHA workflows in breaking ways, such as https://github.com/pytorch/pytorch/issues/54737.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54967
Test Plan: None.
Reviewed By: walterddr, seemethere
Differential Revision: D27435913
Pulled By: samestep
fbshipit-source-id: 405fb419cf015cf88107d5eb2498cfb5bcb7ce33
