Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/69041
`TH_CONCAT_{N}` is still being used by THP so I've moved that into
it's own header but all the compiled code is gone.
Test Plan: Imported from OSS
Reviewed By: anjali411
Differential Revision: D32872477
Pulled By: ngimel
fbshipit-source-id: 06c82d8f96dbcee0715be407c61dfc7d7e8be47a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62550
I noticed that running the build twice in a row resulted in ~80 CUDA files being
rebuilt. Running `ninja -d explain` shows
```
ninja explain: TH/generic/THStorage.h is dirty
ninja explain: TH/generic/THStorageCopy.h is dirty
ninja explain: THC/generic/THCStorage.h is dirty
ninja explain: THC/generic/THCStorageCopy.h is dirty
ninja explain: TH/generic/THTensor.h is dirty
ninja explain: THC/generic/THCTensor.h is dirty
ninja explain: THC/generic/THCTensorCopy.h is dirty
ninja explain: THC/generic/THCTensorMath.h is dirty
ninja explain: THC/generic/THCTensorMathMagma.h is dirty
ninja explain: THC/generic/THCTensorMathPairwise.h is dirty
ninja explain: THC/generic/THCTensorScatterGather.h is dirty
```
considering `ninja` is working relative to the `build` folder, these files don't
actually exist. I traced this back to the output of `nvcc -MD` containing
paths relative to the include directory, instead of being absolute.
This adds a little script to launch the compiler then resolve any relative paths
in the `.d` file before `ninja` looks at it. To use it, I run the build with
```
export CMAKE_CUDA_COMPILER_LAUNCHER="python;`pwd`/tools/nvcc_fix_deps.py;ccache"
```
There are some possible pit-falls here. The same relative path might work for
two include directories, and the compiler could pick a different one. Or,
the compiler might have additional implicit include directories that are needed
to resolve the path. However, this has worked perfectly in my testing and it's
completely opt-in so should be fine.
Test Plan: Imported from OSS
Reviewed By: ejguan
Differential Revision: D31503351
Pulled By: malfet
fbshipit-source-id: b184c4526679d976b93829b5715cafcb1c7db2ae
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/64598
This adds a filter option rather than an all-or-nothing so it's easier to iterate on a specific job.
```bash
python tools/testing/explicit_ci_jobs.py --filter-gha '*generated-linux-*gcc5.4*'
```
See #64600 for an example usage
NB: If you regenerate the worfklows you will need to re-run that command to re-delete everything.
Test Plan: Imported from OSS
Reviewed By: janeyx99
Differential Revision: D30788850
Pulled By: driazati
fbshipit-source-id: a32c266bbd876c396665bceef9a0a961b4586564
Summary:
This PR is created to replace https://github.com/pytorch/pytorch/pull/53180 PR stack, which has all the review discussions. Reason for needing a replacement is due to a messy Sandcastle issue.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/64234
Reviewed By: gmagogsfm
Differential Revision: D30656444
Pulled By: ansley
fbshipit-source-id: 77536c8bcc88162e2c72636026ca3c16891d669a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63385
Correcting a mistake for the pytorch uninstall, and
adding an extra note for Darwin.
Test Plan: Imported from OSS
Reviewed By: janeyx99, heitorschueroff
Differential Revision: D30530234
fbshipit-source-id: e0f88a1725eeadabfb4b28c1da11e369ee878ab4
Summary:
1. In fact, Visual Studio isn't supported as CMAKE generator
2. I was asked many times why there's error as 'Could NOT find OpenMP'
3. Add Newly added Best Practices link in contributing.md
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63843
Reviewed By: seemethere, heitorschueroff
Differential Revision: D30514095
Pulled By: janeyx99
fbshipit-source-id: 76715a1d8c049122546e5a7778cafe54e4dfd5d6
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62549
When building CUDA files with native CMake support, it will respect the
`CMAKE_CUDA_COMPILER_LAUNCHER` setting. So, there's no need for symlinks.
Test Plan: Imported from OSS
Reviewed By: bdhirsh
Differential Revision: D30498488
Pulled By: malfet
fbshipit-source-id: 71c2ae9d4570cfac2a64d777bc95cda3764332a0
Summary:
This option was added in https://github.com/pytorch/pytorch/issues/61940 and fits with this section's theme of improving build times.
I've also changed it to a `cmake_dependent_option` instead of `FATAL_ERROR`ing for older CMake versions.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62827
Reviewed By: astaff
Differential Revision: D30342102
Pulled By: malfet
fbshipit-source-id: 3095b44b7085aee8a884ec95cba9f8998d4442e7
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63228
It is a quick summary and links to a page on the Developer Wiki that has
more detail.
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D30347109
Pulled By: zou3519
fbshipit-source-id: a6242986d275e5279ca3f61ade2294a132d268c4
Summary:
This PR enables installing our custom MacOS clang-tidy binaries. It also updates related documentation.
The binaries are produced by [this CI job](https://github.com/pytorch/test-infra/blob/master/.github/workflows/clang-tidy-macos.yml), and are published to S3.
This PR does not handle versioning of the downloaded binaries as this is being worked on separately. See https://github.com/pytorch/test-infra/issues/73
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62214
Test Plan:
On a MacOS machine, run
```bash
python3 -m tools.linter.install.clang_tidy
.clang-tidy-bin/clang-tidy --checks="*" --list-checks | grep "misc-max-tokens"
```
Reviewed By: janeyx99, mruberry
Differential Revision: D29917728
Pulled By: 1ntEgr8
fbshipit-source-id: 98d0d8b7a57bdebf0ebcdc83228ef391e8c6629e
Summary:
Now expecttest is an independent library but `CONTRIBUTING.md` and `requirements.txt` do not mention the need of the library.
Related: https://github.com/pytorch/pytorch/pull/60658
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61163
Reviewed By: heitorschueroff
Differential Revision: D29660296
Pulled By: ezyang
fbshipit-source-id: e2e86d42526c83bec7cdf7221e19fe83d9686103
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61119
This change spilts the clang-tidy CI job into smaller steps and uses a
refactored version of the clang_tidy.py script.
The new folder structure is as follows:
```
tools/linter/clang_tidy
|_ __main__py
|_ requirements.txt
|_ run.py
|_ setup.sh
```
`__main__.py`
This script will run `tools/linter/clang_tidy/setup.sh` if a `build`
directory doesn't exist, mimicing what used to be done as a separate
step in the CI job.
After that, it will invoke `clang-tidy` with default arguments being
declared in the script itself (as opposed to declaring them in
lint.yml).
The reasoning behind this approach is two-fold:
- Make it easier to run `clang-tidy` locally using this script
- De-duplicate the option passing
`requirements.txt`
Contains a list of additional python dependencies needed by the
`clang-tidy` script.
`setup.sh`
If a build directory doesn't exist, this command will run the necessary
codegen and build commands for running `clang-tidy`
Example usage:
```
python3 tools/linter/clang_tidy --parallel
```
Notice that we don't have to put the `.py` at the end of `clang_tidy`.
Test Plan:
Run the following command:
```
python3 tools/linter/clang_tidy --paths torch/csrc/fx --parallel
```
Reviewed By: walterddr, janeyx99
Differential Revision: D29568582
Pulled By: 1ntEgr8
fbshipit-source-id: cd6d11c5cb8ba9f1344a87c35647a1cd8dd45b04
Summary:
People don't need to self host these anymore since we do it automatically in PRs
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61080
Reviewed By: VitalyFedyunin, janeyx99
Differential Revision: D29506465
Pulled By: driazati
fbshipit-source-id: 45875cb229f8cc565a9a1405f52cef198ee0e687
Summary:
Changes including:
- introduced `linter/`, `testing/`, `stats/` folders in `tools/`
- move appropriate scripts into these folders
- change grepped references in the pytorch/pytorch repo
Next step
- introduce `build/` folder for build scripts
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60473
Test Plan:
- CI (this is important b/c pytorch/test-infra also rely on some script reference.
- tools/tests/
Reviewed By: albanD
Differential Revision: D29352716
Pulled By: walterddr
fbshipit-source-id: bad40b5ce130b35dfd9e59b8af34f9025f3285fd
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58001
Adds a script so that devs can generate a commit (at the base of a stack) that removes all CI jobs but the set that they care about. See CONTRIBUTING.md changes for usage
Test Plan: Imported from OSS
Reviewed By: mruberry
Differential Revision: D28359910
Pulled By: driazati
fbshipit-source-id: 2741570f2bab2c28f4a9d7aef727b1b2399d0ce1
Summary:
Why:
To keep VS version always updated in README
1. update VS version link in CI. It's more convenient for my PR robot to update the version in README once the VS in CI is updated. and permlink isn't stable.
2. Move `building on legacy code` to development tips. The table is big and it looks the REAMD not updated at the first sight.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56686
Reviewed By: janeyx99
Differential Revision: D28272060
Pulled By: samestep
fbshipit-source-id: 4bb879ea2914cc8bcd68343a9ed230418e1f9268
Summary:
This queries the local git repo for changed files (any changed files, not just committed ones) and sends them to mypy/flake8 instead of the default (which is the whole repo, defined the .flake8 and mypy.ini files). This brings a good speedup (from 15 seconds with no cache to < 1 second from my local testing on this PR).
```bash
make quicklint -j 6
```
It should be noted that the results of this aren’t exactly what’s in the CI, since mypy and flake8 ignore the `include` and `exclude` parts of their config when an explicit list of files is passed in.
](https://our.intern.facebook.com/intern/diff/27901577/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56559
Pulled By: driazati
Reviewed By: malfet
Differential Revision: D27901577
fbshipit-source-id: 99f351cdfe5aba007948aea2b8a78f683c5d8583
Summary:
This pulls out shell scripts from an action and runs them locally as a first pass at https://github.com/pytorch/pytorch/issues/55847. A helper script extracts specific steps in some order and runs them:
```bash
$ time -p make lint -j 5 # run lint with 5 CPUs
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'flake8-py3' \
--step 'Run flake8'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'mypy' \
--step 'Run mypy'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'quick-checks' \
--step 'Ensure no trailing spaces' \
--step 'Ensure no tabs' \
--step 'Ensure no non-breaking spaces' \
--step 'Ensure canonical include' \
--step 'Ensure no unqualified noqa' \
--step 'Ensure no direct cub include' \
--step 'Ensure correct trailing newlines'
python scripts/actions_local_runner.py \
--file .github/workflows/lint.yml \
--job 'cmakelint' \
--step 'Run cmakelint'
quick-checks: Ensure no direct cub include
quick-checks: Ensure canonical include
quick-checks: Ensure no unqualified noqa
quick-checks: Ensure no non-breaking spaces
quick-checks: Ensure no tabs
quick-checks: Ensure correct trailing newlines
cmakelint: Run cmakelint
quick-checks: Ensure no trailing spaces
mypy: Run mypy
Success: no issues found in 1316 source files
Success: no issues found in 56 source files
flake8-py3: Run flake8
./test.py:1:1: F401 'torch' imported but unused
real 13.89
user 199.63
sys 6.08
```
Mypy/flake8 are by far the slowest, but that's mostly just because they're wasting a bunch of work linting the entire repo.
In followup, we could/should:
* Improve ergonomics (i.e. no output unless there are errors)
* Speed up lint by only linting files changes between origin and HEAD
* Add clang-tidy
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56439
Reviewed By: samestep
Differential Revision: D27888027
Pulled By: driazati
fbshipit-source-id: d6f2a59a45e9d725566688bdac8e909210175996
Summary:
This PR includes:
- A formatting change to make katex installation instructions more visible for Facebook employees.
- A short tip about how to start a lightweight HTTP server on a remote machine to browse the doc build artifacts.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56018
Reviewed By: H-Huang
Differential Revision: D27765157
Pulled By: cbalioglu
fbshipit-source-id: 67663de0ba7b742e0deb5358d1e45eea9edd840f
Summary:
PRs https://github.com/pytorch/pytorch/issues/53652 and https://github.com/pytorch/pytorch/issues/54693 attempted to increase the consistency of our choice of commit (head vs merge) for CI on PRs, and have so far been unsuccessful. This PR takes a less ambitious approach to the problem by clarifying the choice in one specific way (see the following paragraph) and documenting it in `CONTRIBUTING.md`.
In addition to documentation, this PR also removes the current behavior of our GHA jobs that checkout the PR tip instead of the merge commit. At first glance, this behavior seems to increase consistency (by eliminating the special-case for `ghstack` PRs), but in reality, it actually just means that for non-`ghstack` PRs, the question "Which commit is used in CI?" has *two* answers instead of just one; see the description of https://github.com/pytorch/pytorch/issues/53652 for more details.
Once merged, this PR will unblock other PRs that add modify our GHA workflows in breaking ways, such as https://github.com/pytorch/pytorch/issues/54737.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54967
Test Plan: None.
Reviewed By: walterddr, seemethere
Differential Revision: D27435913
Pulled By: samestep
fbshipit-source-id: 405fb419cf015cf88107d5eb2498cfb5bcb7ce33
Summary:
This is something which I wrote because it was useful during my debugging sessions, but I think it might be generally useful to other people as well so I took the liberty of proposing an official `pytorch-gdb` extension.
`pytorch-gdb` is a gdb script written in python. Currently, it contains only one command: `torch-tensor-repr`, which prints a human-readable repr of an `at::Tensor` object. Example:
```
Breakpoint 1, at::native::neg (self=...) at [...]/pytorch/aten/src/ATen/native/UnaryOps.cpp:520
520 Tensor neg(const Tensor& self) { return unary_op_impl(self, at::neg_out); }
(gdb) # the default repr of 'self' is not very useful
(gdb) p self
$1 = (const at::Tensor &) 0x7ffff72ed780: {impl_ = {target_ = 0x5555559df6e0}}
(gdb) torch-tensor-repr self
Python-level repr of self:
tensor([1., 2., 3., 4.], dtype=torch.float64)
```
The idea is that by having an official place where to put these things, `pytorch-gdb` will slowly grow other useful features and make the pytorch debugging experience nicer and faster.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54339
Reviewed By: bdhirsh
Differential Revision: D27253674
Pulled By: ezyang
fbshipit-source-id: dba219e126cc2fe66b2d26740f3a8e3b886e56f5
Summary:
Context: https://github.com/pytorch/pytorch/pull/53299#discussion_r587882857
These are the only hand-written parts of this diff:
- the addition to `.github/workflows/lint.yml`
- the file endings changed in these four files (to appease FB-internal land-blocking lints):
- `GLOSSARY.md`
- `aten/src/ATen/core/op_registration/README.md`
- `scripts/README.md`
- `torch/csrc/jit/codegen/fuser/README.md`
The rest was generated by running this command (on macOS):
```
git grep -I -l ' $' -- . ':(exclude)**/contrib/**' ':(exclude)third_party' | xargs gsed -i 's/ *$//'
```
I looked over the auto-generated changes and didn't see anything that looked problematic.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53406
Test Plan:
This run (after adding the lint but before removing existing trailing spaces) failed:
- https://github.com/pytorch/pytorch/runs/2043032377
This run (on the tip of this PR) succeeded:
- https://github.com/pytorch/pytorch/runs/2043296348
Reviewed By: walterddr, seemethere
Differential Revision: D26856620
Pulled By: samestep
fbshipit-source-id: 3f0de7f7c2e4b0f1c089eac9b5085a58dd7e0d97
Summary:
Adds a link to existing instructions in CONTRIBUTING.md, so those instructions are more visible to contributors.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50841
Reviewed By: samestep
Differential Revision: D25983089
Pulled By: janeyx99
fbshipit-source-id: 0b777ec760765153c607515ab09441dd0cfddf3c
Summary:
Addresses one of the documentation points in https://github.com/pytorch/pytorch/issues/50513 by making it easier to find our `mypy` wiki page. Also updates the `CONTRIBUTING.md` table of contents and removes some trailing whitespace.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50540
Reviewed By: janeyx99
Differential Revision: D25912366
Pulled By: samestep
fbshipit-source-id: b305f974700a9d9ebedc0c2cb75c92e72d84882a
Summary:
This PR adds instructions on what to do if one committed into a PR branch w/o having a pre-commit hook enabled and having CI report flake8 errors.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49345
Reviewed By: cpuhrsch
Differential Revision: D25683167
Pulled By: soumith
fbshipit-source-id: 3c45c866e1636c116d2cacec438d62c860e6b854
Summary:
Since caffe2 and torch have been consolidated, CAFFE2_API should be merged with TORCH_API. Addresses a TODO.
Manually edited some references of the removed `CAFFE2_API`:
* `CONTRIBUTING.md`
* `caffe2/proto/CMakeLists.txt`
* `cmake/ProtoBuf.cmake`
* `c10/macros/Export.h`
* `torch/csrc/WindowsTorchApiMacro.h`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49496
Reviewed By: malfet, samestep
Differential Revision: D25600726
Pulled By: janeyx99
fbshipit-source-id: 7e068d959e397ac183c097d7e9a9afeca5ddd782
Summary:
In the case people are confused how to make sure ccache is working, I added another sentence in the documentation for how to check that the symlinks are correctly set up in addition to waiting for 2 clean builds of PyTorch.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49337
Reviewed By: walterddr
Differential Revision: D25535659
Pulled By: janeyx99
fbshipit-source-id: 435696255f517c074dd0d9f96534d22b60f795b2
Summary:
This PR moves the list of Flake8 requirements/versions out of `.github/workflows/lint.yml` and into its own file `requirements-flake8.txt`. After (if) this PR is merged, I'll modify the Flake8 installation instructions on [the "Lint as you type" wiki page](https://github.com/pytorch/pytorch/wiki/Lint-as-you-type) (and its internal counterpart) to just say to install from that new file, rather than linking to the GitHub Actions YAML file and/or giving a command with a set of packages to install that keeps becoming out-of-date.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49032
Test Plan:
Either look at CI, or run locally using [act](https://github.com/nektos/act):
```sh
act -P ubuntu-latest=nektos/act-environments-ubuntu:18.04 -j flake8-py3
```
Reviewed By: janeyx99
Differential Revision: D25404037
Pulled By: samestep
fbshipit-source-id: ba4d1e17172a7808435df06cba8298b2b91bb27c
Summary:
This PR tries to make building the docs less confusing for new contributors:
- `npm` is discouraged on devservers for Facebook employees, so I added another way to install `katex`
- the path to `check-doxygen.sh` was wrong, so I fixed it
- while generating the CPP docs, it created two new folders that weren't ignored by Git, so I added those to `.gitignore`
- I wasn't able to get the SSH tunnel to work, so I added instructions to use `scp` as an alternative
I'm not entirely sure how the `docs/cpp/source/{html,latex}/` directories were created since I haven't been able to reproduce them.
I also think that it would be better to use the SSH tunnel since `scp` is so much slower, but I just wasn't able to figure it out; I followed the instructions from `CONTRIBUTING.md` and then ran a [Python `http.server`](https://docs.python.org/3/library/http.server.html) on my devserver:
```bash
python -m http.server 8000 --bind 127.0.0.1 --directory build/html
```
but my browser failed to connect and my (local) terminal printed error messages (presumably from the SSH command).
If anyone knows how to properly set up the SSH tunnel and HTTP server, I add those more detailed instructions to `CONTRIBUTING.md` and remove the `scp` instructions from this PR.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/47539
Reviewed By: malfet
Differential Revision: D24806833
Pulled By: samestep
fbshipit-source-id: 456691018a76efadde28fa5eb783b0895582e72d
Summary:
When attempting to install PyTorch locally on my Macbook, I had some difficulty running the setup steps and understanding what I was really doing. I've added some clarifications and summarized some debugging steps about `python setup.py develop` to lower the barrier of entrance for new contributors.
I'm seeking a lot of review here since I am not sure if what I wrote is entirely the most useful or accurate. Thank you!
Pull Request resolved: https://github.com/pytorch/pytorch/pull/45903
Reviewed By: albanD
Differential Revision: D24140343
Pulled By: janeyx99
fbshipit-source-id: a5e40d1bc804945ae7db2b95ab18cf7fe169e68a
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/45610
Also add to the usual documentation places that this option exists.
Test Plan: Imported from OSS
Reviewed By: gmagogsfm
Differential Revision: D24058199
Pulled By: suo
fbshipit-source-id: 81574fbd042f47587e2c7820c726fac0f68af2a7
