2016-12-23 21:28:04 +00:00
|
|
|
torch
|
2020-05-04 21:29:24 +00:00
|
|
|
=====
|
2022-03-10 22:21:03 +00:00
|
|
|
.. automodule:: torch
|
2020-05-04 21:29:24 +00:00
|
|
|
.. currentmodule:: torch
|
2016-12-23 21:28:04 +00:00
|
|
|
|
2017-01-05 04:20:57 +00:00
|
|
|
Tensors
|
2020-05-04 21:29:24 +00:00
|
|
|
-------
|
|
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
is_tensor
|
|
|
|
|
is_storage
|
|
|
|
|
is_complex
|
Conjugate View (#54987)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54987
Based off of ezyang (https://github.com/pytorch/pytorch/pull/44799) and bdhirsh (https://github.com/pytorch/pytorch/pull/43702) 's prototype:
Here's a summary of the changes in this PR:
This PR adds a new dispatch key called Conjugate. This enables us to make conjugate operation a view and leverage the specialized library functions that fast path with the hermitian operation (conj + transpose).
1. Conjugate operation will now return a view with conj bit (1) for complex tensors and returns self for non-complex tensors as before. This also means `torch.view_as_real` will no longer be a view on conjugated complex tensors and is hence disabled. To fill the gap, we have added `torch.view_as_real_physical` which would return the real tensor agnostic of the conjugate bit on the input complex tensor. The information about conjugation on the old tensor can be obtained by calling `.is_conj()` on the new tensor.
2. NEW API:
a) `.conj()` -- now returning a view.
b) `.conj_physical()` -- does the physical conjugate operation. If the conj bit for input was set, you'd get `self.clone()`, else you'll get a new tensor with conjugated value in its memory.
c) `.conj_physical_()`, and `out=` variant
d) `.resolve_conj()` -- materializes the conjugation. returns self if the conj bit is unset, else returns a new tensor with conjugated values and conj bit set to 0.
e) `.resolve_conj_()` in-place version of (d)
f) `view_as_real_physical` -- as described in (1), it's functionally same as `view_as_real`, just that it doesn't error out on conjugated tensors.
g) `view_as_real` -- existing function, but now errors out on conjugated tensors.
3. Conjugate Fallback
a) Vast majority of PyTorch functions would currently use this fallback when they are called on a conjugated tensor.
b) This fallback is well equipped to handle the following cases:
- functional operation e.g., `torch.sin(input)`
- Mutable inputs and in-place operations e.g., `tensor.add_(2)`
- out-of-place operation e.g., `torch.sin(input, out=out)`
- Tensorlist input args
- NOTE: Meta tensors don't work with conjugate fallback.
4. Autograd
a) `resolve_conj()` is an identity function w.r.t. autograd
b) Everything else works as expected.
5. Testing:
a) All method_tests run with conjugate view tensors.
b) OpInfo tests that run with conjugate views
- test_variant_consistency_eager/jit
- gradcheck, gradgradcheck
- test_conj_views (that only run for `torch.cfloat` dtype)
NOTE: functions like `empty_like`, `zero_like`, `randn_like`, `clone` don't propagate the conjugate bit.
Follow up work:
1. conjugate view RFC
2. Add neg bit to re-enable view operation on conjugated tensors
3. Update linalg functions to call into specialized functions that fast path with the hermitian operation.
Test Plan: Imported from OSS
Reviewed By: VitalyFedyunin
Differential Revision: D28227315
Pulled By: anjali411
fbshipit-source-id: acab9402b9d6a970c6d512809b627a290c8def5f
2021-06-04 21:11:23 +00:00
|
|
|
is_conj
|
2020-05-04 21:29:24 +00:00
|
|
|
is_floating_point
|
2020-05-14 11:44:55 +00:00
|
|
|
is_nonzero
|
2020-05-04 21:29:24 +00:00
|
|
|
set_default_dtype
|
|
|
|
|
get_default_dtype
|
2023-01-09 14:57:21 +00:00
|
|
|
set_default_device
|
2023-12-19 05:08:06 +00:00
|
|
|
get_default_device
|
2020-05-04 21:29:24 +00:00
|
|
|
set_default_tensor_type
|
|
|
|
|
numel
|
|
|
|
|
set_printoptions
|
|
|
|
|
set_flush_denormal
|
2017-01-05 04:20:57 +00:00
|
|
|
|
2018-04-17 20:52:22 +00:00
|
|
|
.. _tensor-creation-ops:
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
Creation Ops
|
2021-04-08 13:39:06 +00:00
|
|
|
~~~~~~~~~~~~
|
2018-04-19 17:16:07 +00:00
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
Random sampling creation ops are listed under :ref:`random-sampling` and
|
|
|
|
|
include:
|
|
|
|
|
:func:`torch.rand`
|
|
|
|
|
:func:`torch.rand_like`
|
|
|
|
|
:func:`torch.randn`
|
|
|
|
|
:func:`torch.randn_like`
|
|
|
|
|
:func:`torch.randint`
|
|
|
|
|
:func:`torch.randint_like`
|
|
|
|
|
:func:`torch.randperm`
|
2018-04-21 11:35:37 +00:00
|
|
|
You may also use :func:`torch.empty` with the :ref:`inplace-random-sampling`
|
|
|
|
|
methods to create :class:`torch.Tensor` s with values sampled from a broader
|
|
|
|
|
range of distributions.
|
2018-04-19 17:16:07 +00:00
|
|
|
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
tensor
|
|
|
|
|
sparse_coo_tensor
|
2023-01-12 19:00:28 +00:00
|
|
|
sparse_csr_tensor
|
|
|
|
|
sparse_csc_tensor
|
|
|
|
|
sparse_bsr_tensor
|
|
|
|
|
sparse_bsc_tensor
|
2021-10-17 04:10:20 +00:00
|
|
|
asarray
|
2020-05-04 21:29:24 +00:00
|
|
|
as_tensor
|
|
|
|
|
as_strided
|
2023-10-25 16:35:09 +00:00
|
|
|
from_file
|
2020-05-04 21:29:24 +00:00
|
|
|
from_numpy
|
2022-02-14 03:30:15 +00:00
|
|
|
from_dlpack
|
Implement NumPy-like `frombuffer` tensor constructor. (#59077)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/59077
Fixes #58549
`from_buffer` constructs a tensor object from an already allocated buffer through
CPython's buffer protocol. Besides the standard `dtype`, `count`, and `offset` parameters,
this function also accepts:
- `device`: where the buffer lives
- `requires_grad`: should autograd record operations on the new tensor
A new test file _test_buffer_protocol.py_ was created. Currently, only CPU tests were
implemented. That's because neither PyTorch nor Numba implements CPython's buffer
protocol. Therefore, there's no way to create a CUDA buffer with the existing
dependencies (could use PyCUDA for that, though).
At the moment, if `device` differs from the device the buffer actually lives, two things
may happen:
- `RuntimeError`, if `device='cuda'`
- Segmentation fault (not tested -- see above), if `device='cpu'`
Test Plan: Imported from OSS
Reviewed By: jbschlosser
Differential Revision: D29870914
Pulled By: mruberry
fbshipit-source-id: 9fa8611aeffedfe39c9af74558178157a11326bb
2021-07-23 20:16:26 +00:00
|
|
|
frombuffer
|
2020-05-04 21:29:24 +00:00
|
|
|
zeros
|
|
|
|
|
zeros_like
|
|
|
|
|
ones
|
|
|
|
|
ones_like
|
|
|
|
|
arange
|
|
|
|
|
range
|
|
|
|
|
linspace
|
|
|
|
|
logspace
|
|
|
|
|
eye
|
|
|
|
|
empty
|
|
|
|
|
empty_like
|
|
|
|
|
empty_strided
|
|
|
|
|
full
|
|
|
|
|
full_like
|
|
|
|
|
quantize_per_tensor
|
|
|
|
|
quantize_per_channel
|
|
|
|
|
dequantize
|
2020-08-14 07:28:38 +00:00
|
|
|
complex
|
|
|
|
|
polar
|
2020-08-31 22:43:51 +00:00
|
|
|
heaviside
|
2017-01-05 04:20:57 +00:00
|
|
|
|
2021-04-08 13:39:06 +00:00
|
|
|
.. _indexing-slicing-joining:
|
|
|
|
|
|
2017-01-05 04:20:57 +00:00
|
|
|
Indexing, Slicing, Joining, Mutating Ops
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
2021-11-24 17:56:08 +00:00
|
|
|
adjoint
|
2021-10-30 22:24:28 +00:00
|
|
|
argwhere
|
2020-05-04 21:29:24 +00:00
|
|
|
cat
|
2021-09-07 06:55:53 +00:00
|
|
|
concat
|
2022-09-15 19:34:44 +00:00
|
|
|
concatenate
|
Conjugate View (#54987)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54987
Based off of ezyang (https://github.com/pytorch/pytorch/pull/44799) and bdhirsh (https://github.com/pytorch/pytorch/pull/43702) 's prototype:
Here's a summary of the changes in this PR:
This PR adds a new dispatch key called Conjugate. This enables us to make conjugate operation a view and leverage the specialized library functions that fast path with the hermitian operation (conj + transpose).
1. Conjugate operation will now return a view with conj bit (1) for complex tensors and returns self for non-complex tensors as before. This also means `torch.view_as_real` will no longer be a view on conjugated complex tensors and is hence disabled. To fill the gap, we have added `torch.view_as_real_physical` which would return the real tensor agnostic of the conjugate bit on the input complex tensor. The information about conjugation on the old tensor can be obtained by calling `.is_conj()` on the new tensor.
2. NEW API:
a) `.conj()` -- now returning a view.
b) `.conj_physical()` -- does the physical conjugate operation. If the conj bit for input was set, you'd get `self.clone()`, else you'll get a new tensor with conjugated value in its memory.
c) `.conj_physical_()`, and `out=` variant
d) `.resolve_conj()` -- materializes the conjugation. returns self if the conj bit is unset, else returns a new tensor with conjugated values and conj bit set to 0.
e) `.resolve_conj_()` in-place version of (d)
f) `view_as_real_physical` -- as described in (1), it's functionally same as `view_as_real`, just that it doesn't error out on conjugated tensors.
g) `view_as_real` -- existing function, but now errors out on conjugated tensors.
3. Conjugate Fallback
a) Vast majority of PyTorch functions would currently use this fallback when they are called on a conjugated tensor.
b) This fallback is well equipped to handle the following cases:
- functional operation e.g., `torch.sin(input)`
- Mutable inputs and in-place operations e.g., `tensor.add_(2)`
- out-of-place operation e.g., `torch.sin(input, out=out)`
- Tensorlist input args
- NOTE: Meta tensors don't work with conjugate fallback.
4. Autograd
a) `resolve_conj()` is an identity function w.r.t. autograd
b) Everything else works as expected.
5. Testing:
a) All method_tests run with conjugate view tensors.
b) OpInfo tests that run with conjugate views
- test_variant_consistency_eager/jit
- gradcheck, gradgradcheck
- test_conj_views (that only run for `torch.cfloat` dtype)
NOTE: functions like `empty_like`, `zero_like`, `randn_like`, `clone` don't propagate the conjugate bit.
Follow up work:
1. conjugate view RFC
2. Add neg bit to re-enable view operation on conjugated tensors
3. Update linalg functions to call into specialized functions that fast path with the hermitian operation.
Test Plan: Imported from OSS
Reviewed By: VitalyFedyunin
Differential Revision: D28227315
Pulled By: anjali411
fbshipit-source-id: acab9402b9d6a970c6d512809b627a290c8def5f
2021-06-04 21:11:23 +00:00
|
|
|
conj
|
2020-05-04 21:29:24 +00:00
|
|
|
chunk
|
2021-04-26 16:37:53 +00:00
|
|
|
dsplit
|
2020-10-29 19:06:11 +00:00
|
|
|
column_stack
|
2020-08-16 03:36:55 +00:00
|
|
|
dstack
|
2020-05-04 21:29:24 +00:00
|
|
|
gather
|
2021-04-26 16:37:53 +00:00
|
|
|
hsplit
|
2020-08-16 03:36:55 +00:00
|
|
|
hstack
|
2021-12-14 19:51:37 +00:00
|
|
|
index_add
|
2022-02-08 22:26:26 +00:00
|
|
|
index_copy
|
2022-05-12 22:19:52 +00:00
|
|
|
index_reduce
|
2020-05-04 21:29:24 +00:00
|
|
|
index_select
|
|
|
|
|
masked_select
|
2020-08-17 21:22:36 +00:00
|
|
|
movedim
|
2020-12-03 18:30:00 +00:00
|
|
|
moveaxis
|
2020-05-04 21:29:24 +00:00
|
|
|
narrow
|
2022-10-16 21:22:01 +00:00
|
|
|
narrow_copy
|
2020-05-04 21:29:24 +00:00
|
|
|
nonzero
|
2021-06-28 18:19:27 +00:00
|
|
|
permute
|
2020-05-04 21:29:24 +00:00
|
|
|
reshape
|
2020-10-29 19:06:11 +00:00
|
|
|
row_stack
|
2021-11-30 19:52:23 +00:00
|
|
|
select
|
2020-12-23 20:20:52 +00:00
|
|
|
scatter
|
2021-11-30 19:52:23 +00:00
|
|
|
diagonal_scatter
|
|
|
|
|
select_scatter
|
|
|
|
|
slice_scatter
|
2020-12-23 20:20:52 +00:00
|
|
|
scatter_add
|
2022-02-22 19:19:49 +00:00
|
|
|
scatter_reduce
|
2020-05-04 21:29:24 +00:00
|
|
|
split
|
|
|
|
|
squeeze
|
|
|
|
|
stack
|
2020-11-25 21:14:13 +00:00
|
|
|
swapaxes
|
|
|
|
|
swapdims
|
2020-05-04 21:29:24 +00:00
|
|
|
t
|
|
|
|
|
take
|
2021-03-28 12:21:30 +00:00
|
|
|
take_along_dim
|
2020-10-08 06:12:41 +00:00
|
|
|
tensor_split
|
An implementation of torch.tile as requested in pytorch/pytorch#38349 (#47974)
Summary:
The approach is to simply reuse `torch.repeat` but adding one more functionality to tile, which is to prepend 1's to reps arrays if there are more dimensions to the tensors than the reps given in input. Thus for a tensor of shape (64, 3, 24, 24) and reps of (2, 2) will become (1, 1, 2, 2), which is what NumPy does.
I've encountered some instability with the test on my end, where I could get a random failure of the test (due to, sometimes, random value of `self.dim()`, and sometimes, segfaults). I'd appreciate any feedback on the test or an explanation for this instability so I can this.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/47974
Reviewed By: ngimel
Differential Revision: D25148963
Pulled By: mruberry
fbshipit-source-id: bf63b72c6fe3d3998a682822e669666f7cc97c58
2020-11-25 02:05:32 +00:00
|
|
|
tile
|
2020-05-04 21:29:24 +00:00
|
|
|
transpose
|
|
|
|
|
unbind
|
2023-10-12 00:55:51 +00:00
|
|
|
unravel_index
|
2020-05-04 21:29:24 +00:00
|
|
|
unsqueeze
|
2021-04-26 16:37:53 +00:00
|
|
|
vsplit
|
2020-08-16 03:36:55 +00:00
|
|
|
vstack
|
2020-05-04 21:29:24 +00:00
|
|
|
where
|
2017-01-05 04:20:57 +00:00
|
|
|
|
2024-07-04 13:03:01 +00:00
|
|
|
.. _accelerators:
|
|
|
|
|
|
|
|
|
|
Accelerators
|
|
|
|
|
----------------------------------
|
|
|
|
|
Within the PyTorch repo, we define an "Accelerator" as a :class:`torch.device` that is being used
|
|
|
|
|
alongside a CPU to speed up computation. These device use an asynchronous execution scheme,
|
|
|
|
|
using :class:`torch.Stream` and :class:`torch.Event` as their main way to perform synchronization.
|
|
|
|
|
We also assume that only one such accelerator can be available at once on a given host. This allows
|
|
|
|
|
us to use the current accelerator as the default device for relevant concepts such as pinned memory,
|
2024-08-08 05:25:15 +00:00
|
|
|
Stream device_type, FSDP, etc.
|
2024-07-04 13:03:01 +00:00
|
|
|
|
|
|
|
|
As of today, accelerator devices are (in no particular order) :doc:`"CUDA" <cuda>`, :doc:`"MTIA" <mtia>`,
|
|
|
|
|
:doc:`"XPU" <xpu>`, and PrivateUse1 (many device not in the PyTorch repo itself).
|
|
|
|
|
|
|
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
Stream
|
|
|
|
|
Event
|
|
|
|
|
|
2019-06-12 19:51:19 +00:00
|
|
|
.. _generators:
|
|
|
|
|
|
|
|
|
|
Generators
|
|
|
|
|
----------------------------------
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
2020-05-22 16:00:03 +00:00
|
|
|
Generator
|
2019-06-12 19:51:19 +00:00
|
|
|
|
2018-04-19 17:16:07 +00:00
|
|
|
.. _random-sampling:
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
Random sampling
|
|
|
|
|
----------------------------------
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
seed
|
|
|
|
|
manual_seed
|
|
|
|
|
initial_seed
|
|
|
|
|
get_rng_state
|
|
|
|
|
set_rng_state
|
|
|
|
|
|
2019-06-12 19:51:19 +00:00
|
|
|
.. autoattribute:: torch.default_generator
|
|
|
|
|
:annotation: Returns the default CPU torch.Generator
|
2019-10-13 17:32:28 +00:00
|
|
|
|
|
|
|
|
.. The following doesn't actually seem to exist.
|
|
|
|
|
https://github.com/pytorch/pytorch/issues/27780
|
|
|
|
|
.. autoattribute:: torch.cuda.default_generators
|
|
|
|
|
:annotation: If cuda is available, returns a tuple of default CUDA torch.Generator-s.
|
|
|
|
|
The number of CUDA torch.Generator-s returned is equal to the number of
|
|
|
|
|
GPUs available in the system.
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
bernoulli
|
|
|
|
|
multinomial
|
|
|
|
|
normal
|
|
|
|
|
poisson
|
|
|
|
|
rand
|
|
|
|
|
rand_like
|
|
|
|
|
randint
|
|
|
|
|
randint_like
|
|
|
|
|
randn
|
|
|
|
|
randn_like
|
|
|
|
|
randperm
|
2017-09-14 14:03:17 +00:00
|
|
|
|
2018-04-21 11:35:37 +00:00
|
|
|
.. _inplace-random-sampling:
|
|
|
|
|
|
2017-09-14 14:03:17 +00:00
|
|
|
In-place random sampling
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
There are a few more in-place random sampling functions defined on Tensors as well. Click through to refer to their documentation:
|
|
|
|
|
|
|
|
|
|
- :func:`torch.Tensor.bernoulli_` - in-place version of :func:`torch.bernoulli`
|
|
|
|
|
- :func:`torch.Tensor.cauchy_` - numbers drawn from the Cauchy distribution
|
|
|
|
|
- :func:`torch.Tensor.exponential_` - numbers drawn from the exponential distribution
|
|
|
|
|
- :func:`torch.Tensor.geometric_` - elements drawn from the geometric distribution
|
|
|
|
|
- :func:`torch.Tensor.log_normal_` - samples from the log-normal distribution
|
|
|
|
|
- :func:`torch.Tensor.normal_` - in-place version of :func:`torch.normal`
|
|
|
|
|
- :func:`torch.Tensor.random_` - numbers sampled from the discrete uniform distribution
|
2018-01-09 16:47:48 +00:00
|
|
|
- :func:`torch.Tensor.uniform_` - numbers sampled from the continuous uniform distribution
|
2017-01-05 04:20:57 +00:00
|
|
|
|
2019-03-26 14:49:58 +00:00
|
|
|
Quasi-random sampling
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
:template: sobolengine.rst
|
2019-03-26 14:49:58 +00:00
|
|
|
|
2020-05-04 21:29:24 +00:00
|
|
|
quasirandom.SobolEngine
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
Serialization
|
|
|
|
|
----------------------------------
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
2017-01-05 04:20:57 +00:00
|
|
|
|
2020-05-04 21:29:24 +00:00
|
|
|
save
|
|
|
|
|
load
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
Parallelism
|
|
|
|
|
----------------------------------
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
get_num_threads
|
|
|
|
|
set_num_threads
|
|
|
|
|
get_num_interop_threads
|
|
|
|
|
set_num_interop_threads
|
2017-01-05 04:20:57 +00:00
|
|
|
|
2022-06-01 22:28:29 +00:00
|
|
|
.. _torch-rst-local-disable-grad:
|
|
|
|
|
|
2018-04-21 11:35:37 +00:00
|
|
|
Locally disabling gradient computation
|
|
|
|
|
--------------------------------------
|
|
|
|
|
The context managers :func:`torch.no_grad`, :func:`torch.enable_grad`, and
|
|
|
|
|
:func:`torch.set_grad_enabled` are helpful for locally disabling and enabling
|
|
|
|
|
gradient computation. See :ref:`locally-disable-grad` for more details on
|
2019-06-13 20:39:40 +00:00
|
|
|
their usage. These context managers are thread local, so they won't
|
2019-10-13 17:32:28 +00:00
|
|
|
work if you send work to another thread using the ``threading`` module, etc.
|
2018-04-21 11:35:37 +00:00
|
|
|
|
|
|
|
|
Examples::
|
|
|
|
|
|
|
|
|
|
>>> x = torch.zeros(1, requires_grad=True)
|
|
|
|
|
>>> with torch.no_grad():
|
|
|
|
|
... y = x * 2
|
|
|
|
|
>>> y.requires_grad
|
|
|
|
|
False
|
|
|
|
|
|
|
|
|
|
>>> is_train = False
|
|
|
|
|
>>> with torch.set_grad_enabled(is_train):
|
|
|
|
|
... y = x * 2
|
|
|
|
|
>>> y.requires_grad
|
|
|
|
|
False
|
|
|
|
|
|
|
|
|
|
>>> torch.set_grad_enabled(True) # this can also be used as a function
|
|
|
|
|
>>> y = x * 2
|
|
|
|
|
>>> y.requires_grad
|
|
|
|
|
True
|
|
|
|
|
|
|
|
|
|
>>> torch.set_grad_enabled(False)
|
|
|
|
|
>>> y = x * 2
|
|
|
|
|
>>> y.requires_grad
|
|
|
|
|
False
|
|
|
|
|
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
no_grad
|
|
|
|
|
enable_grad
|
2024-01-29 21:51:28 +00:00
|
|
|
autograd.grad_mode.set_grad_enabled
|
2021-05-27 02:25:59 +00:00
|
|
|
is_grad_enabled
|
2024-01-29 21:51:28 +00:00
|
|
|
autograd.grad_mode.inference_mode
|
2021-05-27 02:25:59 +00:00
|
|
|
is_inference_mode_enabled
|
2017-01-05 04:20:57 +00:00
|
|
|
|
2016-12-23 21:28:04 +00:00
|
|
|
Math operations
|
2020-05-04 21:29:24 +00:00
|
|
|
---------------
|
2016-12-23 21:28:04 +00:00
|
|
|
|
2024-10-31 05:46:44 +00:00
|
|
|
Constants
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
======================================= ===========================================
|
|
|
|
|
``inf`` A floating-point positive infinity. Alias for :attr:`math.inf`.
|
|
|
|
|
``nan`` A floating-point "not a number" value. This value is not a legal number. Alias for :attr:`math.nan`.
|
|
|
|
|
======================================= ===========================================
|
|
|
|
|
|
2017-01-05 04:20:57 +00:00
|
|
|
Pointwise Ops
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
abs
|
|
|
|
|
absolute
|
|
|
|
|
acos
|
2020-08-21 17:50:57 +00:00
|
|
|
arccos
|
2020-06-04 18:38:14 +00:00
|
|
|
acosh
|
2020-08-17 05:10:46 +00:00
|
|
|
arccosh
|
2020-05-04 21:29:24 +00:00
|
|
|
add
|
|
|
|
|
addcdiv
|
|
|
|
|
addcmul
|
|
|
|
|
angle
|
|
|
|
|
asin
|
2020-08-21 17:50:57 +00:00
|
|
|
arcsin
|
2020-06-04 18:38:14 +00:00
|
|
|
asinh
|
2020-08-28 20:57:43 +00:00
|
|
|
arcsinh
|
2020-05-04 21:29:24 +00:00
|
|
|
atan
|
2020-08-21 17:50:57 +00:00
|
|
|
arctan
|
2020-06-04 18:38:14 +00:00
|
|
|
atanh
|
2020-08-28 20:57:43 +00:00
|
|
|
arctanh
|
2020-05-04 21:29:24 +00:00
|
|
|
atan2
|
2021-11-16 17:39:42 +00:00
|
|
|
arctan2
|
2020-05-04 21:29:24 +00:00
|
|
|
bitwise_not
|
|
|
|
|
bitwise_and
|
|
|
|
|
bitwise_or
|
|
|
|
|
bitwise_xor
|
2021-06-24 06:52:00 +00:00
|
|
|
bitwise_left_shift
|
|
|
|
|
bitwise_right_shift
|
2020-05-04 21:29:24 +00:00
|
|
|
ceil
|
|
|
|
|
clamp
|
2020-08-09 09:44:23 +00:00
|
|
|
clip
|
Conjugate View (#54987)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54987
Based off of ezyang (https://github.com/pytorch/pytorch/pull/44799) and bdhirsh (https://github.com/pytorch/pytorch/pull/43702) 's prototype:
Here's a summary of the changes in this PR:
This PR adds a new dispatch key called Conjugate. This enables us to make conjugate operation a view and leverage the specialized library functions that fast path with the hermitian operation (conj + transpose).
1. Conjugate operation will now return a view with conj bit (1) for complex tensors and returns self for non-complex tensors as before. This also means `torch.view_as_real` will no longer be a view on conjugated complex tensors and is hence disabled. To fill the gap, we have added `torch.view_as_real_physical` which would return the real tensor agnostic of the conjugate bit on the input complex tensor. The information about conjugation on the old tensor can be obtained by calling `.is_conj()` on the new tensor.
2. NEW API:
a) `.conj()` -- now returning a view.
b) `.conj_physical()` -- does the physical conjugate operation. If the conj bit for input was set, you'd get `self.clone()`, else you'll get a new tensor with conjugated value in its memory.
c) `.conj_physical_()`, and `out=` variant
d) `.resolve_conj()` -- materializes the conjugation. returns self if the conj bit is unset, else returns a new tensor with conjugated values and conj bit set to 0.
e) `.resolve_conj_()` in-place version of (d)
f) `view_as_real_physical` -- as described in (1), it's functionally same as `view_as_real`, just that it doesn't error out on conjugated tensors.
g) `view_as_real` -- existing function, but now errors out on conjugated tensors.
3. Conjugate Fallback
a) Vast majority of PyTorch functions would currently use this fallback when they are called on a conjugated tensor.
b) This fallback is well equipped to handle the following cases:
- functional operation e.g., `torch.sin(input)`
- Mutable inputs and in-place operations e.g., `tensor.add_(2)`
- out-of-place operation e.g., `torch.sin(input, out=out)`
- Tensorlist input args
- NOTE: Meta tensors don't work with conjugate fallback.
4. Autograd
a) `resolve_conj()` is an identity function w.r.t. autograd
b) Everything else works as expected.
5. Testing:
a) All method_tests run with conjugate view tensors.
b) OpInfo tests that run with conjugate views
- test_variant_consistency_eager/jit
- gradcheck, gradgradcheck
- test_conj_views (that only run for `torch.cfloat` dtype)
NOTE: functions like `empty_like`, `zero_like`, `randn_like`, `clone` don't propagate the conjugate bit.
Follow up work:
1. conjugate view RFC
2. Add neg bit to re-enable view operation on conjugated tensors
3. Update linalg functions to call into specialized functions that fast path with the hermitian operation.
Test Plan: Imported from OSS
Reviewed By: VitalyFedyunin
Differential Revision: D28227315
Pulled By: anjali411
fbshipit-source-id: acab9402b9d6a970c6d512809b627a290c8def5f
2021-06-04 21:11:23 +00:00
|
|
|
conj_physical
|
Implement copysign (#46396)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/46396
Related #38349
[numpy](https://numpy.org/doc/stable/reference/generated/numpy.copysign.html?highlight=copysign#numpy.copysign)
- No in-place function
- No method
- Optional output
- Available: byte, char, bool, int, short, long, float, double, half
- Integral promoted to float
- Not available: float/double complex
`c = np.copysign(a, b)`
| a | b | c | a.grad |
| -1 | -1 | -1 | 1 |
| -0 | -1 | -0 | 0 |
| 0 | -1 | -0 | 0 |
| 1 | -1 | -1 | -1 |
| -1 | -0 | -1 | 1 |
| -0 | -0 | 0 | 0 |
| 0 | -0 | 0 | 0 |
| 1 | -0 | -1 | -1 |
| -1 | 0 | 1 | -1 |
| -0 | 0 | 0 | 0 |
| 0 | 0 | 0 | 0 |
| 1 | 0 | 1 | 1 |
| -1 | 1 | 1 | -1 |
| -0 | 1 | 0 | 0 |
| 0 | 1 | 0 | 0 |
| 1 | 1 | 1 | 1 |
This function becomes **non-differentiable** at `a=0` for any `b`. So, in my opinion, we may set the gradient for `a=0` to 0.
TODO:
- [x] test (cpu/gpu)
- [x] doc
- [x] ~kernel_vec~
Test Plan: Imported from OSS
Reviewed By: mruberry
Differential Revision: D24401366
Pulled By: ejguan
fbshipit-source-id: 3621c5ff74b185376a3705589983bb5197ab896d
2020-11-04 16:07:03 +00:00
|
|
|
copysign
|
2020-05-04 21:29:24 +00:00
|
|
|
cos
|
|
|
|
|
cosh
|
2020-06-04 05:17:17 +00:00
|
|
|
deg2rad
|
2020-05-04 21:29:24 +00:00
|
|
|
div
|
2020-09-19 22:39:49 +00:00
|
|
|
divide
|
2020-05-04 21:29:24 +00:00
|
|
|
digamma
|
|
|
|
|
erf
|
|
|
|
|
erfc
|
|
|
|
|
erfinv
|
|
|
|
|
exp
|
2020-09-14 10:39:04 +00:00
|
|
|
exp2
|
2020-05-04 21:29:24 +00:00
|
|
|
expm1
|
2021-02-06 01:50:40 +00:00
|
|
|
fake_quantize_per_channel_affine
|
|
|
|
|
fake_quantize_per_tensor_affine
|
2020-08-21 04:46:13 +00:00
|
|
|
fix
|
2020-11-28 01:59:48 +00:00
|
|
|
float_power
|
2020-05-04 21:29:24 +00:00
|
|
|
floor
|
|
|
|
|
floor_divide
|
|
|
|
|
fmod
|
|
|
|
|
frac
|
2021-03-15 17:42:19 +00:00
|
|
|
frexp
|
2021-05-12 01:51:07 +00:00
|
|
|
gradient
|
2020-05-04 21:29:24 +00:00
|
|
|
imag
|
2020-11-20 12:07:48 +00:00
|
|
|
ldexp
|
2020-05-04 21:29:24 +00:00
|
|
|
lerp
|
|
|
|
|
lgamma
|
|
|
|
|
log
|
|
|
|
|
log10
|
|
|
|
|
log1p
|
|
|
|
|
log2
|
2020-05-28 03:24:21 +00:00
|
|
|
logaddexp
|
|
|
|
|
logaddexp2
|
2020-05-04 21:29:24 +00:00
|
|
|
logical_and
|
|
|
|
|
logical_not
|
|
|
|
|
logical_or
|
|
|
|
|
logical_xor
|
2020-07-14 02:31:26 +00:00
|
|
|
logit
|
2020-08-12 20:14:36 +00:00
|
|
|
hypot
|
2020-09-06 06:09:43 +00:00
|
|
|
i0
|
2020-10-29 18:38:18 +00:00
|
|
|
igamma
|
2020-11-19 07:42:52 +00:00
|
|
|
igammac
|
2020-05-04 21:29:24 +00:00
|
|
|
mul
|
2020-09-19 22:39:49 +00:00
|
|
|
multiply
|
2020-05-04 21:29:24 +00:00
|
|
|
mvlgamma
|
2020-10-05 08:36:53 +00:00
|
|
|
nan_to_num
|
2020-05-04 21:29:24 +00:00
|
|
|
neg
|
2020-08-24 08:12:55 +00:00
|
|
|
negative
|
2020-08-14 07:33:36 +00:00
|
|
|
nextafter
|
2020-05-04 21:29:24 +00:00
|
|
|
polygamma
|
2021-04-27 20:22:54 +00:00
|
|
|
positive
|
2020-05-04 21:29:24 +00:00
|
|
|
pow
|
2021-08-16 05:44:44 +00:00
|
|
|
quantized_batch_norm
|
2021-08-16 05:44:44 +00:00
|
|
|
quantized_max_pool1d
|
|
|
|
|
quantized_max_pool2d
|
2020-06-04 05:17:17 +00:00
|
|
|
rad2deg
|
2020-05-04 21:29:24 +00:00
|
|
|
real
|
|
|
|
|
reciprocal
|
|
|
|
|
remainder
|
|
|
|
|
round
|
|
|
|
|
rsqrt
|
|
|
|
|
sigmoid
|
|
|
|
|
sign
|
2021-02-01 20:35:59 +00:00
|
|
|
sgn
|
2020-07-30 18:18:37 +00:00
|
|
|
signbit
|
2020-05-04 21:29:24 +00:00
|
|
|
sin
|
2020-12-18 23:44:28 +00:00
|
|
|
sinc
|
2020-05-04 21:29:24 +00:00
|
|
|
sinh
|
2022-12-28 15:06:24 +00:00
|
|
|
softmax
|
2020-05-04 21:29:24 +00:00
|
|
|
sqrt
|
|
|
|
|
square
|
2020-08-30 22:42:19 +00:00
|
|
|
sub
|
|
|
|
|
subtract
|
2020-05-04 21:29:24 +00:00
|
|
|
tan
|
|
|
|
|
tanh
|
|
|
|
|
true_divide
|
|
|
|
|
trunc
|
2020-12-22 23:01:53 +00:00
|
|
|
xlogy
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
Reduction Ops
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
argmax
|
|
|
|
|
argmin
|
2020-08-31 11:52:50 +00:00
|
|
|
amax
|
|
|
|
|
amin
|
2021-08-03 23:03:27 +00:00
|
|
|
aminmax
|
2021-01-08 19:01:52 +00:00
|
|
|
all
|
|
|
|
|
any
|
2020-08-31 11:52:50 +00:00
|
|
|
max
|
|
|
|
|
min
|
2020-05-04 21:29:24 +00:00
|
|
|
dist
|
|
|
|
|
logsumexp
|
|
|
|
|
mean
|
2021-09-13 12:50:27 +00:00
|
|
|
nanmean
|
2020-05-04 21:29:24 +00:00
|
|
|
median
|
2020-10-08 18:18:42 +00:00
|
|
|
nanmedian
|
2020-05-04 21:29:24 +00:00
|
|
|
mode
|
|
|
|
|
norm
|
2020-08-12 05:24:27 +00:00
|
|
|
nansum
|
2020-05-04 21:29:24 +00:00
|
|
|
prod
|
2020-08-11 19:06:28 +00:00
|
|
|
quantile
|
2020-09-17 12:50:41 +00:00
|
|
|
nanquantile
|
2020-05-04 21:29:24 +00:00
|
|
|
std
|
|
|
|
|
std_mean
|
|
|
|
|
sum
|
|
|
|
|
unique
|
|
|
|
|
unique_consecutive
|
|
|
|
|
var
|
|
|
|
|
var_mean
|
2020-06-30 13:37:52 +00:00
|
|
|
count_nonzero
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
Comparison Ops
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
allclose
|
|
|
|
|
argsort
|
|
|
|
|
eq
|
|
|
|
|
equal
|
|
|
|
|
ge
|
2020-09-06 16:34:46 +00:00
|
|
|
greater_equal
|
2020-05-04 21:29:24 +00:00
|
|
|
gt
|
2020-09-06 16:34:46 +00:00
|
|
|
greater
|
2020-05-04 21:29:24 +00:00
|
|
|
isclose
|
|
|
|
|
isfinite
|
Implementation of torch.isin() (#53125)
Summary:
Fixes https://github.com/pytorch/pytorch/issues/3025
## Background
This PR implements a function similar to numpy's [`isin()`](https://numpy.org/doc/stable/reference/generated/numpy.isin.html#numpy.isin).
The op supports integral and floating point types on CPU and CUDA (+ half & bfloat16 for CUDA). Inputs can be one of:
* (Tensor, Tensor)
* (Tensor, Scalar)
* (Scalar, Tensor)
Internally, one of two algorithms is selected based on the number of elements vs. test elements. The heuristic for deciding which algorithm to use is taken from [numpy's implementation](https://github.com/numpy/numpy/blob/fb215c76967739268de71aa4bda55dd1b062bc2e/numpy/lib/arraysetops.py#L575): if `len(test_elements) < 10 * len(elements) ** 0.145`, then a naive brute-force checking algorithm is used. Otherwise, a stablesort-based algorithm is used.
I've done some preliminary benchmarking to verify this heuristic on a devgpu, and determined for a limited set of tests that a power value of `0.407` instead of `0.145` is a better inflection point. For now, the heuristic has been left to match numpy's, but input is welcome for the best way to select it or whether it should be left the same as numpy's.
Tests are adapted from numpy's [isin and in1d tests](https://github.com/numpy/numpy/blob/7dcd29aaafe1ab8be4be04d3c793e5bcaf17459f/numpy/lib/tests/test_arraysetops.py).
Note: my locally generated docs look terrible for some reason, so I'm not including the screenshot for them until I figure out why.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53125
Test Plan:
```
python test/test_ops.py # Ex: python test/test_ops.py TestOpInfoCPU.test_supported_dtypes_isin_cpu_int32
python test/test_sort_and_select.py # Ex: python test/test_sort_and_select.py TestSortAndSelectCPU.test_isin_cpu_int32
```
Reviewed By: soulitzer
Differential Revision: D29101165
Pulled By: jbschlosser
fbshipit-source-id: 2dcc38d497b1e843f73f332d837081e819454b4e
2021-06-14 20:49:38 +00:00
|
|
|
isin
|
2020-05-04 21:29:24 +00:00
|
|
|
isinf
|
2020-07-29 10:27:02 +00:00
|
|
|
isposinf
|
|
|
|
|
isneginf
|
2020-05-04 21:29:24 +00:00
|
|
|
isnan
|
2020-07-18 05:05:40 +00:00
|
|
|
isreal
|
2020-05-04 21:29:24 +00:00
|
|
|
kthvalue
|
|
|
|
|
le
|
2020-09-06 16:34:46 +00:00
|
|
|
less_equal
|
2020-05-04 21:29:24 +00:00
|
|
|
lt
|
2020-09-06 16:34:46 +00:00
|
|
|
less
|
2020-08-26 23:54:33 +00:00
|
|
|
maximum
|
|
|
|
|
minimum
|
2021-01-20 14:42:21 +00:00
|
|
|
fmax
|
|
|
|
|
fmin
|
2020-05-04 21:29:24 +00:00
|
|
|
ne
|
2020-09-06 16:34:46 +00:00
|
|
|
not_equal
|
2020-05-04 21:29:24 +00:00
|
|
|
sort
|
|
|
|
|
topk
|
2020-12-04 12:30:23 +00:00
|
|
|
msort
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
|
2017-12-18 17:28:23 +00:00
|
|
|
Spectral Ops
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
stft
|
|
|
|
|
istft
|
|
|
|
|
bartlett_window
|
|
|
|
|
blackman_window
|
|
|
|
|
hamming_window
|
|
|
|
|
hann_window
|
2020-09-17 03:37:11 +00:00
|
|
|
kaiser_window
|
2017-12-18 17:28:23 +00:00
|
|
|
|
|
|
|
|
|
2017-01-05 04:20:57 +00:00
|
|
|
Other Operations
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
2020-05-04 21:29:24 +00:00
|
|
|
|
|
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
2020-07-17 17:06:52 +00:00
|
|
|
atleast_1d
|
|
|
|
|
atleast_2d
|
|
|
|
|
atleast_3d
|
2020-05-04 21:29:24 +00:00
|
|
|
bincount
|
|
|
|
|
block_diag
|
|
|
|
|
broadcast_tensors
|
2020-12-21 19:22:14 +00:00
|
|
|
broadcast_to
|
2020-12-03 10:40:23 +00:00
|
|
|
broadcast_shapes
|
2020-05-12 04:52:24 +00:00
|
|
|
bucketize
|
2020-05-04 21:29:24 +00:00
|
|
|
cartesian_prod
|
|
|
|
|
cdist
|
2020-08-16 11:17:10 +00:00
|
|
|
clone
|
2020-05-04 21:29:24 +00:00
|
|
|
combinations
|
2021-06-30 19:29:55 +00:00
|
|
|
corrcoef
|
2021-06-29 20:59:46 +00:00
|
|
|
cov
|
2020-05-04 21:29:24 +00:00
|
|
|
cross
|
|
|
|
|
cummax
|
|
|
|
|
cummin
|
|
|
|
|
cumprod
|
|
|
|
|
cumsum
|
|
|
|
|
diag
|
|
|
|
|
diag_embed
|
|
|
|
|
diagflat
|
|
|
|
|
diagonal
|
Implement `np.diff` for single order differences (#50569)
Summary:
Implements `np.diff` for single order differences only:
- method and function variants for `diff` and function variant for `diff_out`
- supports out variant, but not in-place since shape changes
- adds OpInfo entry, and test in `test_torch`
- automatic autograd because we are using the `Math` dispatch
_Update: we only support Tensors for prepend and append in this PR. See discussion below and comments for more details._
Currently there is a quirk in the c++ API based on how this is implemented: it is not possible to specify scalar prepend and appends without also specifying all 4 arguments.
That is because the goal is to match NumPy's diff signature of `diff(int n=1, int dim=-1, Union[Scalar, Tensor] prepend=None, Union[Scalar, Tensor] append)=None` where all arguments are optional, positional and in the correct order.
There are a couple blockers. One is c++ ambiguity. This prevents us from simply doing `diff(int n=1, int dim=-1, Scalar? prepend=None, Tensor? append=None)` etc for all combinations of {Tensor, Scalar} x {Tensor, Scalar}.
Why not have append, prepend not have default args and then write out the whole power set of {Tensor, Scalar, omitted} x {Tensor, Scalar, omitted} you might ask. Aside from having to write 18 overloads, this is actually illegal because arguments with defaults must come after arguments without defaults. This would mean having to write `diff(prepend, append, n, dim)` which is not desired. Finally writing out the entire power set of all arguments n, dim, prepend, append is out of the question because that would actually involve 2 * 2 * 3 * 3 = 36 combinations. And if we include the out variant, that would be 72 overloads!
With this in mind, the current way this is implemented is actually to still do `diff(int n=1, int dim=-1, Scalar? prepend=None, Tensor? append=None)`. But also make use of `cpp_no_default_args`. The idea is to only have one of the 4 {Tensor, Scalar} x {Tensor, Scalar} provide default arguments for the c++ api, and add `cpp_no_default_args` for the remaining 3 overloads. With this, Python api works as expected, but some calls such as `diff(prepend=1)` won't work on c++ api.
We can optionally add 18 more overloads that cover the {dim, n, no-args} x {scalar-tensor, tensor-scalar, scalar-scalar} x {out, non-out} cases for c++ api. _[edit: counting is hard - just realized this number is still wrong. We should try to count the cases we do cover instead and subtract that from the total: (2 * 2 * 3 * 3) - (3 + 2^4) = 17. 3 comes from the 3 of 4 combinations of {tensor, scalar}^2 that we declare to be `cpp_no_default_args`, and the one remaining case that has default arguments has covers 2^4 cases. So actual count is 34 additional overloads to support all possible calls]_
_[edit: thanks to https://github.com/pytorch/pytorch/issues/50767 hacky_wrapper is no longer necessary; it is removed in the latest commit]_
hacky_wrapper was also necessary here because `Tensor?` will cause dispatch to look for the `const optional<Tensor>&` schema but also generate a `const Tensor&` declaration in Functions.h. hacky_wrapper allows us to define our function as `const Tensor&` but wraps it in optional for us, so this avoids both the errors while linking and loading.
_[edit: rewrote the above to improve clarity and correct the fact that we actually need 18 more overloads (26 total), not 18 in total to complete the c++ api]_
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50569
Reviewed By: H-Huang
Differential Revision: D26176105
Pulled By: soulitzer
fbshipit-source-id: cd8e77cc2de1117c876cd71c29b312887daca33f
2021-02-03 04:20:15 +00:00
|
|
|
diff
|
2020-05-04 21:29:24 +00:00
|
|
|
einsum
|
|
|
|
|
flatten
|
|
|
|
|
flip
|
2020-06-09 14:15:25 +00:00
|
|
|
fliplr
|
|
|
|
|
flipud
|
2020-11-03 20:39:52 +00:00
|
|
|
kron
|
2020-05-04 21:29:24 +00:00
|
|
|
rot90
|
2020-07-16 03:54:53 +00:00
|
|
|
gcd
|
2020-05-04 21:29:24 +00:00
|
|
|
histc
|
2021-06-22 17:04:51 +00:00
|
|
|
histogram
|
2021-11-16 19:54:14 +00:00
|
|
|
histogramdd
|
2020-05-04 21:29:24 +00:00
|
|
|
meshgrid
|
2020-07-16 03:54:53 +00:00
|
|
|
lcm
|
2020-05-21 16:09:41 +00:00
|
|
|
logcumsumexp
|
2020-10-12 22:58:56 +00:00
|
|
|
ravel
|
2020-05-04 21:29:24 +00:00
|
|
|
renorm
|
|
|
|
|
repeat_interleave
|
|
|
|
|
roll
|
2020-05-12 04:52:24 +00:00
|
|
|
searchsorted
|
2020-05-04 21:29:24 +00:00
|
|
|
tensordot
|
|
|
|
|
trace
|
|
|
|
|
tril
|
|
|
|
|
tril_indices
|
|
|
|
|
triu
|
|
|
|
|
triu_indices
|
2022-07-29 09:48:38 +00:00
|
|
|
unflatten
|
2020-05-04 21:29:24 +00:00
|
|
|
vander
|
2020-06-22 22:13:37 +00:00
|
|
|
view_as_real
|
|
|
|
|
view_as_complex
|
Conjugate View (#54987)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54987
Based off of ezyang (https://github.com/pytorch/pytorch/pull/44799) and bdhirsh (https://github.com/pytorch/pytorch/pull/43702) 's prototype:
Here's a summary of the changes in this PR:
This PR adds a new dispatch key called Conjugate. This enables us to make conjugate operation a view and leverage the specialized library functions that fast path with the hermitian operation (conj + transpose).
1. Conjugate operation will now return a view with conj bit (1) for complex tensors and returns self for non-complex tensors as before. This also means `torch.view_as_real` will no longer be a view on conjugated complex tensors and is hence disabled. To fill the gap, we have added `torch.view_as_real_physical` which would return the real tensor agnostic of the conjugate bit on the input complex tensor. The information about conjugation on the old tensor can be obtained by calling `.is_conj()` on the new tensor.
2. NEW API:
a) `.conj()` -- now returning a view.
b) `.conj_physical()` -- does the physical conjugate operation. If the conj bit for input was set, you'd get `self.clone()`, else you'll get a new tensor with conjugated value in its memory.
c) `.conj_physical_()`, and `out=` variant
d) `.resolve_conj()` -- materializes the conjugation. returns self if the conj bit is unset, else returns a new tensor with conjugated values and conj bit set to 0.
e) `.resolve_conj_()` in-place version of (d)
f) `view_as_real_physical` -- as described in (1), it's functionally same as `view_as_real`, just that it doesn't error out on conjugated tensors.
g) `view_as_real` -- existing function, but now errors out on conjugated tensors.
3. Conjugate Fallback
a) Vast majority of PyTorch functions would currently use this fallback when they are called on a conjugated tensor.
b) This fallback is well equipped to handle the following cases:
- functional operation e.g., `torch.sin(input)`
- Mutable inputs and in-place operations e.g., `tensor.add_(2)`
- out-of-place operation e.g., `torch.sin(input, out=out)`
- Tensorlist input args
- NOTE: Meta tensors don't work with conjugate fallback.
4. Autograd
a) `resolve_conj()` is an identity function w.r.t. autograd
b) Everything else works as expected.
5. Testing:
a) All method_tests run with conjugate view tensors.
b) OpInfo tests that run with conjugate views
- test_variant_consistency_eager/jit
- gradcheck, gradgradcheck
- test_conj_views (that only run for `torch.cfloat` dtype)
NOTE: functions like `empty_like`, `zero_like`, `randn_like`, `clone` don't propagate the conjugate bit.
Follow up work:
1. conjugate view RFC
2. Add neg bit to re-enable view operation on conjugated tensors
3. Update linalg functions to call into specialized functions that fast path with the hermitian operation.
Test Plan: Imported from OSS
Reviewed By: VitalyFedyunin
Differential Revision: D28227315
Pulled By: anjali411
fbshipit-source-id: acab9402b9d6a970c6d512809b627a290c8def5f
2021-06-04 21:11:23 +00:00
|
|
|
resolve_conj
|
2021-07-13 20:49:22 +00:00
|
|
|
resolve_neg
|
2017-01-05 04:20:57 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
BLAS and LAPACK Operations
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
addbmm
|
|
|
|
|
addmm
|
|
|
|
|
addmv
|
|
|
|
|
addr
|
|
|
|
|
baddbmm
|
|
|
|
|
bmm
|
|
|
|
|
chain_matmul
|
|
|
|
|
cholesky
|
|
|
|
|
cholesky_inverse
|
|
|
|
|
cholesky_solve
|
|
|
|
|
dot
|
|
|
|
|
geqrf
|
|
|
|
|
ger
|
2020-12-03 19:36:04 +00:00
|
|
|
inner
|
2020-05-04 21:29:24 +00:00
|
|
|
inverse
|
|
|
|
|
det
|
|
|
|
|
logdet
|
|
|
|
|
slogdet
|
|
|
|
|
lu
|
|
|
|
|
lu_solve
|
|
|
|
|
lu_unpack
|
|
|
|
|
matmul
|
|
|
|
|
matrix_power
|
2020-08-18 21:12:49 +00:00
|
|
|
matrix_exp
|
2020-05-04 21:29:24 +00:00
|
|
|
mm
|
|
|
|
|
mv
|
|
|
|
|
orgqr
|
|
|
|
|
ormqr
|
Adds linalg.det alias, fixes outer alias, updates alias testing (#42802)
Summary:
This PR:
- updates test_op_normalization.py, which verifies that aliases are correctly translated in the JIT
- adds torch.linalg.det as an alias for torch.det
- moves the torch.linalg.outer alias to torch.outer (to be consistent with NumPy)
The torch.linalg.outer alias was put the linalg namespace erroneously as a placeholder since it's a "linear algebra op" according to NumPy but is actually still in the main NumPy namespace.
The updates to test_op_normalization are necessary. Previously it was using method_tests to generate tests, and method_tests assumes test suites using it also use the device generic framework, which test_op_normalization did not. For example, some ops require decorators like `skipCPUIfNoLapack`, which only works in device generic test classes. Moving test_op_normalization to the device generic framework also lets these tests run on CPU and CUDA.
Continued reliance on method_tests() is excessive since the test suite is only interested in testing aliasing, and a simpler and more readable `AliasInfo` class is used for the required information. An example impedance mismatch between method_tests and the new tests, for example, was how to handle ops in namespaces like torch.linalg.det. In the future this information will likely be folded into a common 'OpInfo' registry in the test suite.
The actual tests performed are similar to what they were previously: a scripted and traced version of the op is run and the test verifies that both graphs do not contain the alias name and do contain the aliased name.
The guidance for adding an alias has been updated accordingly.
cc mattip
Note:
ngimel suggests:
- deprecating and then removing the `torch.ger` name
- reviewing the implementation of `torch.outer`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/42802
Reviewed By: zou3519
Differential Revision: D23059883
Pulled By: mruberry
fbshipit-source-id: 11321c2a7fb283a6e7c0d8899849ad7476be42d1
2020-08-12 04:47:08 +00:00
|
|
|
outer
|
2020-05-04 21:29:24 +00:00
|
|
|
pinverse
|
|
|
|
|
qr
|
|
|
|
|
svd
|
|
|
|
|
svd_lowrank
|
|
|
|
|
pca_lowrank
|
|
|
|
|
lobpcg
|
|
|
|
|
trapz
|
2021-07-21 01:13:32 +00:00
|
|
|
trapezoid
|
2021-08-03 15:01:52 +00:00
|
|
|
cumulative_trapezoid
|
2020-05-04 21:29:24 +00:00
|
|
|
triangular_solve
|
2020-09-02 15:58:51 +00:00
|
|
|
vdot
|
2020-05-04 16:35:17 +00:00
|
|
|
|
2023-01-23 08:28:06 +00:00
|
|
|
Foreach Operations
|
|
|
|
|
~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
.. warning::
|
|
|
|
|
This API is in beta and subject to future changes.
|
|
|
|
|
Forward-mode AD is not supported.
|
|
|
|
|
|
|
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
_foreach_abs
|
|
|
|
|
_foreach_abs_
|
|
|
|
|
_foreach_acos
|
|
|
|
|
_foreach_acos_
|
|
|
|
|
_foreach_asin
|
|
|
|
|
_foreach_asin_
|
|
|
|
|
_foreach_atan
|
|
|
|
|
_foreach_atan_
|
|
|
|
|
_foreach_ceil
|
|
|
|
|
_foreach_ceil_
|
|
|
|
|
_foreach_cos
|
|
|
|
|
_foreach_cos_
|
|
|
|
|
_foreach_cosh
|
|
|
|
|
_foreach_cosh_
|
|
|
|
|
_foreach_erf
|
|
|
|
|
_foreach_erf_
|
|
|
|
|
_foreach_erfc
|
|
|
|
|
_foreach_erfc_
|
|
|
|
|
_foreach_exp
|
|
|
|
|
_foreach_exp_
|
|
|
|
|
_foreach_expm1
|
|
|
|
|
_foreach_expm1_
|
|
|
|
|
_foreach_floor
|
|
|
|
|
_foreach_floor_
|
|
|
|
|
_foreach_log
|
|
|
|
|
_foreach_log_
|
|
|
|
|
_foreach_log10
|
|
|
|
|
_foreach_log10_
|
|
|
|
|
_foreach_log1p
|
|
|
|
|
_foreach_log1p_
|
|
|
|
|
_foreach_log2
|
|
|
|
|
_foreach_log2_
|
|
|
|
|
_foreach_neg
|
|
|
|
|
_foreach_neg_
|
|
|
|
|
_foreach_tan
|
|
|
|
|
_foreach_tan_
|
|
|
|
|
_foreach_sin
|
|
|
|
|
_foreach_sin_
|
|
|
|
|
_foreach_sinh
|
|
|
|
|
_foreach_sinh_
|
|
|
|
|
_foreach_round
|
|
|
|
|
_foreach_round_
|
|
|
|
|
_foreach_sqrt
|
|
|
|
|
_foreach_sqrt_
|
|
|
|
|
_foreach_lgamma
|
|
|
|
|
_foreach_lgamma_
|
|
|
|
|
_foreach_frac
|
|
|
|
|
_foreach_frac_
|
|
|
|
|
_foreach_reciprocal
|
|
|
|
|
_foreach_reciprocal_
|
|
|
|
|
_foreach_sigmoid
|
|
|
|
|
_foreach_sigmoid_
|
|
|
|
|
_foreach_trunc
|
|
|
|
|
_foreach_trunc_
|
|
|
|
|
_foreach_zero_
|
|
|
|
|
|
2020-05-04 17:55:56 +00:00
|
|
|
Utilities
|
|
|
|
|
----------------------------------
|
2020-05-04 21:29:24 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
compiled_with_cxx11_abi
|
|
|
|
|
result_type
|
|
|
|
|
can_cast
|
|
|
|
|
promote_types
|
2021-01-22 19:17:28 +00:00
|
|
|
use_deterministic_algorithms
|
|
|
|
|
are_deterministic_algorithms_enabled
|
2021-11-11 20:44:12 +00:00
|
|
|
is_deterministic_algorithms_warn_only_enabled
|
|
|
|
|
set_deterministic_debug_mode
|
|
|
|
|
get_deterministic_debug_mode
|
2022-05-04 20:40:13 +00:00
|
|
|
set_float32_matmul_precision
|
|
|
|
|
get_float32_matmul_precision
|
2021-02-08 16:18:50 +00:00
|
|
|
set_warn_always
|
torch.mtia module for MTIA device backend (#123612)
MTIA device has its own Module in PyTorch now.
torch.mtia has following APIs similar to other backends. The lazy_init is also supported.
```
__all__ = [
"init",
"is_available",
"synchronize",
"device_count",
"current_device",
"current_stream",
"default_stream",
"set_stream",
"stream",
"device",
]
```
------------
For device management. We expand AccleratorHooksInterface to support generic device management and it can be used in both C++ and PyThon.
```
def _accelerator_hooks_device_count() -> _int: ...
def _accelerator_hooks_set_current_device(device_index: _int) -> None: ...
def _accelerator_hooks_get_current_device() -> _int : ...
def _accelerator_hooks_exchange_device(device_index: _int) -> _int : ...
def _accelerator_hooks_maybe_exchange_device(device_index: _int) -> _int : ...
```
---------
Adding get_device_module API to retrieve device modules for different device types.
```
def get_device_module(device: Optional[Union[torch.device, str]] = None)
```
---------
Pull Request resolved: https://github.com/pytorch/pytorch/pull/123612
Approved by: https://github.com/albanD
ghstack dependencies: #123611
2024-04-25 18:48:49 +00:00
|
|
|
get_device_module
|
2021-02-08 16:18:50 +00:00
|
|
|
is_warn_always_enabled
|
2020-09-18 20:23:52 +00:00
|
|
|
vmap
|
2020-11-16 19:39:33 +00:00
|
|
|
_assert
|
2023-01-17 23:18:33 +00:00
|
|
|
|
|
|
|
|
Symbolic Numbers
|
|
|
|
|
----------------
|
Implement SymBool (#92149)
We have known for a while that we should in principle support SymBool as a separate concept from SymInt and SymFloat ( in particular, every distinct numeric type should get its own API). However, recent work with unbacked SymInts in, e.g., https://github.com/pytorch/pytorch/pull/90985 have made this a priority to implement. The essential problem is that our logic for computing the contiguity of tensors performs branches on the passed in input sizes, and this causes us to require guards when constructing tensors from unbacked SymInts. Morally, this should not be a big deal because, we only really care about the regular (non-channels-last) contiguity of the tensor, which should be guaranteed since most people aren't calling `empty_strided` on the tensor, however, because we store a bool (not a SymBool, prior to this PR it doesn't exist) on TensorImpl, we are forced to *immediately* compute these values, even if the value ends up not being used at all. In particular, even when a user allocates a contiguous tensor, we still must compute channels-last contiguity (as some contiguous tensors are also channels-last contiguous, but others are not.)
This PR implements SymBool, and makes TensorImpl use SymBool to store the contiguity information in ExtraMeta. There are a number of knock on effects, which I now discuss below.
* I introduce a new C++ type SymBool, analogous to SymInt and SymFloat. This type supports logical and, logical or and logical negation. I support the bitwise operations on this class (but not the conventional logic operators) to make it clear that logical operations on SymBool are NOT short-circuiting. I also, for now, do NOT support implicit conversion of SymBool to bool (creating a guard in this case). This does matter too much in practice, as in this PR I did not modify the equality operations (e.g., `==` on SymInt) to return SymBool, so all preexisting implicit guards did not need to be changed. I also introduced symbolic comparison functions `sym_eq`, etc. on SymInt to make it possible to create SymBool. The current implementation of comparison functions makes it unfortunately easy to accidentally introduce guards when you do not mean to (as both `s0 == s1` and `s0.sym_eq(s1)` are valid spellings of equality operation); in the short term, I intend to prevent excess guarding in this situation by unit testing; in the long term making the equality operators return SymBool is probably the correct fix.
* ~~I modify TensorImpl to store SymBool for the `is_contiguous` fields and friends on `ExtraMeta`. In practice, this essentially meant reverting most of the changes from https://github.com/pytorch/pytorch/pull/85936 . In particular, the fields on ExtraMeta are no longer strongly typed; at the time I was particularly concerned about the giant lambda I was using as the setter getting a desynchronized argument order, but now that I have individual setters for each field the only "big list" of boolean arguments is in the constructor of ExtraMeta, which seems like an acceptable risk. The semantics of TensorImpl are now that we guard only when you actually attempt to access the contiguity of the tensor via, e.g., `is_contiguous`. By in large, the contiguity calculation in the implementations now needs to be duplicated (as the boolean version can short circuit, but the SymBool version cannot); you should carefully review the duplicate new implementations. I typically use the `identity` template to disambiguate which version of the function I need, and rely on overloading to allow for implementation sharing. The changes to the `compute_` functions are particularly interesting; for most of the functions, I preserved their original non-symbolic implementation, and then introduce a new symbolic implementation that is branch-less (making use of our new SymBool operations). However, `compute_non_overlapping_and_dense` is special, see next bullet.~~ This appears to cause performance problems, so I am leaving this to an update PR.
* (Update: the Python side pieces for this are still in this PR, but they are not wired up until later PRs.) While the contiguity calculations are relatively easy to write in a branch-free way, `compute_non_overlapping_and_dense` is not: it involves a sort on the strides. While in principle we can still make it go through by using a data oblivious sorting network, this seems like too much complication for a field that is likely never used (because typically, it will be obvious that a tensor is non overlapping and dense, because the tensor is contiguous.) So we take a different approach: instead of trying to trace through the logic computation of non-overlapping and dense, we instead introduce a new opaque operator IsNonOverlappingAndDenseIndicator which represents all of the compute that would have been done here. This function returns an integer 0 if `is_non_overlapping_and_dense` would have returned `False`, and an integer 1 otherwise, for technical reasons (Sympy does not easily allow defining custom functions that return booleans). The function itself only knows how to evaluate itself if all of its arguments are integers; otherwise it is left unevaluated. This means we can always guard on it (as `size_hint` will always be able to evaluate through it), but otherwise its insides are left a black box. We typically do NOT expect this custom function to show up in actual boolean expressions, because we will typically shortcut it due to the tensor being contiguous. It's possible we should apply this treatment to all of the other `compute_` operations, more investigation necessary. As a technical note, because this operator takes a pair of a list of SymInts, we need to support converting `ArrayRef<SymNode>` to Python, and I also unpack the pair of lists into a single list because I don't know if Sympy operations can actually validly take lists of Sympy expressions as inputs. See for example `_make_node_sizes_strides`
* On the Python side, we also introduce a SymBool class, and update SymNode to track bool as a valid pytype. There is some subtlety here: bool is a subclass of int, so one has to be careful about `isinstance` checks (in fact, in most cases I replaced `isinstance(x, int)` with `type(x) is int` for expressly this reason.) Additionally, unlike, C++, I do NOT define bitwise inverse on SymBool, because it does not do the correct thing when run on booleans, e.g., `~True` is `-2`. (For that matter, they don't do the right thing in C++ either, but at least in principle the compiler can warn you about it with `-Wbool-operation`, and so the rule is simple in C++; only use logical operations if the types are statically known to be SymBool). Alas, logical negation is not overrideable, so we have to introduce `sym_not` which must be used in place of `not` whenever a SymBool can turn up. To avoid confusion with `__not__` which may imply that `operators.__not__` might be acceptable to use (it isn't), our magic method is called `__sym_not__`. The other bitwise operators `&` and `|` do the right thing with booleans and are acceptable to use.
* There is some annoyance working with booleans in Sympy. Unlike int and float, booleans live in their own algebra and they support less operations than regular numbers. In particular, `sympy.expand` does not work on them. To get around this, I introduce `safe_expand` which only calls expand on operations which are known to be expandable.
TODO: this PR appears to greatly regress performance of symbolic reasoning. In particular, `python test/functorch/test_aotdispatch.py -k max_pool2d` performs really poorly with these changes. Need to investigate.
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92149
Approved by: https://github.com/albanD, https://github.com/Skylion007
2023-01-20 19:10:29 +00:00
|
|
|
.. autoclass:: SymInt
|
|
|
|
|
:members:
|
|
|
|
|
|
|
|
|
|
.. autoclass:: SymFloat
|
|
|
|
|
:members:
|
|
|
|
|
|
|
|
|
|
.. autoclass:: SymBool
|
|
|
|
|
:members:
|
|
|
|
|
|
2023-01-17 23:18:33 +00:00
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
2022-12-27 21:59:38 +00:00
|
|
|
sym_float
|
2024-12-05 12:13:33 +00:00
|
|
|
sym_fresh_size
|
2022-12-27 21:59:38 +00:00
|
|
|
sym_int
|
2023-01-17 23:18:33 +00:00
|
|
|
sym_max
|
|
|
|
|
sym_min
|
Implement SymBool (#92149)
We have known for a while that we should in principle support SymBool as a separate concept from SymInt and SymFloat ( in particular, every distinct numeric type should get its own API). However, recent work with unbacked SymInts in, e.g., https://github.com/pytorch/pytorch/pull/90985 have made this a priority to implement. The essential problem is that our logic for computing the contiguity of tensors performs branches on the passed in input sizes, and this causes us to require guards when constructing tensors from unbacked SymInts. Morally, this should not be a big deal because, we only really care about the regular (non-channels-last) contiguity of the tensor, which should be guaranteed since most people aren't calling `empty_strided` on the tensor, however, because we store a bool (not a SymBool, prior to this PR it doesn't exist) on TensorImpl, we are forced to *immediately* compute these values, even if the value ends up not being used at all. In particular, even when a user allocates a contiguous tensor, we still must compute channels-last contiguity (as some contiguous tensors are also channels-last contiguous, but others are not.)
This PR implements SymBool, and makes TensorImpl use SymBool to store the contiguity information in ExtraMeta. There are a number of knock on effects, which I now discuss below.
* I introduce a new C++ type SymBool, analogous to SymInt and SymFloat. This type supports logical and, logical or and logical negation. I support the bitwise operations on this class (but not the conventional logic operators) to make it clear that logical operations on SymBool are NOT short-circuiting. I also, for now, do NOT support implicit conversion of SymBool to bool (creating a guard in this case). This does matter too much in practice, as in this PR I did not modify the equality operations (e.g., `==` on SymInt) to return SymBool, so all preexisting implicit guards did not need to be changed. I also introduced symbolic comparison functions `sym_eq`, etc. on SymInt to make it possible to create SymBool. The current implementation of comparison functions makes it unfortunately easy to accidentally introduce guards when you do not mean to (as both `s0 == s1` and `s0.sym_eq(s1)` are valid spellings of equality operation); in the short term, I intend to prevent excess guarding in this situation by unit testing; in the long term making the equality operators return SymBool is probably the correct fix.
* ~~I modify TensorImpl to store SymBool for the `is_contiguous` fields and friends on `ExtraMeta`. In practice, this essentially meant reverting most of the changes from https://github.com/pytorch/pytorch/pull/85936 . In particular, the fields on ExtraMeta are no longer strongly typed; at the time I was particularly concerned about the giant lambda I was using as the setter getting a desynchronized argument order, but now that I have individual setters for each field the only "big list" of boolean arguments is in the constructor of ExtraMeta, which seems like an acceptable risk. The semantics of TensorImpl are now that we guard only when you actually attempt to access the contiguity of the tensor via, e.g., `is_contiguous`. By in large, the contiguity calculation in the implementations now needs to be duplicated (as the boolean version can short circuit, but the SymBool version cannot); you should carefully review the duplicate new implementations. I typically use the `identity` template to disambiguate which version of the function I need, and rely on overloading to allow for implementation sharing. The changes to the `compute_` functions are particularly interesting; for most of the functions, I preserved their original non-symbolic implementation, and then introduce a new symbolic implementation that is branch-less (making use of our new SymBool operations). However, `compute_non_overlapping_and_dense` is special, see next bullet.~~ This appears to cause performance problems, so I am leaving this to an update PR.
* (Update: the Python side pieces for this are still in this PR, but they are not wired up until later PRs.) While the contiguity calculations are relatively easy to write in a branch-free way, `compute_non_overlapping_and_dense` is not: it involves a sort on the strides. While in principle we can still make it go through by using a data oblivious sorting network, this seems like too much complication for a field that is likely never used (because typically, it will be obvious that a tensor is non overlapping and dense, because the tensor is contiguous.) So we take a different approach: instead of trying to trace through the logic computation of non-overlapping and dense, we instead introduce a new opaque operator IsNonOverlappingAndDenseIndicator which represents all of the compute that would have been done here. This function returns an integer 0 if `is_non_overlapping_and_dense` would have returned `False`, and an integer 1 otherwise, for technical reasons (Sympy does not easily allow defining custom functions that return booleans). The function itself only knows how to evaluate itself if all of its arguments are integers; otherwise it is left unevaluated. This means we can always guard on it (as `size_hint` will always be able to evaluate through it), but otherwise its insides are left a black box. We typically do NOT expect this custom function to show up in actual boolean expressions, because we will typically shortcut it due to the tensor being contiguous. It's possible we should apply this treatment to all of the other `compute_` operations, more investigation necessary. As a technical note, because this operator takes a pair of a list of SymInts, we need to support converting `ArrayRef<SymNode>` to Python, and I also unpack the pair of lists into a single list because I don't know if Sympy operations can actually validly take lists of Sympy expressions as inputs. See for example `_make_node_sizes_strides`
* On the Python side, we also introduce a SymBool class, and update SymNode to track bool as a valid pytype. There is some subtlety here: bool is a subclass of int, so one has to be careful about `isinstance` checks (in fact, in most cases I replaced `isinstance(x, int)` with `type(x) is int` for expressly this reason.) Additionally, unlike, C++, I do NOT define bitwise inverse on SymBool, because it does not do the correct thing when run on booleans, e.g., `~True` is `-2`. (For that matter, they don't do the right thing in C++ either, but at least in principle the compiler can warn you about it with `-Wbool-operation`, and so the rule is simple in C++; only use logical operations if the types are statically known to be SymBool). Alas, logical negation is not overrideable, so we have to introduce `sym_not` which must be used in place of `not` whenever a SymBool can turn up. To avoid confusion with `__not__` which may imply that `operators.__not__` might be acceptable to use (it isn't), our magic method is called `__sym_not__`. The other bitwise operators `&` and `|` do the right thing with booleans and are acceptable to use.
* There is some annoyance working with booleans in Sympy. Unlike int and float, booleans live in their own algebra and they support less operations than regular numbers. In particular, `sympy.expand` does not work on them. To get around this, I introduce `safe_expand` which only calls expand on operations which are known to be expandable.
TODO: this PR appears to greatly regress performance of symbolic reasoning. In particular, `python test/functorch/test_aotdispatch.py -k max_pool2d` performs really poorly with these changes. Need to investigate.
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92149
Approved by: https://github.com/albanD, https://github.com/Skylion007
2023-01-20 19:10:29 +00:00
|
|
|
sym_not
|
2023-10-20 22:47:13 +00:00
|
|
|
sym_ite
|
2024-10-23 07:30:45 +00:00
|
|
|
sym_sum
|
2022-03-10 22:21:03 +00:00
|
|
|
|
2023-08-16 10:47:23 +00:00
|
|
|
Export Path
|
|
|
|
|
-------------
|
|
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
.. warning::
|
|
|
|
|
This feature is a prototype and may have compatibility breaking changes in the future.
|
|
|
|
|
|
|
|
|
|
export
|
|
|
|
|
generated/exportdb/index
|
|
|
|
|
|
2023-10-06 21:40:21 +00:00
|
|
|
Control Flow
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
|
|
.. warning::
|
|
|
|
|
This feature is a prototype and may have compatibility breaking changes in the future.
|
|
|
|
|
|
|
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
cond
|
|
|
|
|
|
2022-12-01 20:17:50 +00:00
|
|
|
Optimizations
|
|
|
|
|
-------------
|
|
|
|
|
.. autosummary::
|
|
|
|
|
:toctree: generated
|
|
|
|
|
:nosignatures:
|
|
|
|
|
|
|
|
|
|
compile
|
|
|
|
|
|
2024-03-15 15:58:16 +00:00
|
|
|
`torch.compile documentation <https://pytorch.org/docs/main/torch.compiler.html>`__
|
2023-03-15 04:41:09 +00:00
|
|
|
|
2022-06-10 21:48:56 +00:00
|
|
|
Operator Tags
|
|
|
|
|
------------------------------------
|
|
|
|
|
.. autoclass:: Tag
|
|
|
|
|
:members:
|
2022-03-10 22:21:03 +00:00
|
|
|
|
|
|
|
|
.. Empty submodules added only for tracking.
|
|
|
|
|
.. py:module:: torch.contrib
|
|
|
|
|
.. py:module:: torch.utils.backcompat
|
|
|
|
|
|
|
|
|
|
.. This module is only used internally for ROCm builds.
|
|
|
|
|
.. py:module:: torch.utils.hipify
|
|
|
|
|
|
|
|
|
|
.. This module needs to be documented. Adding here in the meantime
|
|
|
|
|
.. for tracking purposes
|
|
|
|
|
.. py:module:: torch.utils.model_dump
|
2023-06-22 18:08:31 +00:00
|
|
|
.. py:module:: torch.utils.viz
|
2023-10-06 14:16:01 +00:00
|
|
|
.. py:module:: torch.functional
|
|
|
|
|
.. py:module:: torch.quasirandom
|
|
|
|
|
.. py:module:: torch.return_types
|
|
|
|
|
.. py:module:: torch.serialization
|
|
|
|
|
.. py:module:: torch.signal.windows.windows
|
|
|
|
|
.. py:module:: torch.sparse.semi_structured
|
|
|
|
|
.. py:module:: torch.storage
|
|
|
|
|
.. py:module:: torch.torch_version
|
|
|
|
|
.. py:module:: torch.types
|
2023-10-20 22:47:13 +00:00
|
|
|
.. py:module:: torch.version
|