pytorch/test
Jane Xu f30a7a2739 Add export-historic-test-times option to dump S3 test times into a JSON file (#54083)
Summary:
This will allow for future work to use the test times file (which will save computation time and also allow for more consistency). (Step one to fixing https://github.com/pytorch/pytorch/issues/53882)

Pull Request resolved: https://github.com/pytorch/pytorch/pull/54083

Test Plan:
export CIRCLE_JOB=your-favorite-circleci-job e.g., pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2
`python test/run_test.py --export-historic-test-times` OR
`python test/run_test.py --export-historic-test-times .your-favorite-file`

When opening either .pytorch-test-times or .your-favorite-file, you should see something like:
```
{"commit": "2d559a09392aabb84dfb4a498010b2f01d99818c", "job_times": {"distributed/test_distributed_spawn": 583.5889999999973, "distributed/test_data_parallel": 4.866999999999997, "test_binary_ufuncs": 171.1569999999998, "test_numpy_interop": 2.5649999999999995, "test_public_bindings": 0.011,...}}
```

Note that no tests will be run when this option is specified.

Reviewed By: walterddr

Differential Revision: D27091351

Pulled By: janeyx99

fbshipit-source-id: e191d739268d86de0a0ba0eea0006969859d1940
2021-03-17 12:22:00 -07:00
..
backward_compatibility Faster backwards for cumsum and cumprod (#53711) 2021-03-16 13:57:43 -07:00
benchmark_utils Quality of life improvements to Timer (#53294) 2021-03-09 08:15:30 -08:00
bottleneck_test
cpp [nnc] Tests for proposed feature: loop bounds conditional simplification (#54121) 2021-03-17 11:01:10 -07:00
cpp_api_parity Huber loss (#50553) 2021-03-02 17:30:45 -08:00
cpp_extensions Pass Scalar by reference (#53583) 2021-03-15 23:17:06 -07:00
custom_backend Adds a bool is_available() method to the backend contract (#53068) 2021-03-10 00:24:16 -08:00
custom_operator [BE] move more unittest.main() to run_tests() (#50923) 2021-01-25 07:23:09 -08:00
distributed add tests for ddp with activation check points (#52894) 2021-03-17 10:16:20 -07:00
distributions Fix distributions which don't properly honor validate_args=False (#53600) 2021-03-10 13:16:32 -08:00
error_messages
expect [TB] Support writing new style scalar (#53496) 2021-03-12 19:03:13 -08:00
fx Test forward reference annotations (#53713) 2021-03-15 19:40:26 -07:00
jit [JIT] Update Namespace from cuda to _cuda (#53378) 2021-03-11 00:52:01 -08:00
jit_hooks [WIP] JIT Static Hooks: cpp tests (#49547) 2021-01-20 09:12:57 -08:00
mobile [pytorch] reintroduce static dispatch (#51957) 2021-02-19 11:41:39 -08:00
onnx [ONNX] Update embedding export wrt padding_idx (#53931) 2021-03-15 10:03:53 -07:00
optim
package [package] split tests (#53749) 2021-03-10 16:07:36 -08:00
quantization ns for fx: move linear activation test case to new API (#53777) 2021-03-12 10:02:52 -08:00
scripts
test_img
type_hint_tests Fix mypy type hint for AdaptiveAvgPool2,3d, AdaptiveMaxPool2,3d (#49963) 2021-01-06 09:47:15 -08:00
typing/reveal Port NumPy typing testing style to PyTorch (#52408) 2021-03-10 12:18:01 -08:00
HowToWriteTestsUsingFileCheck.md
linear.py Move conv to mkldnn (#51483) 2021-03-01 21:19:27 -08:00
run_test.py Add export-historic-test-times option to dump S3 test times into a JSON file (#54083) 2021-03-17 12:22:00 -07:00
simulate_nccl_errors.py
test_autograd.py Add support for single tensor in inputs argument for backward (#53827) 2021-03-12 08:19:31 -08:00
test_binary_ufuncs.py Modify tests to use assertWarnsOnceRegex instead of maybeWarnsRegex (#52387) 2021-03-08 03:32:14 -08:00
test_bundled_images.py
test_bundled_inputs.py [Pytorch Mobile] Improve Bundled Inputs Error Checking (#52386) 2021-02-24 13:55:45 -08:00
test_complex.py Add serialization logic for complex numbers (#51287) 2021-01-28 17:25:35 -08:00
test_cpp_api_parity.py
test_cpp_extensions_aot.py Marginally improve pytest collection for top-level test files (#53617) 2021-03-10 08:56:39 -08:00
test_cpp_extensions_jit.py Internal gradcheck wrapper in testing._internal that sets certain flags to True (#51133) 2021-01-29 09:13:37 -08:00
test_cuda.py [RELAND] [CUDA graphs] Private mempools for CUDA graphs (#54038) 2021-03-16 12:13:33 -07:00
test_cuda_primary_ctx.py
test_dataloader.py Make ideep honor torch.set_num_thread changes (#53871) 2021-03-13 11:20:44 -08:00
test_datapipe.py [DataLoader] ZipIterDataPipe (#53554) 2021-03-12 08:26:21 -08:00
test_dataset.py [DataLoader] Rename Functional DataSet to DataPipe (#51488) 2021-02-03 07:01:09 -08:00
test_determination.py Improve logic for S3 stats gathering. Uses automatic SLOW_TESTS. (#53549) 2021-03-10 09:37:06 -08:00
test_dispatch.py Improve docs around Math/DefaultBackend & add PythonDispatcher class. (#50854) 2021-01-25 23:10:36 -08:00
test_expecttest.py [BE] replace unittest.main with run_tests (#50451) 2021-01-13 10:33:08 -08:00
test_foreach.py Add division logic to a slow/fast path (#49250) 2021-03-15 12:17:39 -07:00
test_function_schema.py
test_functional_autograd_benchmark.py Skip test_functional_autograd_benchmark during code coverage (#50183) 2021-01-07 11:17:21 -08:00
test_futures.py [JIT/Futures] support set_exception api (#50983) 2021-02-04 20:22:19 -08:00
test_fx.py [fx][trivial] Add TestConstFold coverage to test_fx (#54072) 2021-03-17 10:38:54 -07:00
test_fx_experimental.py [GraphModule] Back out changes to module root version of __init__ (#53791) 2021-03-10 23:18:56 -08:00
test_indexing.py TST Replaces assertEqualIgnoreTypes with assertEqual in test_indexing (#53115) 2021-03-16 16:06:36 -07:00
test_jit.py Add meta device to generic device testing framework, skip NotImplementedError (#53682) 2021-03-14 20:41:19 -07:00
test_jit_cuda_fuser.py [ROCm] disable tests for ROCm 4.0.1 (#51510) 2021-02-02 14:39:08 -08:00
test_jit_disabled.py [BE] replace unittest.main with run_tests (#50451) 2021-01-13 10:33:08 -08:00
test_jit_fuser.py Clean up some type annotations in caffe2/test (#49943) 2021-01-13 10:01:55 -08:00
test_jit_fuser_legacy.py
test_jit_fuser_te.py add a full pipeline test for a TypeCheck (#52933) 2021-03-01 10:58:08 -08:00
test_jit_legacy.py
test_jit_profiling.py Remove redundant code for unsupported Python versions (#49486) 2021-01-06 12:45:46 -08:00
test_jit_py3.py Workaround Python3.9 limitations in test_jit_py3 (#51088) 2021-01-26 12:49:35 -08:00
test_jit_simple.py Remove redundant code for unsupported Python versions (#49486) 2021-01-06 12:45:46 -08:00
test_jit_string.py [JIT] Allow implicit boolean conversion of containers (#51683) 2021-02-05 00:34:35 -08:00
test_kernel_launch_checks.py Fix check_kernel_launches.py for macros and provide extended context (#49365) 2020-12-14 22:09:33 -08:00
test_license.py concantenate LICENSE files when building a wheel (#51634) 2021-02-08 08:28:46 -08:00
test_linalg.py Add meta device to generic device testing framework, skip NotImplementedError (#53682) 2021-03-14 20:41:19 -07:00
test_logging.py
test_metal.py
test_mkldnn.py fix mkldnn_add in-place behavior (#51687) 2021-03-16 12:54:27 -07:00
test_mobile_optimizer.py [Reland] Update XNNPACK (#52691) 2021-02-24 06:40:38 -08:00
test_multiprocessing.py [BE] unified run_process_no_exception code (#49774) 2021-01-04 13:43:09 -08:00
test_multiprocessing_spawn.py
test_namedtensor.py Enhance Tensor.unflatten to support -1 as the inferred size (#51955) 2021-02-18 08:37:41 -08:00
test_namedtuple_return_api.py Implements cpu_kernel_multiple_outputs and torch.frexp (#51097) 2021-03-15 10:44:32 -07:00
test_native_functions.py De-prioritise Dimname and DimnameList in python overload resolution (#51350) 2021-03-02 23:07:08 -08:00
test_nn.py Disabling dispatch to OneDNN for group convolutions when groups size = 24 * n (#53991) 2021-03-15 19:30:19 -07:00
test_numba_integration.py Drop unused imports from test (#49973) 2021-01-07 12:09:38 -08:00
test_numpy_interop.py Mypy fixes for pytorch master (#52090) 2021-02-17 10:39:51 -08:00
test_op_aliases.py OpInfo: Added clamp and trunc tests with aliases (#51167) 2021-02-10 05:36:18 -08:00
test_openmp.py Marginally improve pytest collection for top-level test files (#53617) 2021-03-10 08:56:39 -08:00
test_ops.py [testing] assert no duplicate in method_tests for an OpInfo entry (#53492) 2021-03-14 21:58:39 -07:00
test_optim.py [ROCm] Enable test in test_linalg.py, test_optim.py and test_vmap.py … (#52818) 2021-03-04 02:29:45 -08:00
test_overrides.py Marginally improve pytest collection for top-level test files (#53617) 2021-03-10 08:56:39 -08:00
test_package.py [package] split tests (#53749) 2021-03-10 16:07:36 -08:00
test_profiler.py Skip tests on ROCm (#53339) 2021-03-04 21:55:34 -08:00
test_pruning_op.py Rowwise Prune op (Add the test to OSS run_test), Make the op private. (#46131) 2021-01-29 06:08:18 -08:00
test_public_bindings.py [JIT] Add torch._C._jit submodule (#52910) 2021-02-26 16:05:05 -08:00
test_pytree.py Implement _broadcast_to_and_flatten(pytree, spec) (#46288) 2020-10-20 07:52:14 -07:00
test_quantization.py NS for FX: add test for a simple sparsenn model (#52092) 2021-02-18 08:17:57 -08:00
test_reductions.py Enabled amin & amax for float16 & bfloat16 (#52579) 2021-03-04 07:03:03 -08:00
test_serialization.py Enable tests using named temp files on Windows (#49640) 2020-12-29 09:57:35 -08:00
test_set_default_mobile_cpu_allocator.py [Pytorch] Add python binding to use mobile cpu allocator. (#52323) 2021-02-17 08:42:23 -08:00
test_shape_ops.py Removes nonzero method warning (#51618) 2021-02-04 17:43:43 -08:00
test_show_pickle.py [BE] replace unittest.main with run_tests (#50451) 2021-01-13 10:33:08 -08:00
test_sort_and_select.py Stable sort for the CPU take 2. (#51790) 2021-02-19 09:28:57 -08:00
test_sparse.py Modify tests to use assertWarnsOnceRegex instead of maybeWarnsRegex (#52387) 2021-03-08 03:32:14 -08:00
test_spectral_ops.py Add missing decorators in test_spectral_ops (#53736) 2021-03-12 12:00:25 -08:00
test_static_runtime.py [static runtime] Move all heavy constructor logic into InferenceModule (renamed to StaticModule) (#51564) 2021-03-05 10:15:26 -08:00
test_tensor_creation_ops.py Add meta device to generic device testing framework, skip NotImplementedError (#53682) 2021-03-14 20:41:19 -07:00
test_tensorboard.py [TB] Support writing new style scalar (#53496) 2021-03-12 19:03:13 -08:00
test_tensorexpr.py Add NNC support for aten::hardtanh (a hot operation in mobilenet v2/v3) (#52394) 2021-02-18 22:56:03 -08:00
test_tensorexpr_pybind.py [TensorExpr] Reland: "PyBind: bind ExternalCalls." (#53063) 2021-03-01 20:44:10 -08:00
test_testing.py Don't actually print anomalies in TTRR (#54078) 2021-03-16 14:26:32 -07:00
test_throughput_benchmark.py Enable tests using named temp files on Windows (#49640) 2020-12-29 09:57:35 -08:00
test_torch.py Also propagate storage_access_should_throw_ when copying tensor metadata (#53816) 2021-03-16 15:18:37 -07:00
test_type_hints.py Mypy fixes for pytorch master (#52090) 2021-02-17 10:39:51 -08:00
test_type_info.py
test_type_promotion.py [numpy] torch.erf{c} : promote integer inputs to float (#48472) 2020-11-27 15:08:40 -08:00
test_typing.py Port NumPy typing testing style to PyTorch (#52408) 2021-03-10 12:18:01 -08:00
test_unary_ufuncs.py Implements cpu_kernel_multiple_outputs and torch.frexp (#51097) 2021-03-15 10:44:32 -07:00
test_utils.py Store test file in S3 as well for every TestSuite (#52869) 2021-03-02 07:36:00 -08:00
test_view_ops.py Add tensor.view(dtype) (#47951) 2021-01-08 06:55:21 -08:00
test_vmap.py Revert "Revert D26733731: [pytorch][PR] Skip dispatch for `is_floatin… (#53242) 2021-03-11 09:46:25 -08:00
test_vulkan.py
test_xnnpack_integration.py