pytorch/test
Yidi Wu bdc2c2a237 [be] fix flaky test aot_export_ cond caused by free symbol lifting and automatic dynamic shape (#145330)
Fixes https://github.com/pytorch/pytorch/issues/139998#issuecomment-2605908426.

It seems to be an issue caused by the interaction between dynamoed hop X automatic dynamic shape X auto_lift_free symbols. The immediate error is that the asserteExpectedInline of the graph can sometimes be different e.g. see https://hud.pytorch.org/flakytest?name=test_aot_export_with_torch_cond&suite=TestAOTExport&limit=100, where sometimes the shapes are lifted as input to the cond and sometimes they're not.

The root cause of the flakyness is that the two invocations of torch.cond triggers two torch.compile on the same code object ([code](https://github.com/pytorch/pytorch/blob/main/torch/_higher_order_ops/cond.py#L192)), and triggers automatic dynamic shape because in test_aot_export_with_torch_cond, x has shape (3, 4) while the pre_dispatch one has shape (2, 2). Because of we auto lift free symbols for dynamic shaped input, this causes cond sometimes have the shape as arguments and sometimes not.

This PR adds a simple fix by adding a _dynamo.reset before each torch.cond tests. This fixes the error by not triggering automatic dynamic shape.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/145330
Approved by: https://github.com/zou3519
2025-01-23 18:12:58 +00:00
..
ao/sparsity [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
autograd
backends/xeon
benchmark_utils PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
bottleneck_test
cpp [c10d] Fix CudaEventCache for dangling references (#144496) 2025-01-15 05:11:48 +00:00
cpp_api_parity
cpp_extensions PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
custom_backend
custom_operator PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
distributed PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
distributions use torch.special.xlogy to implement x_log_x (#144220) 2025-01-08 17:41:55 +00:00
dynamo Adapt Dynamo tests to HPUs using instantiate_device_type_tests (#144387) 2025-01-23 09:24:42 +00:00
dynamo_expected_failures [BE] Fix edge case in translation validation bisector (#145414) 2025-01-23 17:35:28 +00:00
dynamo_skips Move Dynamo test to skip from expected_failures (#145390) 2025-01-22 19:06:39 +00:00
edge
error_messages
expect Reverting the PR adding Kleidiai-based int4 kernels (#145392) 2025-01-22 20:11:49 +00:00
export [BE][export] Change custom_op registeration style (#145315) 2025-01-22 23:46:51 +00:00
forward_backward_compatibility
functorch [be] fix flaky test aot_export_ cond caused by free symbol lifting and automatic dynamic shape (#145330) 2025-01-23 18:12:58 +00:00
fx PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
higher_order_ops PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
inductor Enable non power of 2 head_dim for FlexAttention (#133495) 2025-01-23 17:05:38 +00:00
inductor_expected_failures [BE] Remove test_modules from FIXME_inductor_dont_reset_dynamo (#145306) 2025-01-23 06:37:21 +00:00
inductor_skips [BE] Remove test_ops_gradients from FIXME_inductor_dont_reset_dynamo (#145308) 2025-01-23 17:25:04 +00:00
jit Revert "Use absolute path path.resolve() -> path.absolute() (#129409)" 2025-01-04 14:17:20 +00:00
jit_hooks
lazy [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
mobile PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
nn Revert "Align CPU behavior with CUDA for ConvTranspose when out_channels=0 (#142859)" 2025-01-23 01:10:31 +00:00
onnx PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
optim Adding support for differentiable lr, weight_decay, and betas in Adam/AdamW (#143726) 2024-12-30 01:11:57 +00:00
package Revert "Use absolute path path.resolve() -> path.absolute() (#129409)" 2025-01-04 14:17:20 +00:00
profiler [Profiler] Hide Kineto Step Tracker Behind Env Var (#144494) 2025-01-10 07:00:56 +00:00
quantization PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
scripts
test_img
torch_np [dynamo] Fix numpy test accuracy error induced by randomness divergence (#145293) 2025-01-22 20:53:02 +00:00
typing PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
xpu
_test_bazel.py
allowlist_for_publicAPI.json Revert "Add flop formula for _scaled_mm (#144872)" 2025-01-16 15:16:18 +00:00
conftest.py PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
create_dummy_torchscript_model.py
delete.py
hi.py
HowToWriteTestsUsingFileCheck.md
linear.py
load_torchscript_model.py
minioptest_failures_dict.json
mkl_verbose.py
mkldnn_verbose.py
pytest_shard_custom.py
run_doctests.sh
run_test.py Move privateuse1 test out of test_utils and make them serial (#145380) 2025-01-23 00:31:39 +00:00
simulate_nccl_errors.py
slow_tests.json Update slow tests (#144670) 2025-01-13 12:06:22 +00:00
test_accelerator.py Generalize pin memory logic for accelerator when non blocking copy happened (#143783) 2025-01-23 03:43:05 +00:00
test_ao_sparsity.py
test_autocast.py
test_autograd.py Allow GradientEdge as torch.autograd.backward outputs (#144744) 2025-01-14 21:31:44 +00:00
test_autograd_fallback.py
test_autoload.py
test_binary_ufuncs.py
test_bundled_images.py
test_bundled_inputs.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_ci_sanity_check_fail.py
test_comparison_utils.py
test_compile_benchmark_util.py
test_complex.py
test_content_store.py
test_cpp_api_parity.py
test_cpp_extensions_aot.py
test_cpp_extensions_jit.py
test_cpp_extensions_mtia_backend.py
test_cpp_extensions_open_device_registration.py Update pin memory related APIs to not pass 'device' argument (#131858) 2025-01-15 17:23:35 +00:00
test_cpp_extensions_stream_and_event.py
test_cuda.py [CUDA][cuBLAS] Add fp16 accumulate option to cuBLAS/cuBLASLt (#144441) 2025-01-22 22:42:48 +00:00
test_cuda_expandable_segments.py Revert "Use absolute path path.resolve() -> path.absolute() (#129409)" 2025-01-04 14:17:20 +00:00
test_cuda_multigpu.py Add get_stream_from_external API for CUDA backend (#143799) 2024-12-31 11:15:59 +00:00
test_cuda_nvml_based_avail.py
test_cuda_primary_ctx.py
test_cuda_sanitizer.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_cuda_trace.py
test_custom_ops.py PEP585 update - torch/_dynamo (#145105) 2025-01-18 20:47:11 +00:00
test_dataloader.py Update pin memory related APIs to not pass 'device' argument (#131858) 2025-01-15 17:23:35 +00:00
test_datapipe.py PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
test_decomp.py
test_deploy.py
test_determination.py
test_dispatch.py
test_dlpack.py
test_dynamic_shapes.py
test_expanded_weights.py
test_extension_utils.py Move privateuse1 test out of test_utils and make them serial (#145380) 2025-01-23 00:31:39 +00:00
test_fake_tensor.py Revert "Output of nonzero is transposed, fix fake tensor (#144695)" 2025-01-22 23:04:50 +00:00
test_file_check.py
test_flop_counter.py Add flop formula for _scaled_mm (#144973) 2025-01-17 09:38:30 +00:00
test_foreach.py
test_function_schema.py
test_functional_autograd_benchmark.py
test_functional_optim.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_functionalization.py
test_functionalization_of_rng_ops.py
test_futures.py
test_fx.py Fix for failure in D68425364 (#145304) 2025-01-22 23:33:02 +00:00
test_fx_experimental.py PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
test_fx_passes.py
test_fx_reinplace_pass.py
test_hub.py
test_import_stats.py
test_indexing.py
test_itt.py
test_jit.py
test_jit_autocast.py PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
test_jit_disabled.py
test_jit_fuser.py
test_jit_fuser_legacy.py
test_jit_fuser_te.py PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
test_jit_legacy.py
test_jit_llga_fuser.py
test_jit_profiling.py
test_jit_simple.py
test_jit_string.py PEP585 update - test (#145176) 2025-01-22 04:48:28 +00:00
test_jiterator.py
test_kernel_launch_checks.py
test_legacy_vmap.py
test_license.py
test_linalg.py Reverting the PR adding Kleidiai-based int4 kernels (#145392) 2025-01-22 20:11:49 +00:00
test_logging.py
test_masked.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_maskedtensor.py
test_matmul_cuda.py [CUDA][cuBLAS] Add fp16 accumulate option to cuBLAS/cuBLASLt (#144441) 2025-01-22 22:42:48 +00:00
test_meta.py [inductor] fix index.Tensor fallback (#144736) 2025-01-16 09:38:29 +00:00
test_metal.py
test_mkl_verbose.py
test_mkldnn.py
test_mkldnn_fusion.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_mkldnn_verbose.py
test_mobile_optimizer.py
test_model_exports_to_core_aten.py
test_module_tracker.py
test_modules.py
test_monitor.py
test_mps.py [MPS] Support includes in metal objects (#145087) 2025-01-18 05:35:22 +00:00
test_multiprocessing.py [dynamo] Re-enable test_fs family for dynamo (#145302) 2025-01-22 17:50:05 +00:00
test_multiprocessing_spawn.py
test_namedtensor.py
test_namedtuple_return_api.py
test_native_functions.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_native_mha.py
test_nestedtensor.py Implement backward for NJT matmul (#144587) 2025-01-21 18:27:50 +00:00
test_nn.py Fix boundary conditions for hardswish backward (#143899) 2025-01-16 20:26:27 +00:00
test_nnapi.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_numba_integration.py
test_numpy_interop.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_openmp.py
test_ops.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_ops_fwd_gradients.py
test_ops_gradients.py
test_ops_jit.py
test_optim.py Fix loading older state_dict into AdamW after refactor (#144972) 2025-01-16 19:50:31 +00:00
test_out_dtype_op.py
test_overrides.py Add support for torch function on dtype arguments (#145085) 2025-01-21 17:44:47 +00:00
test_package.py
test_per_overload_api.py
test_prims.py
test_proxy_tensor.py detect fake mode in proxy_tensor creation in make_fx (#144168) 2025-01-06 19:02:08 +00:00
test_pruning_op.py
test_public_bindings.py Fix test_modules_can_be_imported (#145387) 2025-01-23 16:03:00 +00:00
test_python_dispatch.py Delete torch._library.register_functional_op (#145110) 2025-01-18 00:58:25 +00:00
test_pytree.py [TreeSpec] Support enum in defaultdict (#144235) 2025-01-07 00:10:46 +00:00
test_quantization.py
test_reductions.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_scatter_gather_ops.py
test_schema_check.py
test_segment_reductions.py
test_serialization.py Prevent legacy_load when weights_only=True (correctly) (#145020) 2025-01-17 20:10:22 +00:00
test_set_default_mobile_cpu_allocator.py
test_shape_ops.py
test_show_pickle.py
test_sort_and_select.py [cpu/sorting] Throw an error when trying to sort complex numbers. (#144113) 2025-01-08 05:15:36 +00:00
test_sparse.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_sparse_csr.py Use random64 in Fischer-Yates algorithm for large N (#143682) 2025-01-07 03:48:56 +00:00
test_sparse_semi_structured.py
test_spectral_ops.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_stateless.py
test_static_runtime.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_subclass.py
test_sympy_utils.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_tensor_creation_ops.py restore rng generation for fbcode (#144819) 2025-01-16 06:46:26 +00:00
test_tensorboard.py
test_tensorexpr.py
test_tensorexpr_pybind.py
test_testing.py [BE] fix ruff rule E226: add missing whitespace around operator in f-strings (#144415) 2025-01-08 21:55:00 +00:00
test_throughput_benchmark.py
test_torch.py [ROCm] Enable post-merge trunk workflow on MI300 runners; skip and fix MI300 related failed tests (#143673) 2025-01-09 05:18:57 +00:00
test_transformers.py [1/N] OpenReg: Replace open_registration_extension.cpp with openreg (#141815) 2025-01-14 15:59:00 +00:00
test_type_hints.py Revert "Use absolute path path.resolve() -> path.absolute() (#129409)" 2025-01-04 14:17:20 +00:00
test_type_info.py
test_type_promotion.py
test_typing.py [4/N] Apply py39 ruff and pyupgrade fixes (#143257) 2025-01-04 10:47:51 +00:00
test_unary_ufuncs.py
test_utils.py Move privateuse1 test out of test_utils and make them serial (#145380) 2025-01-23 00:31:39 +00:00
test_utils_config_module.py [minifier] Fix config generator for callables (#144518) 2025-01-14 17:18:13 +00:00
test_utils_filelock.py
test_view_ops.py
test_vulkan.py
test_weak.py
test_xnnpack_integration.py
test_xpu.py Revert "Fix poision child process issue when call getAccelerator() (#144368)" 2025-01-10 23:36:43 +00:00