mirror of
https://github.com/saymrwulf/pytorch.git
synced 2026-05-14 20:57:59 +00:00
f295eff512
Summary: To support iteration-based on-demand we have step tracker hooks for both the scheduler and for the optimizer to control Kineto's backend FSM. We already hide the optimizer step tracker behind and ENV_VAR to prevent any extra overhead from the frontend profiler down to the kineto backend, but we don't do any such thing for the profiler step tracker. It also seems to cause errors occasionally in the FSM having both auto-trace and on-demand occurring at the same time. To remedy this issue, lets put in a patch to guard the step incrementer for the frontend step function. This will bypass all of the on-demand logic which shouldn't occur in auto-trace Test Plan: Ran `buck run mode/dev-nosan kineto/libkineto/fb/integration_tests:pytorch_resnet_integration_test -- --enable_profiling --trace_handler=auto_trace --with_stack` and added prints in on-demand functions (performLoopStep and collectTrace) and saw that neither were called even though they were called on main. Also got following healthy traces: Auto-Trace (schedule-based): https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/traces/dynocli/devvm2185.cco0.facebook.com/rank-0.Jan_09_12_43_37.1122140.pt.trace.json.gz&bucket=gpu_traces Timing Based On-demand: https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/traces/dynocli/0/1736456722/localhost/libkineto_activities_1286261.json.gz&bucket=gpu_traces Iteration Based On-demand: https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/traces/dynocli/0/1736456889/localhost/libkineto_activities_1304781.json.gz&bucket=gpu_traces Differential Revision: D67990080 Pull Request resolved: https://github.com/pytorch/pytorch/pull/144494 Approved by: https://github.com/ngimel |
||
|---|---|---|
| .. | ||
| profiler_utils_mock_events.json | ||
| test_cpp_thread.cpp | ||
| test_cpp_thread.py | ||
| test_cpp_thread_lib.pyi | ||
| test_execution_trace.py | ||
| test_kineto.py | ||
| test_memory_profiler.py | ||
| test_profiler.py | ||
| test_profiler_tree.py | ||
| test_record_function.py | ||
| test_torch_tidy.py | ||