saymrwulf/pytorch
1
0
Fork 0
mirror of https://github.com/saymrwulf/pytorch.git synced 2026-05-14 20:57:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
pytorch/tools/stats
History
Xuehai Pan b6bdb67f82 [BE][Easy] use pathlib.Path instead of dirname / ".." / pardir (#129374) 
Changes by apply order:

1. Replace all `".."` and `os.pardir` usage with `os.path.dirname(...)`.
2. Replace nested `os.path.dirname(os.path.dirname(...))` call with `str(Path(...).parent.parent)`.
3. Reorder `.absolute()` ~/ `.resolve()`~ and `.parent`: always resolve the path first.

    `.parent{...}.absolute()` -> `.absolute().parent{...}`

4. Replace chained `.parent x N` with `.parents[${N - 1}]`: the code is easier to read (see 5.)

    `.parent.parent.parent.parent` -> `.parents[3]`

5. ~Replace `.parents[${N - 1}]` with `.parents[${N} - 1]`: the code is easier to read and does not introduce any runtime overhead.~

    ~`.parents[3]` -> `.parents[4 - 1]`~

6. ~Replace `.parents[2 - 1]` with `.parent.parent`: because the code is shorter and easier to read.~

Pull Request resolved: https://github.com/pytorch/pytorch/pull/129374
Approved by: https://github.com/justinchuby, https://github.com/malfet
2024-12-29 17:23:13 +00:00
..
__init__.py
check_disabled_tests.py
export_test_times.py [BE][Easy] use pathlib.Path instead of dirname / ".." / pardir (#129374) 2024-12-29 17:23:13 +00:00
import_test_stats.py [BE][Easy] use pathlib.Path instead of dirname / ".." / pardir (#129374) 2024-12-29 17:23:13 +00:00
monitor.py Fix unused Python variables outside torch/ and test/ (#136359) 2024-12-11 17:10:23 +00:00
README.md
sccache_stats_to_benchmark_format.py Revert "Use absolute path path.resolve() -> path.absolute() (#129409)" 2024-12-26 17:26:06 +00:00
test_dashboard.py Upload all run attempts when in upload_test_stats_intermediate (#140459) 2024-11-18 21:40:10 +00:00
upload_artifacts.py Revert "Use absolute path path.resolve() -> path.absolute() (#129409)" 2024-12-26 17:26:06 +00:00
upload_dynamo_perf_stats.py
upload_external_contrib_stats.py
upload_metrics.py
upload_sccache_stats.py
upload_stats_lib.py
upload_test_stats.py
upload_test_stats_intermediate.py Upload all run attempts when in upload_test_stats_intermediate (#140459) 2024-11-18 21:40:10 +00:00
upload_test_stats_running_jobs.py Continuous job for pulling artifacts and doing upload (#140453) 2024-11-20 20:41:52 +00:00

README.md

PyTorch CI Stats

We track various stats about each CI job.

  1. Jobs upload their artifacts to an intermediate data store (either GitHub Actions artifacts or S3, depending on what permissions the job has). Example: a9f6a35a33/.github/workflows/_linux-build.yml (L144-L151)
  2. When a workflow completes, a workflow_run event triggers upload-test-stats.yml.
  3. upload-test-stats downloads the raw stats from the intermediate data store and uploads them as JSON to s3, which then uploads to our database backend
graph LR
    J1[Job with AWS creds<br>e.g. linux, win] --raw stats--> S3[(AWS S3)]
    J2[Job w/o AWS creds<br>e.g. mac] --raw stats--> GHA[(GH artifacts)]

    S3 --> uts[upload-test-stats.yml]
    GHA --> uts

    uts --json--> s3[(s3)]
    s3 --> DB[(database)]

Why this weird indirection? Because writing to the database requires special permissions which, for security reasons, we do not want to give to pull request CI. Instead, we implemented GitHub's recommended pattern for cases like this.

For more details about what stats we export, check out upload-test-stats.yml