mirror of
https://github.com/saymrwulf/pytorch.git
synced 2026-05-14 20:57:59 +00:00
dab272eed8
Move the pytest cache downloading into the build step and store it in additional ci files so that it stays consistent during sharding. Only build env is taken into account now instead of also test config since we might not have the test config during build time, making it less specific, but I also think this might be better since tests are likely to fail across the same test config (I also think it might be worth not even looking at build env but thats a different topic) Each cache upload should only include information from the current run. Do not merge current cache with downloaded cache during upload (shouldn't matter anyways since the downloaded cache won't exist at the time) From what I cant tell of the s3 retention policy, pytest cache files will be deleted after 30 days (cc @ZainRizvi to confirm), so we never have to worry about space or pulling old versions. Pull Request resolved: https://github.com/pytorch/pytorch/pull/113804 Approved by: https://github.com/ZainRizvi |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| check_disabled_tests.py | ||
| export_test_times.py | ||
| import_test_stats.py | ||
| monitor.py | ||
| README.md | ||
| upload_artifacts.py | ||
| upload_dynamo_perf_stats.py | ||
| upload_external_contrib_stats.py | ||
| upload_metrics.py | ||
| upload_sccache_stats.py | ||
| upload_stats_lib.py | ||
| upload_test_stat_aggregates.py | ||
| upload_test_stats.py | ||
PyTorch CI Stats
We track various stats about each CI job.
- Jobs upload their artifacts to an intermediate data store (either GitHub
Actions artifacts or S3, depending on what permissions the job has). Example:
a9f6a35a33/.github/workflows/_linux-build.yml (L144-L151) - When a workflow completes, a
workflow_runevent triggersupload-test-stats.yml. upload-test-statsdownloads the raw stats from the intermediate data store and uploads them as JSON to Rockset, our metrics backend.
graph LR
J1[Job with AWS creds<br>e.g. linux, win] --raw stats--> S3[(AWS S3)]
J2[Job w/o AWS creds<br>e.g. mac] --raw stats--> GHA[(GH artifacts)]
S3 --> uts[upload-test-stats.yml]
GHA --> uts
uts --json--> R[(Rockset)]
Why this weird indirection? Because writing to Rockset requires special permissions which, for security reasons, we do not want to give to pull request CI. Instead, we implemented GitHub's recommended pattern for cases like this.
For more details about what stats we export, check out
upload-test-stats.yml