saymrwulf/pytorch
1
0
Fork 0
mirror of https://github.com/saymrwulf/pytorch.git synced 2026-05-15 21:00:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
pytorch/.github/scripts
History
Ning Xu 2c76838d7f Issue-88098: extract utils from check labels (#94597) 
Fixes #88098

This is a mirror of the same PR (https://github.com/Goldspear/pytorch/pull/2) that has been reviewed in my fork (due to it's a stacked PR).

======================
## Context

This is the 2nd of the 3 PRs to address issue-88098.

## What Changed
1. Extract comment related utils from trymerge.py to github_utils.py
2. Extract label related utils from trymerge.py and check_labels.py to label_utils.py

## Tests
* pytorch-dummy repo [trymerge run ](https://github.com/Goldspear/pytorch-dummy/actions/runs/4118944174)merged the test PR [OK](https://github.com/Goldspear/pytorch-dummy/pull/2).

## Note to Reviewers
Due to higher degree of complexity involved to extract GitHubPR class, it's worth having a separate issue to handle that part of refactoring. This issue only focusing on refactoring where necessary to ship the functional diff.

* 1st PR: https://github.com/pytorch/pytorch/pull/94179
* 2nd PR: this one
* 3rd PR: https://github.com/Goldspear/pytorch/pull/3

Pull Request resolved: https://github.com/pytorch/pytorch/pull/94597
Approved by: https://github.com/ZainRizvi
2023-02-12 12:18:53 +00:00
..
build_triton_wheel.py Extend torch-trition conda to 3.11 (#93117) 2023-02-06 22:14:57 +00:00
check_labels.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
collect_ciflow_labels.py [CI] Add known ciflow labels to probot (#94368) 2023-02-08 17:37:27 +00:00
comment_on_pr.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
convert_lintrunner_annotations_to_github.py
ensure_actions_will_cancel.py Disable mem leak check (#88373) 2022-11-04 20:47:42 +00:00
export_pytorch_labels.py Refactor to extract label_utils from export_pytorch_labels (#94179) 2023-02-09 19:17:05 +00:00
fetch_latest_green_commit.py Update rockset version (#94005) 2023-02-03 21:38:35 +00:00
filter_test_configs.py Add support for keep-going label (#90902) 2022-12-16 06:47:06 +00:00
generate_binary_build_matrix.py Enable Python-3.11 binary builds across the board (#94430) 2023-02-10 06:10:27 +00:00
generate_ci_workflows.py [BE] Import Literal, Protocol, and Final from standard library typing as of Python 3.8+ (#94490) 2023-02-09 19:17:49 +00:00
generate_pytorch_version.py Fix Docker image generation (#88741) 2022-11-10 00:06:31 +00:00
get_workflow_job_id.py [BE][ez] Move hardcoded constants to function args (#93874) 2023-02-02 00:47:18 +00:00
github_utils.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
gitutils.py [GHF] Fix ghstack branches in sync logic (#93298) 2023-01-30 21:00:51 +00:00
gql_mocks.json [GHF] Fix pushDate handling (#94364) 2023-02-08 05:52:03 +00:00
kill_active_ssh_sessions.ps1
label_utils.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
lint_native_functions.py
parse_ref.py [BE] Get rid of deprecation warnings in workflows (take 3) (#87152) 2022-10-18 13:53:30 +00:00
pr-sanity-check.sh .github: Improve sanity check for generated files (#86143) 2022-10-03 23:38:55 +00:00
README.md Refactor to extract label_utils from export_pytorch_labels (#94179) 2023-02-09 19:17:05 +00:00
report_git_status.sh
rockset_mocks.json trymerge to ignore certain failures (#91134) 2023-02-07 17:19:57 +00:00
run_torchbench.py Upload the benchmark result to S3 and post the URL (#84726) 2022-09-09 22:01:23 +00:00
test_check_labels.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
test_fetch_latest_green_commit.py Allow viable/strict promotion even if periodic or docker-release-builds jobs are failing (#86827) 2022-10-13 00:38:48 +00:00
test_filter_test_configs.py Add a mode to rerun all disabled tests (without running anything else) (#88646) 2022-11-15 05:08:26 +00:00
test_gitutils.py [GHF] Fix ghstack branches in sync logic (#93298) 2023-01-30 21:00:51 +00:00
test_label_utils.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
test_trymerge.py [GHF] Fix pushDate handling (#94364) 2023-02-08 05:52:03 +00:00
test_tryrebase.py Fix for tryrebase after PR was merged (#91337) 2022-12-28 18:44:08 +00:00
trymerge.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
trymerge_explainer.py [mergebot] Flatten workflows into jobs, fix bugs (#92097) 2023-01-18 23:38:16 +00:00
tryrebase.py Issue-88098: extract utils from check labels (#94597) 2023-02-12 12:18:53 +00:00
update_commit_hashes.py
wait_for_ssh_to_drain.ps1

README.md

pytorch/.github

NOTE: This README contains information for the .github directory but cannot be located there because it will overwrite the repo README.

This directory contains workflows and scripts to support our CI infrastructure that runs on GitHub Actions.

Workflows

  • Pull CI (pull.yml) is run on PRs and on master.
  • Trunk CI (trunk.yml) is run on trunk to validate incoming commits. Trunk jobs are usually more expensive to run so we do not run them on PRs unless specified.
  • Scheduled CI (periodic.yml) is a subset of trunk CI that is run every few hours on master.
  • Binary CI is run to package binaries for distribution for all platforms.

Templates

Templates written in Jinja are located in the .github/templates directory and used to generate workflow files for binary jobs found in the .github/workflows/ directory. These are also a couple of utility templates used to discern common utilities that can be used amongst different templates.

(Re)Generating workflow files

You will need jinja2 in order to regenerate the workflow files which can be installed using:

pip install -r .github/requirements/regenerate-requirements.txt

Workflows can be generated / regenerated using the following command:

.github/regenerate.sh

Adding a new generated binary workflow

New generated binary workflows can be added in the .github/scripts/generate_ci_workflows.py script. You can reference examples from that script in order to add the workflow to the stream that is relevant to what you particularly care about.

Different parameters can be used to achieve different goals, i.e. running jobs on a cron, running only on trunk, etc.

ciflow (trunk)

The label ciflow/trunk can be used to run trunk only workflows. This is especially useful if trying to re-land a PR that was reverted for failing a non-default workflow.

Infra

Currently most of our self hosted runners are hosted on AWS, for a comprehensive list of available runner types you can reference .github/scale-config.yml.

Exceptions to AWS for self hosted:

  • ROCM runners

Adding new runner types

New runner types can be added by committing changes to .github/scale-config.yml. Example: https://github.com/pytorch/pytorch/pull/70474

NOTE: New runner types can only be used once the changes to .github/scale-config.yml have made their way into the default branch

Testing pytorch/builder changes

In order to test changes to the builder scripts:

  1. Specify your builder PR's branch and repo as builder_repo and builder_branch in .github/templates/common.yml.j2.
  2. Regenerate workflow files with .github/regenerate.sh (see above).
  3. Submit fake PR to PyTorch. If changing binaries build, add an appropriate label like ciflow/binaries to trigger the builds.