mirror of
https://github.com/saymrwulf/pytorch.git
synced 2026-05-15 21:00:47 +00:00
2ac6ee7f12
This is reopening of the PR https://github.com/pytorch/pytorch/pull/100377
# About this PR
Due to increased pressure over our windows runners, and the elevated cost of instantiating and bringing down those instances, we want to migrate instances from ephemeral to not ephemeral.
Possible impacts are related to breakages in or misbehaves on CI jobs that puts the runners in a bad state. Other possible impacts are related to exhaustion of resources, especially disk space, but memory might be a contender, as CI trash piles up on those instances.
As a somewhat middle of the road approach to this, currently nonephemeral instances are stochastically rotated as older instances get higher priority to be terminated when demand is lower.
Instances definition can be found here: https://github.com/pytorch/test-infra/pull/4072
This is a first in a multi-step approach where we will migrate away from all ephemeral windows instances and follow the lead of the `windows.g5.4xlarge.nvidia.gpu` in order to help reduce queue times for those instances. The phased approach follows:
* migrate `windows.4xlarge` to `windows.4xlarge.nonephemeral` instances under `pytorch/pytorch`
* migrate `windows.8xlarge.nvidia.gpu` to `windows.8xlarge.nvidia.gpu.nonephemeral` instances under `pytorch/pytorch`
* submit PRs to all repositories under `pytorch/` organization to migrate `windows.4xlarge` to `windows.4xlarge.nonephemeral`
* submit PRs to all repositories under `pytorch/` organization to migrate `windows.8xlarge.nvidia.gpu` to `windows.8xlarge.nvidia.gpu.nonephemeral`
* terminate the existence of `windows.4xlarge` and `windows.8xlarge.nvidia.gpu`
* evaluate and start the work related to the adoption of `windows.g5.4xlarge.nvidia.gpu` to replace `windows.8xlarge.nvidia.gpu.nonephemeral` in other repositories and use cases (proposed by @huydhn)
The reasoning for this phased approach is to reduce the scope of possible contenders to investigate in case of misbehave of particular CI jobs.
# Copilot Summary
<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 579d87a</samp>
This pull request migrates some windows workflows to use `nonephemeral` runners for better performance and reliability. It also adds support for new Python and CUDA versions for some binary builds. It affects the following files: `.github/templates/windows_binary_build_workflow.yml.j2`, `.github/workflows/generated-windows-binary-*.yml`, `.github/workflows/pull.yml`, `.github/actionlint.yaml`, `.github/workflows/_win-build.yml`, `.github/workflows/periodic.yml`, and `.github/workflows/trunk.yml`.
# Copilot Poem
<!--
copilot:poem
-->
### <samp>🤖 Generated by Copilot at 579d87a</samp>
> _We're breaking free from the ephemeral chains_
> _We're running on the nonephemeral lanes_
> _We're building faster, testing stronger, supporting newer_
> _We're the non-ephemeral runners of fire_
Pull Request resolved: https://github.com/pytorch/pytorch/pull/100377
Approved by: https://github.com/huydhn, https://github.com/malfet, https://github.com/atalman
(cherry picked from commit 7caac545b1)
Fixes #ISSUE_NUMBER
Pull Request resolved: https://github.com/pytorch/pytorch/pull/100548
Approved by: https://github.com/jeanschmidt, https://github.com/janeyx99
4006 lines
173 KiB
YAML
Generated
4006 lines
173 KiB
YAML
Generated
# @generated DO NOT EDIT MANUALLY
|
|
|
|
# Template is at: .github/templates/windows_binary_build_workflow.yml.j2
|
|
# Generation script: .github/scripts/generate_ci_workflows.py
|
|
name: windows-binary-libtorch-debug
|
|
|
|
on:
|
|
push:
|
|
# NOTE: Meta Employees can trigger new nightlies using: https://fburl.com/trigger_pytorch_nightly_build
|
|
branches:
|
|
- nightly
|
|
tags:
|
|
# NOTE: Binary build pipelines should only get triggered on release candidate builds
|
|
# Release candidate tags look like: v1.11.0-rc1
|
|
- v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+
|
|
- 'ciflow/binaries/*'
|
|
- 'ciflow/binaries_libtorch/*'
|
|
workflow_dispatch:
|
|
|
|
env:
|
|
# Needed for conda builds
|
|
ALPINE_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine"
|
|
ANACONDA_USER: pytorch
|
|
AWS_DEFAULT_REGION: us-east-1
|
|
BUILD_ENVIRONMENT: windows-binary-libtorch-debug
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
SHA1: ${{ github.event.pull_request.head.sha || github.sha }}
|
|
SKIP_ALL_TESTS: 1
|
|
concurrency:
|
|
group: windows-binary-libtorch-debug-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
libtorch-cpu-shared-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cpu-shared-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-shared-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-shared-with-deps-debug-build
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cpu-shared-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-shared-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-shared-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cpu-shared-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cpu-shared-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cpu-shared-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-shared-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-shared-without-deps-debug-build
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cpu-shared-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-shared-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-shared-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cpu-shared-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cpu-static-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cpu-static-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-static-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-static-with-deps-debug-build
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cpu-static-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-static-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-static-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cpu-static-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cpu-static-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cpu-static-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-static-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-static-without-deps-debug-build
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cpu-static-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cpu-static-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cpu-static-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cpu
|
|
GPU_ARCH_TYPE: cpu
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cpu-static-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_7-shared-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_7-shared-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-shared-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-shared-with-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_7-shared-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-shared-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-shared-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_7-shared-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_7-shared-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_7-shared-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-shared-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-shared-without-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_7-shared-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-shared-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-shared-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_7-shared-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_7-static-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_7-static-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-static-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-static-with-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_7-static-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-static-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-static-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_7-static-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_7-static-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_7-static-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-static-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-static-without-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_7-static-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_7-static-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_7-static-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu117
|
|
GPU_ARCH_VERSION: 11.7
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_7-static-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_8-shared-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_8-shared-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-shared-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-shared-with-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_8-shared-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-shared-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-shared-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_8-shared-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_8-shared-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_8-shared-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-shared-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-shared-without-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_8-shared-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-shared-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-shared-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_8-shared-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_8-static-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_8-static-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-static-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-static-with-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_8-static-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-static-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-static-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_8-static-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda11_8-static-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda11_8-static-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-static-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-static-without-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda11_8-static-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda11_8-static-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda11_8-static-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu118
|
|
GPU_ARCH_VERSION: 11.8
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda11_8-static-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda12_1-shared-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda12_1-shared-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-shared-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-shared-with-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda12_1-shared-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-shared-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-shared-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda12_1-shared-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda12_1-shared-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda12_1-shared-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-shared-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-shared-without-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda12_1-shared-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-shared-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-shared-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: shared-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda12_1-shared-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda12_1-static-with-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda12_1-static-with-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-static-with-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-static-with-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda12_1-static-with-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-static-with-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-static-with-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-with-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda12_1-static-with-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|
|
libtorch-cuda12_1-static-without-deps-debug-build:
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
runs-on: windows.4xlarge.nonephemeral
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Build PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_build.sh"
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: libtorch-cuda12_1-static-without-deps-debug
|
|
retention-days: 14
|
|
if-no-files-found: error
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-static-without-deps-debug-test: # Testing
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-static-without-deps-debug-build
|
|
runs-on: windows.8xlarge.nvidia.gpu
|
|
timeout-minutes: 240
|
|
env:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
SKIP_ALL_TESTS: 1
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
steps:
|
|
- name: Display EC2 information
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
function get_ec2_metadata() {
|
|
# Pulled from instance metadata endpoint for EC2
|
|
# see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
|
|
category=$1
|
|
curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
|
|
}
|
|
echo "ami-id: $(get_ec2_metadata ami-id)"
|
|
echo "instance-id: $(get_ec2_metadata instance-id)"
|
|
echo "instance-type: $(get_ec2_metadata instance-type)"
|
|
echo "system info $(uname -a)"
|
|
- name: "[FB EMPLOYEES] Enable SSH (Click me for login details)"
|
|
uses: seemethere/add-github-ssh-key@v1
|
|
with:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Needed for binary builds, see: https://github.com/pytorch/pytorch/issues/73339#issuecomment-1058981560
|
|
- name: Enable long paths on Windows
|
|
shell: powershell
|
|
run: |
|
|
Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
|
|
# Since it's just a defensive command, the workflow should continue even the command fails
|
|
- name: Disables Windows Defender scheduled and real-time scanning for files in directories used by PyTorch
|
|
shell: powershell
|
|
run: |
|
|
Add-MpPreference -ExclusionPath $(Get-Location).tostring(),$Env:TEMP -ErrorAction Ignore
|
|
# Let's both exclude the path and disable Windows Defender completely just to be sure
|
|
# that it doesn't interfere
|
|
Set-MpPreference -DisableRealtimeMonitoring $True
|
|
# NOTE: These environment variables are put here so that they can be applied on every job equally
|
|
# They are also here because setting them at a workflow level doesn't give us access to the
|
|
# runner.temp variable, which we need.
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
echo "BINARY_ENV_FILE=${RUNNER_TEMP}/env" >> "${GITHUB_ENV}"
|
|
echo "PYTORCH_FINAL_PACKAGE_DIR=${RUNNER_TEMP}/artifacts" >> "${GITHUB_ENV}"
|
|
echo "WIN_PACKAGE_WORK_DIR=${RUNNER_TEMP}"
|
|
- uses: actions/download-artifact@v3
|
|
name: Download Build Artifacts
|
|
with:
|
|
name: libtorch-cuda12_1-static-without-deps-debug
|
|
path: "${{ env.PYTORCH_FINAL_PACKAGE_DIR }}"
|
|
- name: Checkout PyTorch
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
submodules: recursive
|
|
path: pytorch
|
|
quiet-checkout: true
|
|
- name: Clean PyTorch checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: pytorch
|
|
- name: Checkout pytorch/builder
|
|
uses: malfet/checkout@silent-checkout
|
|
with:
|
|
ref: main
|
|
submodules: recursive
|
|
repository: pytorch/builder
|
|
path: builder
|
|
quiet-checkout: true
|
|
- name: Clean pytorch/builder checkout
|
|
run: |
|
|
# Remove any artifacts from the previous checkouts
|
|
git clean -fxd
|
|
working-directory: builder
|
|
- name: Populate binary env
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_populate_env.sh"
|
|
- name: Test PyTorch binary
|
|
shell: bash
|
|
run: |
|
|
"${PYTORCH_ROOT}/.circleci/scripts/binary_windows_test.sh"
|
|
- name: Wait until all sessions have drained
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
timeout-minutes: 120
|
|
run: |
|
|
.github\scripts\wait_for_ssh_to_drain.ps1
|
|
- name: Kill active ssh sessions if still around (Useful if workflow was cancelled)
|
|
shell: powershell
|
|
working-directory: pytorch
|
|
if: always()
|
|
run: |
|
|
.github\scripts\kill_active_ssh_sessions.ps1
|
|
libtorch-cuda12_1-static-without-deps-debug-upload: # Uploading
|
|
if: ${{ github.repository_owner == 'pytorch' }}
|
|
needs: libtorch-cuda12_1-static-without-deps-debug-test
|
|
with:
|
|
PYTORCH_ROOT: ${{ github.workspace }}/pytorch
|
|
BUILDER_ROOT: ${{ github.workspace }}/builder
|
|
PACKAGE_TYPE: libtorch
|
|
# TODO: This is a legacy variable that we eventually want to get rid of in
|
|
# favor of GPU_ARCH_VERSION
|
|
DESIRED_CUDA: cu121
|
|
GPU_ARCH_VERSION: 12.1
|
|
GPU_ARCH_TYPE: cuda
|
|
LIBTORCH_CONFIG: debug
|
|
LIBTORCH_VARIANT: static-without-deps
|
|
# This is a dummy value for libtorch to work correctly with our batch scripts
|
|
# without this value pip does not get installed for some reason
|
|
DESIRED_PYTHON: "3.8"
|
|
build_name: libtorch-cuda12_1-static-without-deps-debug
|
|
secrets:
|
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
aws-access-key-id: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
|
|
aws-pytorch-uploader-secret-access-key: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
|
|
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
|
|
uses: ./.github/workflows/_binary-upload.yml
|