TheRock/external-builds/pytorch/README.md

Build PyTorch with ROCm support

This directory provides tooling for building PyTorch with ROCm Python wheels.

Tip

If you want to install our prebuilt PyTorch packages instead of building them from source, see RELEASES.md instead.

Table of contents:

• Support status
• Build instructions
• Running/testing PyTorch
• Advanced build instructions
• Development instructions

These build procedures are meant to run as part of ROCm CI and development flows and thus leave less room for interpretation than in upstream repositories. Some of this tooling is being migrated upstream as part of [RFC] Enable native Windows CI/CD on ROCm.

This incorporates advice from:

• https://github.com/pytorch/pytorch#from-source
• .ci/manywheel/build_rocm.sh and friends

Support status

Project and feature support status

The following projects and features are packaged and released through TheRock's stable, nightly, and dev release channels.

Project / feature	Linux support	Windows support
torch	✅ Supported	✅ Supported
↳ aotriton	✅ Supported	✅ Supported
↳ triton	✅ Supported	🚧 In progress - triton-windows#2
↳ FBGEMM GenAI	✅ Supported	❌ Not supported
torchaudio	✅ Supported	✅ Supported
torchvision	✅ Supported	✅ Supported
apex	✅ Supported	❌ Not supported

The following projects are not currently packaged and released by TheRock, functionality may vary.

Project	Linux support	Windows support	Notes
torch-scatter	✅ Community support	❌ Not supported	ROCm enablement in upstream
torch-sparse	✅ Community support	❌ Not supported	ROCm enablement in upstream
torchcodec	❌ Not supported	❌ Not supported	See Issue #1490

Supported PyTorch versions

We support building various PyTorch versions compatible with the latest ROCm sources and release packages.

Support for the latest upstream PyTorch code (i.e. main or nightly) uses the upstream projects directly, while extended support for older releases is maintained via backported release branches in https://github.com/ROCm/pytorch. Developers can also build variations of these versions to suite their own requirements.

Note

We build "nightly" versions with alpha suffixes. Once PyTorch promotes a new release version, we switch from the upstream branches to the backported release branches in https://github.com/ROCm/pytorch without such version suffixes (e.g. 2.7.0a0-> 2.7.1).

Historical builds are archived at https://rocm.nightlies.amd.com according to some retention policy.

Each PyTorch version uses a combination of:

• Git repository URLs for each project
• Git "repo hashtags" (branch names, tag names, or commit refs) for each project

See the following table for how each version is supported:

PyTorch version	Linux	Windows
2.13	✅ Using downstream ROCm/pytorch fork ROCm/pytorch release/2.13 branch ROCm/triton - ci_commit_pins/triton.txt pytorch/audio - "rocm related commit" pytorch/vision - "rocm related commit" ROCm/apex release/1.13.0 branch - "rocm related commit"	✅ Using downstream ROCm/pytorch fork ROCm/pytorch release/2.13 branch pytorch/audio - "rocm related commit" pytorch/vision - "rocm related commit"
2.13 alpha	✅ Using upstream pytorch pytorch/pytorch nightly branch ROCm/triton - ci_commit_pins/triton.txt pytorch/audio nightly branch pytorch/vision nightly branch ROCm/apex master branch	✅ Using upstream pytorch pytorch/pytorch nightly branch triton-lang/triton-windows - ci_commit_pins/triton-windows.txt pytorch/audio nightly branch pytorch/vision nightly branch
2.12	✅ Using downstream ROCm/pytorch fork ROCm/pytorch release/2.12 branch ROCm/triton - ci_commit_pins/triton.txt pytorch/audio - "rocm related commit" pytorch/vision - "rocm related commit" ROCm/apex release/1.12.0 branch - "rocm related commit"	✅ Using downstream ROCm/pytorch fork ROCm/pytorch release/2.12 branch pytorch/audio - "rocm related commit" pytorch/vision - "rocm related commit"
2.12 alpha	Previously built	Previously built
2.11	✅ Using downstream ROCm/pytorch fork ROCm/pytorch release/2.11 branch ROCm/triton - ci_commit_pins/triton.txt pytorch/audio - "rocm related commit" pytorch/vision - "rocm related commit" ROCm/apex release/1.11.0 branch - "rocm related commit"	✅ Using downstream ROCm/pytorch fork ROCm/pytorch release/2.11 branch pytorch/audio - "rocm related commit" pytorch/vision - "rocm related commit"
2.11 alpha	Previously built	Previously built
2.10	Previously built	Previously built
2.10 alpha	Previously built	Previously built
2.9	Previously built	Previously built
2.9 alpha	Previously built	Previously built
2.8	Previously built	Previously built
2.7	Previously built	Unsupported

See also:

• The Alternative branches and versions section for detailed information about configurations
• The upstream PyTorch release documentation
• Workflow source code:
  • .github/workflows/multi_arch_build_portable_linux_pytorch_wheels.yml
  • .github/workflows/multi_arch_build_windows_pytorch_wheels.yml

Build instructions

See the comments in build_prod_wheels.py for detailed instructions. That information is summarized here.

Prerequisites and setup

You will need a supported Python version (3.10+) on a system which we build the rocm[libraries,devel] packages for. See the RELEASES.md: Installing releases using pip and Python Packaging documentation for more background on these rocm packages.

Warning

On Windows, prefer to install Python for the current user only and to a path without spaces like C:\Users\<username>\AppData\Local\Programs\Python\Python312.

Several developers have reported issues building torchvision when using "Install Python for all users" with a default path like C:\Program Files\Python312 (note the space in "Program Files"). See https://github.com/pytorch/vision/issues/9165 for details.

Warning

On Windows, when building with "--enable-pytorch-flash-attention-windows", Make sure to use ninja 1.13.1 or above.

NOTE: If you use ccache and face "invalid argument" errors during the aotriton build, disable ccache and try again.

Quickstart

It is highly recommended to use a virtual environment unless working within a throw-away container or CI environment.

# On Linux
python -m venv .venv && source .venv/bin/activate

# On Windows
python -m venv .venv && .venv\Scripts\activate.bat

Now checkout repositories using their default branches:

• On Linux, use default paths (nested under this folder):

  python pytorch_torch_repo.py checkout
  python pytorch_audio_repo.py checkout
  python pytorch_vision_repo.py checkout
  python pytorch_apex_repo.py checkout
  

• On Windows, use shorter paths to avoid command length limits:

  python pytorch_torch_repo.py checkout --checkout-dir C:/b/pytorch
  python pytorch_audio_repo.py checkout --checkout-dir C:/b/audio
  python pytorch_vision_repo.py checkout --checkout-dir C:/b/vision
  

Now note the gfx target you want to build for and then...

1. Install rocm packages
2. Build PyTorch wheels
3. Install the built PyTorch wheels

...all in one command. See the advanced build instructions for ways to mix/match build steps.

• On Linux:

  python build_prod_wheels.py build \
    --install-rocm --index-url https://rocm.nightlies.amd.com/v2/gfx110X-all/ \
    --output-dir $HOME/tmp/pyout
  

• On Windows:

  python build_prod_wheels.py build ^
    --install-rocm --index-url https://rocm.nightlies.amd.com/v2/gfx110X-all/ ^
    --pytorch-dir C:/b/pytorch ^
    --pytorch-audio-dir C:/b/audio ^
    --pytorch-vision-dir C:/b/vision ^
    --output-dir %HOME%/tmp/pyout
  

Running/testing PyTorch

Prerequisites

On Linux we run automated tests under our no_rocm_image_ubuntu24_04.Dockerfile container (also documented in dockerfiles/README.md). Docker is optional for developers and users. If you want to use our test image, run it like so:

sudo docker run -it \
  --device=/dev/kfd --device=/dev/dri \
  --ipc=host --group-add=video --group-add=render --group-add=110 \
  ghcr.io/rocm/no_rocm_image_ubuntu24_04:latest

Running ROCm and PyTorch sanity checks

The simplest tests for a working PyTorch with ROCm install are:

rocm-sdk test
# Should show passing tests

python -c "import torch; print(torch.cuda.is_available())"
# Should print "True"

Running PyTorch smoketests

We have additional smoketests in smoke-tests that run some sample computations. To run these tests:

# Basic usage (no wrapper script)
pytest -v smoke-tests

# Wrapper script, passing through some useful pytest args:
python run_pytorch_smoke_tests.py -- \
  --log-cli-level=INFO \
  -v

Running full PyTorch tests

We have a run_pytorch_tests.py script which runs PyTorch unit tests using pytest with additional test exclusion capabilities tailored for AMD ROCm GPUs. See the script for detailed instructions. Here are a few examples:

# Install test dependencies
python -m pip install -r pytorch/.ci/docker/requirements-ci.txt

# Basic usage (auto-detect everything, no extra args):
python run_pytorch_tests.py

# Typical usage on CI, passing through some useful pytest args:
python run_pytorch_tests.py -- \
  --continue-on-collection-errors \
  --import-mode=importlib \
  -v

# Custom test selection with pytest -k:
python run_pytorch_tests.py -k "test_nn and not test_dropout"

# Explicit pytorch repo path (for test sources) and GPU family (for filtering)
python run_pytorch_tests.py --pytorch-dir=/tmp/pytorch --amdgpu-family=gfx950

# GPU selection happens in two stages:
#   1. --device-query  decides which GPUs enter the candidate set.
#   2. --gpu-policy    decides how many candidates are made visible to tests.
#
# All GPUs visible at once (for multi-GPU tests):
python run_pytorch_tests.py --device-query all --gpu-policy all

# All GPUs discovered, but only one visible at a time:
python run_pytorch_tests.py --device-query all --gpu-policy single

# One GPU per architecture discovered, all of them visible:
python run_pytorch_tests.py --device-query unique --gpu-policy all

# Multi-GPU run on a single architecture (e.g., use both gfx1201 GPUs on a
# machine where visible devices are {'gfx1100': [0], 'gfx1201': [1, 2]}):
python run_pytorch_tests.py --amdgpu-family=gfx1201 --device-query all --gpu-policy all

Tests can also be run by following the ROCm documentation at https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/3rd-party/pytorch-install.html#testing-the-pytorch-installation. For example:

PYTORCH_TEST_WITH_ROCM=1 python pytorch/test/run_test.py --include test_torch

Nightly releases

Gating releases with Pytorch tests

Per-family releases (non-multi-arch): Passing builds upload wheels to the "v2-staging" directory at https://rocm.nightlies.amd.com/v2-staging/. Only after passing torch tests are wheels promoted to "v2" at https://rocm.nightlies.amd.com/v2/. If no runner is available, promotion is blocked by default. Set bypass_tests_for_releases=true for exceptional cases under amdgpu_family_matrix.py.

Multi-arch releases: Wheels are published directly to https://rocm.nightlies.amd.com/whl-multi-arch/ without a staging step. Tests run post-publish as a signal (visible on https://therock-hud-dev.amd.com/), not as a gate. This avoids pip resolution issues that would arise if shared host torch wheel and per-target device wheels were promoted independently. See the discussion on #3332 for details.

Advanced build instructions

Other ways to install the rocm packages

The rocm[libraries,devel] packages can be installed in multiple ways:

• (As above) during the build_prod_wheels.py build subcommand

• Using the more tightly scoped build_prod_wheels.py install-rocm subcommand:

  build_prod_wheels.py
      --index-url https://rocm.nightlies.amd.com/v2/gfx110X-all/ \
      install-rocm
  

• Manually installing from a release index:

  # From therock-nightly-python
  python -m pip install \
    --index-url https://rocm.nightlies.amd.com/v2/gfx110X-all/ \
    rocm[libraries,devel]
  
  # OR from therock-dev-python
  python -m pip install \
    --index-url https://rocm.devreleases.amd.com/v2/gfx110X-all/ \
    rocm[libraries,devel]
  

• Building the rocm Python packages from artifacts fetched from a CI run:

  # From the repository root
  mkdir $HOME/.therock/17123441166
  mkdir $HOME/.therock/17123441166/artifacts
  python ./build_tools/fetch_artifacts.py \
    --run-id=17123441166 \
    --target=gfx110X-all \
    --output-dir=$HOME/.therock/17123441166/artifacts
  
  python ./build_tools/build_python_packages.py \
    --artifact-dir=$HOME/.therock/17123441166/artifacts \
    --dest-dir=$HOME/.therock/17123441166/packages
  

• Building the rocm Python packages from artifacts built from source:

  # From the repository root
  cmake --build build --target therock-artifacts
  
  python ./build_tools/build_python_packages.py \
    --artifact-dir=build/artifacts \
    --dest-dir=build/packages
  

Bundling PyTorch and ROCm together into a "fat wheel"

By default, Python wheels produced by the PyTorch build do not include ROCm binaries. Instead, they expect those binaries to come from the rocm[libraries,devel] packages. A "fat wheel" bundles the ROCm binaries into the same wheel archive to produce a standalone install including both PyTorch and ROCm, with all necessary patches to shared library / DLL loading for out of the box operation.

To produce such a fat wheel, see windows_patch_fat_wheel.py and a future equivalent script for Linux.

Development instructions

This section covers recommended practices for making changes to PyTorch and other repositories for use with the build scripts and integration with version control systems.

If you want to make changes to PyTorch source code, prefer in this order:

1. Contributing to upstream main branches
2. Contributing to upstream release/ branches while within the release window
3. Contributing to downstream release/ branches in forked repositories

Note

We used to support applying git patches as part of checkout out PyTorch repositories. This system has been removed as ROCm-specific changes are now maintained in the https://github.com/ROCm/pytorch/ fork rather than as patch files.

Checking out PyTorch repositories

For release-branch work, prefer the manifest-based flow in Checking out using manifests. It records the exact torch, audio, vision, triton, and apex commits in one JSON file, then checks out all requested repositories from that file.

Each pytorch_*_repo.py script handles checkout and preparation:

1. Clone the repository as needed from --gitrepo-origin to the --repo-name folder under path --repo
2. Checkout the --repo-hashtag git ref/tag
3. Tag the upstream commit as THEROCK_UPSTREAM_DIFFBASE
4. Run 'hipify' on the repository, editing source files and committing the changes
5. Tag the hipify commit as THEROCK_HIPIFY_DIFFBASE

After running one of the pytorch_*_repo.py scripts you should have a repository with history like this:

$ python pytorch_torch_repo.py checkout --repo-hashtag main
$ cd pytorch
$ git log --oneline

cb53ee6fd45 (HEAD, tag: THEROCK_HIPIFY_DIFFBASE) DO NOT SUBMIT: HIPIFY
96682103026 (tag: THEROCK_UPSTREAM_DIFFBASE, origin/main) Example upstream commit 2
3f5a8e2003f Example upstream commit 1

Note the sequence of commits and tags that were created:

• main is checked out initially and is tagged THEROCK_UPSTREAM_DIFFBASE
• hipify is run and its changes are tagged THEROCK_HIPIFY_DIFFBASE

For build workflows that need package metadata such as torch.version.git_version to identify the fetched upstream PyTorch commit, pass --no-commit-hipify. That still runs HIPIFY, but leaves the source tree dirty instead of creating a local DO NOT SUBMIT: HIPIFY commit.

For test-only source checkouts, pass --no-hipify --no-submodules to fetch the PyTorch test files without HIPIFY preprocessing or submodule checkout.

Alternative branches and versions

PyTorch main branches

This checks out the main branches from https://github.com/pytorch, tracking the latest (potentially unstable) code:

• https://github.com/pytorch/pytorch/tree/main
• https://github.com/pytorch/audio/tree/main
• https://github.com/pytorch/vision/tree/main
• https://github.com/ROCm/apex/tree/master

python pytorch_torch_repo.py checkout --repo-hashtag main
python pytorch_audio_repo.py checkout --repo-hashtag main
python pytorch_vision_repo.py checkout --repo-hashtag main
# Note that triton will be checked out at the PyTorch pin.
python pytorch_triton_repo.py checkout
python pytorch_apex_repo.py checkout --repo-hashtag master

PyTorch nightly branches

This checks out the nightly branches from https://github.com/pytorch, tracking the latest pytorch.org nightly release:

• https://github.com/pytorch/pytorch/tree/nightly
• https://github.com/pytorch/audio/tree/nightly
• https://github.com/pytorch/vision/tree/nightly
• https://github.com/ROCm/apex/tree/master

python pytorch_torch_repo.py checkout --repo-hashtag nightly
python pytorch_audio_repo.py checkout --repo-hashtag nightly
python pytorch_vision_repo.py checkout --repo-hashtag nightly
# Note that triton will be checked out at the PyTorch pin.
python pytorch_triton_repo.py checkout
python pytorch_apex_repo.py checkout --repo-hashtag master

ROCm PyTorch release branches

Because upstream PyTorch freezes at release but AMD needs to keep updating stable versions for a longer period of time, backport branches are maintained in the fork at https://github.com/ROCm/pytorch.

Tip

You are welcome to maintain your own branches that extend one of AMD's. Fork these scripts and change origins/tags/commits/versions as appropriate.

For most work, run these commands from the TheRock repository root to generate a manifest for the release branch and check out from that manifest:

python build_tools/github_actions/generate_pytorch_source_manifest.py \
  --rocm-version 7.13.0a20260501 \
  --output external-builds/pytorch/pytorch_manifest.json \
  --pytorch-git-refs release/2.12

python external-builds/pytorch/checkout_from_manifest.py \
  --manifest external-builds/pytorch/pytorch_manifest.json \
  --checkout-root ./external-builds/pytorch

The manifest generator creates files like pytorch_manifest_nightly.example.json which can then be read by the checkout script.

Tip

You can also use the lower-level checkout scripts that individually read the related_commit pins from the torch repository by running these commands from this external-builds/pytorch/ directory:

python pytorch_torch_repo.py checkout \
  --gitrepo-origin https://github.com/ROCm/pytorch.git \
  --repo-hashtag release/2.12
python pytorch_audio_repo.py checkout --require-related-commit
python pytorch_vision_repo.py checkout --require-related-commit
python pytorch_triton_repo.py checkout
python pytorch_apex_repo.py checkout --require-related-commit

Generating a manifest

# Single version (writes to an explicit path):
python build_tools/github_actions/generate_pytorch_source_manifest.py \
    --rocm-version 7.13.0a20260501 \
    --output external-builds/pytorch/pytorch_manifest.json \
    --pytorch-git-refs "release/2.11"

# Multiple versions (computed filenames in a directory):
python build_tools/github_actions/generate_pytorch_source_manifest.py \
    --rocm-version 7.13.0a20260501 \
    --manifest-dir external-builds/pytorch/manifests/ \
    --pytorch-git-refs "release/2.11 release/2.12 nightly"

# The target platform defaults to the current host. Pass it explicitly when
# generating a manifest for another platform:
python build_tools/github_actions/generate_pytorch_source_manifest.py \
    --rocm-version 7.13.0a20260501 \
    --platform windows \
    --output manifest.json \
    --pytorch-git-refs "release/2.11"

# Only pytorch (skip audio/vision/triton/apex):
python build_tools/github_actions/generate_pytorch_source_manifest.py \
    --rocm-version 7.13.0a20260501 \
    --output external-builds/pytorch/manifest.json \
    --pytorch-git-refs "release/2.11" \
    --projects pytorch

Checking out from a manifest

cd external-builds/pytorch

# Check out all projects at pinned commits:
python checkout_from_manifest.py \
    --manifest manifest.json \
    --checkout-root ./src

# Check out only pytorch:
python checkout_from_manifest.py \
    --manifest manifest.json \
    --checkout-root ./src \
    --projects pytorch

# Skip HIPIFY (e.g. for test-only runs):
python checkout_from_manifest.py \
    --manifest manifest.json \
    --checkout-root ./src \
    --no-hipify