derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
158 lines
13 KiB
Markdown
158 lines
13 KiB
Markdown
# S3 Buckets
|
|
|
|
TheRock uses Amazon S3 buckets to store CI build outputs (artifacts, logs,
|
|
python packages, etc.) and release artifacts. This page lists all buckets
|
|
and explains the authentication needed to upload to them.
|
|
|
|
## Table of contents
|
|
|
|
- [Authentication](#authentication)
|
|
- [Bucket inventory](#bucket-inventory)
|
|
- [CI buckets](#ci-buckets): `therock-ci-artifacts`, `therock-ci-artifacts-external`
|
|
- [Release buckets](#release-buckets): `therock-{dev,nightly,prerelease,release}-{artifacts,packages,python,tarball}`
|
|
- [Build system buckets](#build-system-buckets): `rocm-third-party-deps`
|
|
- [Cache buckets](#cache-buckets): `therock-pytorch-sccache-*`
|
|
- [Legacy buckets](#legacy-buckets): `therock-artifacts`, `therock-artifacts-external`
|
|
|
|
## Authentication
|
|
|
|
Most buckets have public _read_ access for use by developers as well as CI/CD
|
|
systems.
|
|
|
|
To _write_ to most buckets, assuming an IAM role via
|
|
[`aws-actions/configure-aws-credentials`](https://github.com/aws-actions/configure-aws-credentials)
|
|
using OIDC is needed. This requires `id-token: write` in the job's `permissions` block.
|
|
The full ARN pattern is
|
|
`arn:aws:iam::692859939525:role/therock-{ci,dev,nightly,prerelease}`.
|
|
|
|
Use the
|
|
[`configure_aws_artifacts_credentials`](/.github/actions/configure_aws_artifacts_credentials/action.yml)
|
|
composite action to set up credentials. It determines the correct IAM role and
|
|
bucket from the repository, event type, and optional `release_type` input
|
|
by using [`build_tools/_therock_utils/s3_buckets.py`](/build_tools/_therock_utils/s3_buckets.py):
|
|
|
|
```yaml
|
|
jobs:
|
|
build:
|
|
runs-on: azure-linux-scale-rocm
|
|
permissions:
|
|
id-token: write
|
|
# Linux containers only — mount runner baseline credentials
|
|
env:
|
|
AWS_SHARED_CREDENTIALS_FILE: /home/awsconfig/credentials.ini
|
|
|
|
steps:
|
|
# ... build steps ...
|
|
|
|
# Credentials are short-lived — assume the role close to when it's needed.
|
|
- name: Configure AWS Credentials
|
|
uses: ./.github/actions/configure_aws_artifacts_credentials
|
|
|
|
# ... upload steps that use the credentials ...
|
|
```
|
|
|
|
**Platform-specific details:**
|
|
|
|
- **Linux containers** mount runner credentials via
|
|
`AWS_SHARED_CREDENTIALS_FILE: /home/awsconfig/credentials.ini` in the job's
|
|
`env` block. These baseline credentials allow uploading to
|
|
`therock-ci-artifacts-external` without OIDC.
|
|
- **Windows** jobs must pass `special-characters-workaround: true` to
|
|
`aws-actions/configure-aws-credentials`. This retries credential fetching
|
|
until the secret access key contains no special characters, which some
|
|
Windows environments cannot tolerate. (The
|
|
`configure_aws_artifacts_credentials` composite action mentioned above
|
|
handles this automatically)
|
|
|
|
**External repos and forks:**
|
|
|
|
- **External repos** (e.g., `rocm-libraries`) use OIDC with the
|
|
`therock-ci-external` role to upload to `therock-ci-artifacts-external`.
|
|
- **Fork PRs** cannot use OIDC (no trust relationship). They fall back to
|
|
runner base credentials.
|
|
|
|
## Bucket inventory
|
|
|
|
### CI buckets
|
|
|
|
Our CI runners come with baseline credentials that allow uploading to
|
|
`therock-ci-artifacts-external` without any extra setup. Workflows in
|
|
downstream repos like `rocm-libraries`, `rocm-systems`, and `llvm-project`
|
|
upload to this bucket and do not need `aws-actions/configure-aws-credentials`.
|
|
|
|
| Bucket | Contents | IAM role |
|
|
| ------------------------------------------------------------------------------------------ | --------------------------------------- | ------------------------------------------------- |
|
|
| [`therock-ci-artifacts`](https://therock-ci-artifacts.s3.amazonaws.com/) | Build outputs for `ROCm/TheRock` | `therock-ci` |
|
|
| [`therock-ci-artifacts-external`](https://therock-ci-artifacts-external.s3.amazonaws.com/) | Build outputs for forks and other repos | `therock-ci-external`, or runner base credentials |
|
|
|
|
### Release buckets
|
|
|
|
Each release type (`dev`, `nightly`, `prerelease`, `release`) has a matching
|
|
set of buckets.
|
|
|
|
The `dev`, `nightly`, and `prerelease` types are accessed via
|
|
the `therock-{release_type}` IAM role while stable `release` buckets are
|
|
manually promoted from prereleases via IAM user policies (see
|
|
[`how_to_do_release.md`](/build_tools/packaging/how_to_do_release.md)).
|
|
|
|
Python, tarball, and native package buckets are fronted by CloudFront CDNs.
|
|
Developer-facing documentation and manual installs should use the CDN URLs;
|
|
CI may read the backing S3 buckets directly to avoid CloudFront data-transfer
|
|
charges.
|
|
|
|
| Bucket | Contents | IAM role | CDN |
|
|
| ---------------------------------------------------------------------------------------- | --------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| [`therock-dev-artifacts`](https://therock-dev-artifacts.s3.amazonaws.com/) | Build outputs | `therock-dev` | — |
|
|
| [`therock-dev-packages`](https://therock-dev-packages.s3.amazonaws.com/) | Native packages | `therock-dev` | [`rocm.devreleases.amd.com/packages-multi-arch/deb/`](https://rocm.devreleases.amd.com/packages-multi-arch/deb/), [`…/rpm/`](https://rocm.devreleases.amd.com/packages-multi-arch/rpm/) |
|
|
| [`therock-dev-python`](https://therock-dev-python.s3.amazonaws.com/) | Python packages | `therock-dev` | [`rocm.devreleases.amd.com/whl-multi-arch/`](https://rocm.devreleases.amd.com/whl-multi-arch/) |
|
|
| [`therock-dev-tarball`](https://therock-dev-tarball.s3.amazonaws.com/) | ROCm tarballs | `therock-dev` | [`rocm.devreleases.amd.com/tarball-multi-arch/`](https://rocm.devreleases.amd.com/tarball-multi-arch/) |
|
|
| [`therock-nightly-artifacts`](https://therock-nightly-artifacts.s3.amazonaws.com/) | Build outputs | `therock-nightly` | — |
|
|
| [`therock-nightly-packages`](https://therock-nightly-packages.s3.amazonaws.com/) | Native packages | `therock-nightly` | [`rocm.nightlies.amd.com/packages-multi-arch/deb/`](https://rocm.nightlies.amd.com/packages-multi-arch/deb/), [`…/rpm/`](https://rocm.nightlies.amd.com/packages-multi-arch/rpm/) |
|
|
| [`therock-nightly-python`](https://therock-nightly-python.s3.amazonaws.com/) | Python packages | `therock-nightly` | [`rocm.nightlies.amd.com/whl-multi-arch/`](https://rocm.nightlies.amd.com/whl-multi-arch/) |
|
|
| [`therock-nightly-tarball`](https://therock-nightly-tarball.s3.amazonaws.com/) | ROCm tarballs | `therock-nightly` | [`rocm.nightlies.amd.com/tarball-multi-arch/`](https://rocm.nightlies.amd.com/tarball-multi-arch/) |
|
|
| [`therock-prerelease-artifacts`](https://therock-prerelease-artifacts.s3.amazonaws.com/) | Build outputs | `therock-prerelease` | — |
|
|
| `therock-prerelease-packages` | Native packages | `therock-prerelease` | [`rocm.prereleases.amd.com/packages-multi-arch/`](https://rocm.prereleases.amd.com/packages-multi-arch/) |
|
|
| `therock-prerelease-python` | Python packages | `therock-prerelease` | [`rocm.prereleases.amd.com/whl-multi-arch/`](https://rocm.prereleases.amd.com/whl-multi-arch/) |
|
|
| `therock-prerelease-tarball` | ROCm tarballs | `therock-prerelease` | [`rocm.prereleases.amd.com/tarball-multi-arch/`](https://rocm.prereleases.amd.com/tarball-multi-arch/) |
|
|
| [`therock-release-artifacts`](https://therock-release-artifacts.s3.amazonaws.com/) | Build outputs | — | — |
|
|
| `therock-release-packages` | Native packages | — | [`repo.amd.com/rocm/packages-multi-arch/`](https://repo.amd.com/rocm/packages-multi-arch/) |
|
|
| `therock-release-python` | Python packages | — | [`repo.amd.com/rocm/whl-multi-arch/`](https://repo.amd.com/rocm/whl-multi-arch/) |
|
|
| `therock-release-tarball` | ROCm tarballs | — | [`repo.amd.com/rocm/tarball-multi-arch/`](https://repo.amd.com/rocm/tarball-multi-arch/) |
|
|
|
|
### Build system buckets
|
|
|
|
We mirror third-party dependency files into S3 for use by the build system.
|
|
|
|
| Bucket | Contents | Details |
|
|
| ----------------------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|
|
| `rocm-third-party-deps` | Mirrors for [`third_party/`](/third-party/) subprojects | See ["Updating a third-party mirror"](./git_chores.md#updating-a-third-party-mirror) in [`git_chores.md`](./git_chores.md) |
|
|
|
|
### Cache buckets
|
|
|
|
Unlike the other buckets on this page, the PyTorch sccache buckets live in a
|
|
separate AWS account (`324352301041`) in region `us-east-1`. The IAM role names
|
|
are identical to the artifact-pipeline roles but resolve to a different account
|
|
ID; the corresponding role ARNs are
|
|
`arn:aws:iam::324352301041:role/therock-{ci,dev,nightly,prerelease}`. OIDC trust
|
|
for `therock-{dev,nightly}` also covers `repo:ROCm/rockrel:*` so reusable
|
|
workflow invocations from `rockrel` can assume the role.
|
|
|
|
| Bucket | Contents | IAM role |
|
|
| ------------------------------------ | -------------------------- | -------------------- |
|
|
| `therock-pytorch-sccache-ci` | PyTorch CI sccache | `therock-ci` |
|
|
| `therock-pytorch-sccache-dev` | PyTorch dev sccache | `therock-dev` |
|
|
| `therock-pytorch-sccache-nightly` | PyTorch nightly sccache | `therock-nightly` |
|
|
| `therock-pytorch-sccache-prerelease` | PyTorch prerelease sccache | `therock-prerelease` |
|
|
|
|
### Legacy buckets
|
|
|
|
CI runs before 2025-11-11 (see [TheRock#2046](https://github.com/ROCm/TheRock/issues/2046))
|
|
used different bucket names. These are no longer written to but still contain
|
|
historical data. We may remove these once we implement a retention policy for
|
|
artifacts.
|
|
|
|
| Legacy bucket | Replaced by | IAM role |
|
|
| ------------------------------------------------------------------------------------ | ------------------------------- | ---------------------------- |
|
|
| [`therock-artifacts`](https://therock-artifacts.s3.amazonaws.com/) | `therock-ci-artifacts` | `therock-artifacts` |
|
|
| [`therock-artifacts-external`](https://therock-artifacts-external.s3.amazonaws.com/) | `therock-ci-artifacts-external` | `therock-artifacts-external` |
|