TheRock/build_tools/third_party/s3_management

s3_management

Overview

These scripts are forked from https://github.com/pytorch/test-infra/tree/main/s3_management.

• update_dependencies.py mirrors dependency packages from PyPI to our S3 buckets
• manage.py generates PEP 503-compliant index.html files for each subdirectory in our S3 buckets, including packages we build (e.g. rocm, torch) and dependencies (uploaded by update_dependencies.py)

The Python package buckets we maintain are:

S3 bucket name	S3 URL	User-facing URLs
therock-dev-python	https://therock-dev-python.s3.amazonaws.com/	https://rocm.devreleases.amd.com/v2/ https://rocm.devreleases.amd.com/v2-staging/
therock-nightly-python	https://therock-nightly-python.s3.amazonaws.com/	https://rocm.nightlies.amd.com/v2/ https://rocm.nightlies.amd.com/v2-staging/

Each bucket has v2 and v2-staging top level folders at the moment. This may evolve with v3 in the future. Within each folder there are subfolders for each index we publish, currently corresponding to each GPU family that we build releases for. See these other pages for more details:

• Index page listing in RELEASES.md
• Gating releases with Pytorch tests in external-builds/pytorch/README.md

The user-facing URLs can be used with pip install --index-url. For example:

python -m pip install \
  --index-url https://rocm.nightlies.amd.com/v2/gfx94X-dcgpu/ \
  "rocm[libraries,devel]"

Running tests

update_dependencies_test.py contains unit tests for the wheel filtering logic in update_dependencies.py. No AWS credentials or network access are required.

cd build_tools/third_party/s3_management
python -m pytest update_dependencies_test.py -v

Playbook for running the scripts

While these scripts do run as part of some automated workflows with automatically assumed roles that grant CI/CD machines access, these scripts do still need to be run manually by developers under certain conditions.

Note

See also these pytorch/test-infra issues, which we could collaborate on since these scripts are forked:

• test-infra/issues/6097 - Automate upload/sync of dependencies to pytorch s3 index
• test-infra/issues/6105 - Automate creation and population of pytorch index subfolder in s3

Obtaining credentials

You will need credentials to be able to upload to our S3 buckets.

The manual way to do this is to create a user account in our IAM and grant it these policies:

• therock-dev-releases-access
• therock-nightly-releases-access

Warning

Be careful with the "nightly" access, since that is effectively "prod". Always test changes in "dev" first and consider revoking nightly access after you are done using it.

We plan on setting up a separate repository with isolated access for more user-facing releases in place of these developer nightly releases, but the general principle will still apply.

Adding a new package dependency

Let's say pytorch adds a new dependency and we want to include that dependency on our python package index pages. Here's how we would do that:

First, edit update_dependencies.py with the new package name, version, and project mapping.

Note which --package {jax,torch,rocm} your changes affect. Then, roll out those changes to the dev bucket:

1. Sanity test your changes with a dry run:

   export S3_BUCKET_PY=therock-dev-python
   python ./build_tools/third_party/s3_management/update_dependencies.py \
     --package rocm \
     --auto-detect-prefixes \
     --base-prefix v2/ \
     --dry-run
   

2. Update dependencies in the dev bucket:

   export S3_BUCKET_PY=therock-dev-python
   python ./build_tools/third_party/s3_management/update_dependencies.py \
     --package rocm \
     --auto-detect-prefixes \
     --base-prefix v2/
   

3. Regenerate the index pages for the dev bucket:

   export S3_BUCKET_PY=therock-dev-python
   python ./build_tools/third_party/s3_management/manage.py all
   

4. Visit the URL to check that the index pages look as expected and include the new dep. For example: https://rocm.devreleases.amd.com/v2/gfx120X-all/.

Finally, repeat those steps for the nightly bucket:

1. Update dependencies in the nightly bucket:

   export S3_BUCKET_PY=therock-nightly-python
   python ./build_tools/third_party/s3_management/update_dependencies.py \
     --package rocm \
     --auto-detect-prefixes \
     --base-prefix v2/
   

2. Regenerate the index pages for the nightly bucket:

   export S3_BUCKET_PY=therock-nightly-python
   python ./build_tools/third_party/s3_management/manage.py all
   

3. Visit the URL to check that the index pages look as expected and include the new dep. For example: https://rocm.nightlies.amd.com/v2/gfx120X-all/.

4. If no longer needed, revoke your access to therock-nightly-releases-access.

If something goes wrong

Bucket Versioning is enabled in S3 for the therock-nightly-python bucket. Changes can be rolled back to that bucket using that feature.

Bucket Versioning is NOT (yet?) enabled in S3 for the therock-dev-python bucket.