CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
62 lines
2.2 KiB
Python
62 lines
2.2 KiB
Python
"""Backend-agnostic storage location.
|
|
|
|
A ``StorageLocation`` represents a file or directory in S3 (or a local staging
|
|
directory) without coupling to any particular layout or upload/download
|
|
direction. It is the bridge between path computation modules (like
|
|
``workflow_outputs.WorkflowOutputRoot``) and I/O modules (``storage_backend``,
|
|
``artifact_backend``).
|
|
|
|
Usage::
|
|
|
|
from _therock_utils.storage_location import StorageLocation
|
|
|
|
loc = StorageLocation("my-bucket", "some/path/file.tar.xz")
|
|
loc.s3_uri # "s3://my-bucket/some/path/file.tar.xz"
|
|
loc.https_url # "https://my-bucket.s3.amazonaws.com/some/path/file.tar.xz"
|
|
loc.local_path(Path("/tmp/staging")) # Path("/tmp/staging/some/path/file.tar.xz")
|
|
"""
|
|
|
|
from dataclasses import dataclass
|
|
from pathlib import Path
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class StorageLocation:
|
|
"""A location that can be resolved to S3 URI, HTTPS URL, or local path.
|
|
|
|
Represents a single file or directory in a backend-agnostic way.
|
|
Use the properties/methods to get the representation you need:
|
|
|
|
- ``.s3_uri`` - For AWS CLI uploads (``s3://bucket/path/file.tar.xz``)
|
|
- ``.https_url`` - For public links (``https://bucket.s3.amazonaws.com/...``)
|
|
- ``.local_path(staging_dir)`` - For local testing (``Path("/tmp/staging/...")``)
|
|
- ``.relative_path`` - Backend-agnostic relative path from the bucket/staging root
|
|
"""
|
|
|
|
bucket: str
|
|
"""S3 bucket name (used for S3 URI and HTTPS URL construction)."""
|
|
|
|
relative_path: str
|
|
"""Relative path from bucket/staging root (e.g., '12345-linux/file.tar.xz')."""
|
|
|
|
@property
|
|
def s3_uri(self) -> str:
|
|
"""S3 URI (e.g., ``s3://bucket/path/file``)."""
|
|
return f"s3://{self.bucket}/{self.relative_path}"
|
|
|
|
@property
|
|
def https_url(self) -> str:
|
|
"""Public HTTPS URL for browser access."""
|
|
return f"https://{self.bucket}.s3.amazonaws.com/{self.relative_path}"
|
|
|
|
def local_path(self, staging_dir: Path) -> Path:
|
|
"""Local filesystem path for this location.
|
|
|
|
Args:
|
|
staging_dir: Base directory for local staging.
|
|
|
|
Returns:
|
|
Full path: ``{staging_dir}/{relative_path}``
|
|
"""
|
|
return staging_dir / self.relative_path
|