derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
69 lines
2.5 KiB
YAML
69 lines
2.5 KiB
YAML
# Smoke-test recipe for the `race` workload (`mode: fsdp`).
|
|
#
|
|
# `race` is a DISTRIBUTED workload (min_world_size=2), so it MUST be launched
|
|
# under torchrun -- a bare `aorta triage run` starts a single process
|
|
# (WORLD_SIZE=1) and is correctly refused by launch-mode validation. Use the
|
|
# `aorta` console script as torchrun's target (`-m aorta` is NOT a runnable
|
|
# module):
|
|
#
|
|
# # validate only (no GPUs / no launcher needed):
|
|
# aorta triage run --recipe recipes/example-fsdp-smoke.yaml --dry-run
|
|
#
|
|
# # single node, 2 ranks (CX-7 / RCCL smoke; bump to your GPU count):
|
|
# torchrun --standalone --nproc_per_node=2 $(which aorta) triage run -recipe recipes/example-fsdp-smoke.yaml
|
|
#
|
|
# # multi-node (e.g. 1 rank/host x 40 hosts -- the AINIC topology):
|
|
# torchrun --nnodes=40 --nproc_per_node=1 --rdzv-backend=c10d \
|
|
# --rdzv-endpoint=$MIRAGE_HEAD_ADDR:$MIRAGE_HEAD_PORT $(which aorta) triage run \
|
|
# --recipe recipes/example-fsdp-smoke.yaml
|
|
#
|
|
# Results are written by rank 0 only (dispatcher gates on RANK), so multi-rank
|
|
# launches don't clobber each other.
|
|
#
|
|
# Runs the simulated-collective FSDP harness in `aorta.race.modes.fsdp`. This
|
|
# is a PUBLIC, non-NDA smoke: it exercises the adapter -> create_reproducer ->
|
|
# real RCCL/NCCL collectives path end-to-end on any GPU box. It is NOT expected
|
|
# to reproduce the AINIC SDC bug on non-AINIC hardware (e.g. CX-7 / RCCL) -- a
|
|
# clean `passed: True` is the expected smoke result; the point is that the
|
|
# workload runs.
|
|
#
|
|
# Iterations are capped small so a smoke finishes in seconds -- the
|
|
# ReproducerConfig defaults (verify_iterations=10000, simulate_compute=True)
|
|
# would otherwise run for many minutes.
|
|
#
|
|
# Workload names resolve at cell-EXECUTION time, not load time: this recipe
|
|
# validates even before `pip install -e .` registers `race`; an unregistered
|
|
# name surfaces as a per-cell error at run time, not a load error.
|
|
schema_version: 1
|
|
|
|
ticket: EXAMPLE-151
|
|
workload: race
|
|
|
|
trials: 2
|
|
steps: 100
|
|
|
|
workload_config:
|
|
mode: fsdp
|
|
warmup_iterations: 2
|
|
verify_iterations: 5
|
|
simulate_compute: false # skip GEMM work so the smoke is fast
|
|
fsdp_shard_size: 100000
|
|
|
|
confound:
|
|
threshold: 1.15
|
|
# Left unset so the baseline auto-resolves to `baseline-local` via the
|
|
# "first cell starting with baseline-" rule.
|
|
|
|
cells:
|
|
- name: baseline-local
|
|
mitigations: [none]
|
|
environment: local
|
|
|
|
- name: tf32_off-local
|
|
mitigations: [tf32_off]
|
|
environment: local
|
|
|
|
- name: xnack-local
|
|
mitigations: [xnack]
|
|
environment: local
|
|
|