derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
79 lines
2.9 KiB
Markdown
79 lines
2.9 KiB
Markdown
# RCCL
|
|
|
|
## Overview
|
|
|
|
This example builds the rccl-tests performance test suite, which benchmarks RCCL (ROCm Communication Collectives Library) operations across multiple GPUs. It includes performance tests for the major collectives - AllGather, AllReduce, AllToAll, AllToAllV, Broadcast, Reduce, and ReduceScatter - measuring bandwidth and latency for inter-GPU communication. This is useful for profiling GPU-to-GPU collective communication patterns and PCIe/Infinity Fabric interconnect performance.
|
|
|
|
## Source Files
|
|
|
|
- `rccl-tests/` - Vendored copy of the RCCL performance test suite, with source under `src/` and verification utilities under `verifiable/`.
|
|
|
|
## Prerequisites
|
|
|
|
- CMake 3.25+
|
|
- HIP runtime and `hipcc` compiler
|
|
- RCCL library
|
|
- Multiple AMD GPUs (tests target supported GPU architectures)
|
|
|
|
## Building
|
|
|
|
**Standalone build:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/rccl -DCMAKE_PREFIX_PATH=/opt/rocm
|
|
cmake --build <build_dir>
|
|
```
|
|
|
|
**As part of the examples suite:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/ -DCMAKE_PREFIX_PATH=/opt/rocm
|
|
cmake --build <build_dir> --target all_reduce_perf
|
|
```
|
|
|
|
**Targets:**
|
|
|
|
| Target | Description |
|
|
| -------- | ------------- |
|
|
| `all_gather_perf` | AllGather performance test (`ncclAllGather`) |
|
|
| `all_reduce_perf` | AllReduce performance test (`ncclAllReduce`) |
|
|
| `alltoall_perf` | AllToAll performance test (`ncclAllToAll`) |
|
|
| `alltoallv_perf` | AllToAllV performance test (`ncclAllToAllv`, `ncclSend`/`ncclRecv`) |
|
|
| `broadcast_perf` | Broadcast performance test (`ncclBroadcast`, `ncclBcast`) |
|
|
| `reduce_perf` | Reduce performance test (`ncclReduce`) |
|
|
| `reduce_scatter_perf` | ReduceScatter performance test (`ncclReduceScatter`) |
|
|
|
|
## Running
|
|
|
|
```bash
|
|
# Run AllReduce across all GPUs
|
|
./all_reduce_perf -b 8 -e 128M -f 2 -g <num_gpus>
|
|
|
|
# Run AllToAll
|
|
./alltoall_perf -b 1K -e 64M -g <num_gpus>
|
|
```
|
|
|
|
## Profiling with rocprofiler-systems
|
|
|
|
```bash
|
|
rocprof-sys-run -- ./all_reduce_perf -b 8 -e 128M -f 2 -g 2
|
|
```
|
|
|
|
### Recommended Configuration
|
|
|
|
| Variable | Value | Purpose |
|
|
| ---------- | ------- | --------- |
|
|
| `ROCPROFSYS_ROCM_DOMAINS` | `rccl_api, hip_runtime_api,kernel_dispatch,memory_copy` | RCCL collective API calls, Trace HIP API and GPU operations |
|
|
| `ROCPROFSYS_USE_RCCLP` | `true` | Alternative way to enable RCCL API tracing (implicitly adds the `rccl_api` domain) |
|
|
| `ROCPROFSYS_TRACE` | `true` | Generate Perfetto trace for timeline analysis |
|
|
| `ROCPROFSYS_PROFILE` | `true` | Generate call-stack profile |
|
|
|
|
> **Note:** RCCL collective calls (`ncclAllReduce`, `ncclBroadcast`, ...) are only captured when the `rccl_api` domain is enabled. Add `rccl_api` to `ROCPROFSYS_ROCM_DOMAINS`, or set `ROCPROFSYS_USE_RCCLP=true` (which enables the `rccl_api` domain for you).
|
|
|
|
```bash
|
|
rocprof-sys-run \
|
|
-e ROCPROFSYS_ROCM_DOMAINS=rccl_api,hip_runtime_api,kernel_dispatch,memory_copy \
|
|
-e ROCPROFSYS_TRACE=true \
|
|
-- ./all_reduce_perf -b 8 -e 128M -f 2 -g 2
|
|
```
|