derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
146 lines
6.5 KiB
Markdown
146 lines
6.5 KiB
Markdown
# RCCL Topology Explorer (topo_expl)
|
|
|
|
The RCCL Topology Explorer is a tool for analyzing and exploring network topologies for RCCL (ROCm Communication Collectives Library) collective operations. It simulates various hardware configurations and displays the actual algo/proto combo selections that RCCL would make.
|
|
|
|
## Building
|
|
|
|
### Prerequisites
|
|
- ROCm/HIP development environment
|
|
- RCCL source code
|
|
- hipify-perl (for source transformation)
|
|
|
|
### Build Instructions
|
|
|
|
```bash
|
|
cd tools/topo_expl
|
|
make
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./topo_expl [-m|--model model_id] [-n|--nodes numNodes=1] [-a|--arch gpu_arch] [-t] [-h]
|
|
```
|
|
|
|
### Parameters
|
|
|
|
- `-m, --model id`: Specifies the topology model to use (required to run a model)
|
|
- `-n, --nodes N`: Number of nodes to simulate (default: 1)
|
|
- `-a, --arch arch`: Show available models by GPU architecture (e.g., gfx906, gfx908, gfx910, gfx942, gfx950)
|
|
- `-t, --test`: Run comprehensive test suite (all models with 1,2,4,8,16 nodes by default)
|
|
- `-h, --help`: Display help message and available models
|
|
|
|
### Test Suite Options
|
|
|
|
When using `-t` or `--test`, you can further customize the test run:
|
|
|
|
- `--include-models=ids`: Comma-separated model IDs to test (e.g., `--include-models=0,1,2,4,8`)
|
|
- `--include-nodes=counts`: Comma-separated node counts to test (e.g., `--include-nodes=1,4,8`)
|
|
- `--exclude-models=ids`: Comma-separated model IDs to exclude from test (e.g., `--exclude-models=57,59`)
|
|
- `--exclude-nodes=counts`: Comma-separated node counts to exclude from test (e.g., `--exclude-nodes=16`)
|
|
|
|
### Available Models
|
|
|
|
Run `./topo_expl` without arguments to see the list of available models. Each model represents a different hardware configuration. [Each model file](./models) pertains to a particular GPU model and node configuration. It can be output by RCCL through setting the environment variable `NCCL_TOPO_DUMP_FILE`. Model XMLs have been generated for simplicity.
|
|
|
|
## Example Usage: Print RCCL's algorithm/protocol selections
|
|
|
|
The tool is typically run with the `NCCL_DEBUG=INFO` environment variable to show the topology information and print out the constructed rings/trees. However, for the convenience of just printing the algo/proto table, we use version `NCCL_DEBUG=version` in this example to avoid printing topo details.
|
|
|
|
```bash
|
|
# List available models
|
|
./topo_expl
|
|
|
|
# List models for a specific GPU architecture
|
|
./topo_expl -a gfx942
|
|
# Or: ./topo_expl --arch gfx942
|
|
|
|
# Display help message
|
|
./topo_expl -h
|
|
|
|
# Test MI300 configuration (model 55)
|
|
NCCL_DEBUG=version ./topo_expl -m 55
|
|
|
|
# Test a multi-node MI300 configuration with 8 nodes
|
|
NCCL_DEBUG=version ./topo_expl -m 55 -n 8
|
|
# Or: ./topo_expl --model 55 --nodes 8
|
|
|
|
# Test a multi-node MI350 configuration with 2 nodes
|
|
NCCL_DEBUG=version ./topo_expl -m 59 -n 2
|
|
|
|
# Test MI250 configuration (model 42)
|
|
NCCL_DEBUG=version ./topo_expl -m 42
|
|
|
|
# Test a multi-node MI250 configuration with 4 nodes
|
|
NCCL_DEBUG=version ./topo_expl -m 42 -n 4
|
|
```
|
|
|
|
## Test Suite Usage
|
|
|
|
The integrated test suite allows comprehensive testing of all models with various node configurations:
|
|
|
|
```bash
|
|
# Run test suite for all models with default node counts (1,2,4,8,16)
|
|
./topo_expl -t
|
|
|
|
# Test only specific models
|
|
./topo_expl -t --include-models=0,1,2,4,8
|
|
|
|
# Test all models with specific node counts
|
|
./topo_expl -t --include-nodes=1,4
|
|
|
|
# Test all models except specific ones
|
|
./topo_expl -t --exclude-models=57,59
|
|
|
|
# Test all models except 16-node configurations
|
|
./topo_expl -t --exclude-nodes=16
|
|
|
|
# Combine include and exclude options
|
|
./topo_expl -t --include-models=0,1,2 --include-nodes=1,4 --exclude-models=1
|
|
```
|
|
|
|
|
|
## Sample output
|
|
|
|
```bash
|
|
# cmd used (-m/-n or --model/--nodes)
|
|
NCCL_DEBUG=version ./topo_expl -m 55 -n 8
|
|
```
|
|
|
|
```bash
|
|
|
|
Running fp32 production choices for algorithm/protocol/maxChannels
|
|
| Max Size(B) | Count | Collective | Algorithm | Protocol | Max Channels |
|
|
|-----------------|-----------------|-----------------|------------|------------|--------------|
|
|
| 32 | 8 | AllReduce | Tree | LL | 1 |
|
|
| 64 | 16 | AllReduce | Tree | LL | 1 |
|
|
| 128 | 32 | AllReduce | Tree | LL | 1 |
|
|
| 256 | 64 | AllReduce | Tree | LL | 1 |
|
|
| 512 | 128 | AllReduce | Tree | LL | 1 |
|
|
| 1024 | 256 | AllReduce | Tree | LL | 1 |
|
|
| 2048 | 512 | AllReduce | Tree | LL | 1 |
|
|
| 4096 | 1024 | AllReduce | Tree | LL | 2 |
|
|
| 8192 | 2048 | AllReduce | Tree | LL | 4 |
|
|
| 16384 | 4096 | AllReduce | Tree | LL | 8 |
|
|
| 32768 | 8192 | AllReduce | Tree | LL | 16 |
|
|
| 65536 | 16384 | AllReduce | Tree | LL | 32 |
|
|
| 131072 | 32768 | AllReduce | Tree | LL | 64 |
|
|
| 262144 | 65536 | AllReduce | Tree | LL | 64 |
|
|
| 524288 | 131072 | AllReduce | Tree | LL | 64 |
|
|
| 1048576 | 262144 | AllReduce | Tree | LL | 64 |
|
|
| 2097152 | 524288 | AllReduce | Tree | LL128 | 64 |
|
|
| 4194304 | 1048576 | AllReduce | Tree | LL128 | 64 |
|
|
| 8388608 | 2097152 | AllReduce | Tree | LL128 | 64 |
|
|
| 16777216 | 4194304 | AllReduce | Tree | LL128 | 64 |
|
|
| 33554432 | 8388608 | AllReduce | Tree | LL128 | 64 |
|
|
| 67108864 | 16777216 | AllReduce | Tree | Simple | 64 |
|
|
| 134217728 | 33554432 | AllReduce | Tree | Simple | 64 |
|
|
| 268435456 | 67108864 | AllReduce | Tree | Simple | 64 |
|
|
| 536870912 | 134217728 | AllReduce | Ring | Simple | 64 |
|
|
| 1073741824 | 268435456 | AllReduce | Ring | Simple | 64 |
|
|
| 2147483648 | 536870912 | AllReduce | Ring | Simple | 64 |
|
|
| 4294967296 | 1073741824 | AllReduce | Ring | Simple | 64 |
|
|
...
|
|
```
|
|
|