derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
83 lines
2.9 KiB
Markdown
83 lines
2.9 KiB
Markdown
# RCCL CSV Tuner Configuration
|
|
|
|
This directory is installed to `${ROCM_PATH}/share/rccl/tuner/` when RCCL is built and installed. Any CSV files added here will be automatically copied to the install location.
|
|
|
|
## How to Use
|
|
|
|
### For Development (running from build directory)
|
|
|
|
Place your CSV config files in the `tuner/` directory within your build directory:
|
|
|
|
```bash
|
|
# From the build directory (e.g., build/release/)
|
|
cp rccl_tuner_gfx950.csv tuner/
|
|
./rccl-tests/all_reduce_perf # Will auto-detect tuner/rccl_tuner_gfx950.csv
|
|
```
|
|
|
|
### For Production (after installing RCCL)
|
|
|
|
Place your CSV config files in the installed location:
|
|
|
|
```bash
|
|
sudo cp rccl_tuner_gfx950.csv /opt/rocm/share/rccl/tuner/
|
|
```
|
|
|
|
RCCL will automatically detect and load the config file at runtime.
|
|
|
|
## Auto-Discovery Order
|
|
|
|
The built-in CSV tuner searches for config files in this order:
|
|
|
|
1. `NCCL_TUNER_CONFIG_FILE` environment variable (if set)
|
|
2. `<librccl.so dir>/tuner/rccl_tuner_<arch>.csv` (adjacent to library, for development builds)
|
|
3. `<librccl.so dir>/tuner/rccl_tuner.csv` (adjacent to library, for development builds)
|
|
4. `<librccl.so dir>/../share/rccl/tuner/rccl_tuner_<arch>.csv` (relative share path, for installed RCCL)
|
|
5. `<librccl.so dir>/../share/rccl/tuner/rccl_tuner.csv` (relative share path, for installed RCCL)
|
|
6. `${ROCM_PATH}/share/rccl/tuner/rccl_tuner_<arch>.csv` (fallback, GPU-specific)
|
|
7. `${ROCM_PATH}/share/rccl/tuner/rccl_tuner.csv` (fallback, generic)
|
|
|
|
At each location, if GPU architecture is unknown, the directory is scanned for any `rccl_tuner*.csv` file.
|
|
|
|
## CSV Format
|
|
|
|
Each line contains 8-10 comma-separated fields:
|
|
|
|
```
|
|
colltype,minbytes,maxbytes,algorithm,protocol,channels,nNodes,nRanks[,numPipeOps][,regBuff]
|
|
```
|
|
|
|
### Fields
|
|
|
|
| Field | Values | Description |
|
|
|-------|--------|-------------|
|
|
| colltype | `broadcast`, `reduce`, `allgather`, `reducescatter`, `allreduce` | Collective operation type |
|
|
| minbytes | integer | Minimum message size in bytes |
|
|
| maxbytes | integer | Maximum message size in bytes |
|
|
| algorithm | `tree`, `ring`, `collnet_direct`, `collnet_chain`, `nvls`, `nvls_tree`, `pat` | Algorithm to use |
|
|
| protocol | `ll`, `ll128`, `simple` | Protocol to use |
|
|
| channels | integer or `-1` | Number of channels (`-1` = RCCL default) |
|
|
| nNodes | integer or `-1` | Node count to match (`-1` = any) |
|
|
| nRanks | integer or `-1` | Rank count to match (`-1` = any) |
|
|
| numPipeOps | integer or `-1` | (Optional) Pipeline ops (`-1` = any) |
|
|
| regBuff | `0`, `1`, or `-1` | (Optional) Buffer registration (`-1` = any) |
|
|
|
|
### Example Config
|
|
|
|
```csv
|
|
# Tuning for 2-node gfx950 with 16 ranks
|
|
allreduce,0,65536,tree,ll,8,2,16,-1,-1
|
|
allreduce,65537,1048576,ring,ll128,16,2,16,-1,-1
|
|
allreduce,1048577,17179869184,ring,simple,64,2,16,-1,-1
|
|
|
|
# AllGather tuning
|
|
allgather,0,32768,ring,ll,4,-1,-1,-1,-1
|
|
```
|
|
|
|
## Disabling the Tuner
|
|
|
|
To disable the built-in CSV tuner even when a config file exists:
|
|
|
|
```bash
|
|
export NCCL_TUNER_PLUGIN=none
|
|
```
|