derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
RCCL CSV Tuner Configuration
This directory is installed to ${ROCM_PATH}/share/rccl/tuner/ when RCCL is built and installed. Any CSV files added here will be automatically copied to the install location.
How to Use
For Development (running from build directory)
Place your CSV config files in the tuner/ directory within your build directory:
# From the build directory (e.g., build/release/)
cp rccl_tuner_gfx950.csv tuner/
./rccl-tests/all_reduce_perf # Will auto-detect tuner/rccl_tuner_gfx950.csv
For Production (after installing RCCL)
Place your CSV config files in the installed location:
sudo cp rccl_tuner_gfx950.csv /opt/rocm/share/rccl/tuner/
RCCL will automatically detect and load the config file at runtime.
Auto-Discovery Order
The built-in CSV tuner searches for config files in this order:
NCCL_TUNER_CONFIG_FILEenvironment variable (if set)<librccl.so dir>/tuner/rccl_tuner_<arch>.csv(adjacent to library, for development builds)<librccl.so dir>/tuner/rccl_tuner.csv(adjacent to library, for development builds)<librccl.so dir>/../share/rccl/tuner/rccl_tuner_<arch>.csv(relative share path, for installed RCCL)<librccl.so dir>/../share/rccl/tuner/rccl_tuner.csv(relative share path, for installed RCCL)${ROCM_PATH}/share/rccl/tuner/rccl_tuner_<arch>.csv(fallback, GPU-specific)${ROCM_PATH}/share/rccl/tuner/rccl_tuner.csv(fallback, generic)
At each location, if GPU architecture is unknown, the directory is scanned for any rccl_tuner*.csv file.
CSV Format
Each line contains 8-10 comma-separated fields:
colltype,minbytes,maxbytes,algorithm,protocol,channels,nNodes,nRanks[,numPipeOps][,regBuff]
Fields
| Field | Values | Description |
|---|---|---|
| colltype | broadcast, reduce, allgather, reducescatter, allreduce |
Collective operation type |
| minbytes | integer | Minimum message size in bytes |
| maxbytes | integer | Maximum message size in bytes |
| algorithm | tree, ring, collnet_direct, collnet_chain, nvls, nvls_tree, pat |
Algorithm to use |
| protocol | ll, ll128, simple |
Protocol to use |
| channels | integer or -1 |
Number of channels (-1 = RCCL default) |
| nNodes | integer or -1 |
Node count to match (-1 = any) |
| nRanks | integer or -1 |
Rank count to match (-1 = any) |
| numPipeOps | integer or -1 |
(Optional) Pipeline ops (-1 = any) |
| regBuff | 0, 1, or -1 |
(Optional) Buffer registration (-1 = any) |
Example Config
# Tuning for 2-node gfx950 with 16 ranks
allreduce,0,65536,tree,ll,8,2,16,-1,-1
allreduce,65537,1048576,ring,ll128,16,2,16,-1,-1
allreduce,1048577,17179869184,ring,simple,64,2,16,-1,-1
# AllGather tuning
allgather,0,32768,ring,ll,4,-1,-1,-1,-1
Disabling the Tuner
To disable the built-in CSV tuner even when a config file exists:
export NCCL_TUNER_PLUGIN=none