TheRock/rocm-systems/projects/rccl/examples

RCCL Library Examples

This directory contains RCCL (ROCm Communication Collectives Library) examples, ported from the upstream NCCL examples. Because RCCL exposes the same ncclXxx API surface as NCCL, the application code is largely identical — only the underlying GPU runtime calls have been translated from CUDA to HIP, and the build system retargeted to hipcc and librccl.

The examples progress from basic concepts to advanced usage patterns, each with its own README. See the RCCL documentation for a full reference.

Categories

Folder	What it shows
01_communicators	Communicator creation: single-process multi-GPU, one device per pthread, one device per MPI process
02_point_to_point	Send/Recv in a simple ring pattern
03_collectives	Basic AllReduce
04_user_buffer_registration	ncclCommRegister to pre-register user buffers for faster collectives
05_symmetric_memory	ncclCommWindowRegister symmetric memory windows
06_device_api	Device-side API (kept as the RCCL-specific examples already present in the project)

Prerequisites

• ROCm (default: /opt/rocm). Override with ROCM_PATH=....
• RCCL headers and library (shipped with ROCm; override location with RCCL_HOME=...).
• hipcc in $ROCM_PATH/bin.
• Optional: an MPI implementation for the MPI example.

Build

From this directory:

make GPU_TARGETS="gfx942"            # build all 01..05 categories
make -C 03_collectives/01_allreduce  # build one example
make MPI=1 -C 01_communicators/03_one_device_per_process_mpi

Useful variables:

• GPU_TARGETS — space-separated AMD GPU archs, e.g. "gfx90a gfx942 gfx1100". Defaults to those three.
• ROCM_PATH — ROCm install root (default /opt/rocm).
• RCCL_HOME — RCCL install root (default $(ROCM_PATH)).
• MPI=1 — build with MPI support. Optionally set MPI_HOME.
• PREFIX — install prefix for make install (default /usr/local).

The build uses hipcc for both compilation and linking. For the MPI example, OMPI_CXX/MPICH_CXX are exported so the MPI wrapper invokes hipcc.

Run

./03_collectives/01_allreduce/allreduce
mpirun -np 2 ./01_communicators/03_one_device_per_process_mpi/one_device_per_process_mpi

By default the threaded/single-process examples use every visible GPU; restrict with HIP_VISIBLE_DEVICES=0,1,2,3.

Environment Variables (runtime)

• HIP_VISIBLE_DEVICES — comma-separated list of GPUs visible to the application.
• NCCL_DEBUG=INFO — verbose RCCL logging (RCCL honors the NCCL_* env vars).
• All other RCCL environment variables apply.

Troubleshooting

• Use NCCL_DEBUG=INFO for detailed RCCL logging.
• Ensure your GPU's gfx arch is included in GPU_TARGETS (check with rocminfo).
• For MPI builds, confirm mpicxx/mpirun are on PATH or set MPI_HOME.