TheRock/rocm-systems/projects/rccl/examples/03_collectives

NCCL Collective Communication Examples

Overview

This directory contains minimal examples that demonstrate NCCL collective communication operations on a single node using a single process managing all GPUs. The focus is clarity, correct resource management, and result verification.

Examples

01_allreduce

AllReduce Collective Operation

• Pattern: All participants reduce and distribute the result
• API: ncclCommInitAll, ncclAllReduce
• Use case: Global reductions in ML and HPC (e.g., gradient averaging)
• Key features:
  • Initializes all GPUs in a single process
  • Each GPU contributes its rank value
  • Executes AllReduce sum across all GPUs
  • Verifies the expected global sum

Choosing the Right Pattern

Scenario : Parallel training needs efficient global communication Addresses : Most commonly used collective algorithms Dependencies : A functional NCCL library and its dependencies

Why ncclCommInitAll here?

For single-node collective examples we use ncclCommInitAll as it creates a clique of communicators in one call.

// Initialize all GPUs in one call
ncclComm_t* comms;
int num_gpus;
NCCLCHECK(ncclCommInitAll(comms, num_gpus, NULL));

A more advanced setup using MPI to initialize communicators across multiple nodes is shown in 01_communicators/03_one_device_per_process_mpi

Building

Quick Start

# Build example by directory name
make 01_allreduce

Individual Examples

# Build and run AllReduce
cd 01_allreduce && make
./allreduce

References

• NCCL User Guide: Examples
• NCCL API Reference
• HIP Programming Guide