TheRock/rocm-systems/projects/rccl/examples/03_collectives/01_allreduce

NCCL AllReduce Collective Operation Example

This example demonstrates the fundamental AllReduce collective operation using NCCL's single-process, multi-GPU approach in which a single process manages all GPUs to perform a sum reduction.

Overview

AllReduce combines data from all participants using a reduction operation (sum, max, min, etc.) and distributes the result to all participants. This example shows how each GPU contributes its rank value and all GPUs receive the combined sum using ncclCommInitAll for simplified setup.

What This Example Does

1. Detects available GPUs and initializes NCCL communicators for all devices using ncclCommInitAll
2. Initializes data with each GPU contributing its rank value (GPU 0→0, GPU 1→1, etc.)
3. Performs AllReduce sum operation where all GPU values are summed and distributed to all participants
4. Verifies correctness by checking that all GPUs received the expected sum: 0+1+2+...+(n-1)

Building and Running

Build the Example

cd examples/03_collectives/01_allreduce
make [ROCM_PATH=<path-to-rocm>] [RCCL_HOME=<path-to-rccl>]

Run with All Available GPUs

./allreduce

Run with Specific GPUs

HIP_VISIBLE_DEVICES=0,1,2,3 ./allreduce

Code Walk-through

Data Initialization

Each GPU sets a send buffer allocated on the GPU to its rank value:

float** sendbuff;
float rank_value = (float)i;
size_t size; // size is the number of float to be sent

// Allocate device memory for send buffers
CUDACHECK(hipMalloc((void **)&sendbuff[i], size * sizeof(float)));

// Each GPU contributes its rank (GPU i contributes value i)
// Zero the entire buffer, then set first element to rank
CUDACHECK(hipMemset(sendbuff[i], 0, size * sizeof(float)));
CUDACHECK(hipMemcpy(sendbuff[i], &rank_value, sizeof(float), hipMemcpyHostToDevice));

AllReduce Operation

All GPUs participate in the sum reduction. The operations are evaluated in parallel within a NCCL group to avoid any deadlocks.

float** recvbuff;
ncclComm_t *comms;  // comms are set during ncclCommInitAll
hipStream_t *streams; // streams are set in hipStreamCreate

// Allocate device memory for receive buffers
CUDACHECK(hipMalloc((void **)&recvbuff[i], size * sizeof(float)));

NCCLCHECK(ncclGroupStart());
for (int i = 0; i < num_gpus; i++) {
    NCCLCHECK(ncclAllReduce(sendbuff[i], recvbuff[i], size, ncclFloat,
                            ncclSum, comms[i], streams[i]));
}
NCCLCHECK(ncclGroupEnd());

Expected Output

Using 4 devices for collective communication
Memory allocated for 4 communicators and streams
NCCL communicators initialized for all devices
  Device 0 initialized with data value 0
  Device 1 initialized with data value 1
  Device 2 initialized with data value 2
  Device 3 initialized with data value 3
Starting collective sum operation across all devices
Collective operation completed
Verifying results (expected sum: 6)
  Device 0 correctly received sum: 6
  Device 1 correctly received sum: 6
  Device 2 correctly received sum: 6
  Device 3 correctly received sum: 6
Example completed successfully!

When to Use

• Deep learning: Gradient averaging in data-parallel training
• Scientific computing: Global reductions in parallel algorithms
• Statistics: Computing global sums, averages, or other reductions
• Distributed algorithms: Any scenario requiring collective reduction operations

Key Insights

• ncclCommInitAll simplifies single-node multi-GPU setup
• No MPI or pthreads needed for single-node patterns
• Allocate device buffer via hipMalloc and initialize via hipMemset.
• Best practices to wrap all collective calls in ncclGroupStart/End
• All communication happens in parallel

Common Issues and Solutions

Issue: Verification failures

Solution: Ensure each GPU initializes its buffer correctly with its rank value.

Issue: Out of memory errors

Solution: Reduce the buffer size in the code or use fewer GPUs.

Error Handling

This example uses comprehensive error checking with NCCLCHECK and CUDACHECK macros that immediately exit on any failure. In production code, consider more graceful error handling and recovery mechanisms.

Next Steps

After understanding AllReduce, explore:

• Point-to-point communication: Examples in 02_point_to_point/
• Other collectives: Implement Broadcast, Reduce, AllGather operations using this example
• Multi-node approach: Use the MPI implementation from 01_communicators to send data across nodes.