TheRock/rocm-systems/projects/rocprofiler-systems/examples/mpi/README.md

MPI

Overview

This example benchmarks various MPI communication patterns including point-to-point (MPI_Send/MPI_Recv) and collective operations (MPI_Alltoall, MPI_Allgather, MPI_Bcast, MPI_Reduce, MPI_Scatter/MPI_Gather, MPI_Allreduce). The main C++ program exercises send/recv and all-to-all with different data types across duplicated and split communicators, while individual C programs each demonstrate a specific MPI collective. This suite is useful for profiling MPI interposition (ROCPROFSYS_USE_MPIP) and understanding communication overhead in distributed workloads.

Source Files

• mpi.cpp - Main benchmark with all2all<Tp,N>() and send_recv<Tp,N>() templated over data types, plus communicator duplication and splitting by prime/non-prime rank.
• allgather.c - MPI_Allgather example
• allreduce.c - MPI_Allreduce example
• all2all.c - MPI_Alltoall example
• bcast.c - MPI_Bcast example
• reduce.c - MPI_Reduce example
• scatter-gather.c - MPI_Scatter/MPI_Gather example
• send-recv.c - MPI_Send/MPI_Recv example

Prerequisites

• CMake 3.25+
• MPI implementation (OpenMPI, MPICH, etc.)
• C and C++ compilers

Building

Standalone build:

cmake -B <build_dir> -S <project_root>/examples/mpi
cmake --build <build_dir>

As part of the examples suite:

cmake -B <build_dir> -S <project_root>/examples/
cmake --build <build_dir> --target mpi-example

Targets:

Target	Description
mpi-example	C++ benchmark (send/recv + all2all with communicator ops)
mpi-allgather	C allgather example
mpi-allreduce	C allreduce example
mpi-all2all	C all-to-all example
mpi-bcast	C broadcast example
mpi-reduce	C reduce example
mpi-scatter-gather	C scatter/gather example
mpi-send-recv	C send/recv example

Running

# Run with 2 MPI processes
mpirun -np 2 ./mpi-example

# Run individual collective tests
mpirun -np 4 ./mpi-allreduce
mpirun -np 4 ./mpi-scatter-gather

Arguments (mpi-example):

Position	Description	Default
1	Number of iterations	1

Profiling with rocprofiler-systems

mpirun -np 2 rocprof-sys-run -- ./mpi-example

Recommended Configuration

Variable	Value	Purpose
ROCPROFSYS_USE_MPIP	ON	Enable MPI profiling interposition
ROCPROFSYS_USE_MPI	ON	Enable MPI-aware output merging
ROCPROFSYS_TRACE	true	Generate Perfetto trace
ROCPROFSYS_PROFILE	true	Generate call-stack profile
ROCPROFSYS_USE_SAMPLING	ON	Enable sampling

mpirun -np 2 rocprof-sys-run \
    -e ROCPROFSYS_USE_MPIP=ON \
    -e ROCPROFSYS_TRACE=true \
    -- ./mpi-example

For flat profiling with collapsed processes/threads:

mpirun -np 2 rocprof-sys-run \
    -e ROCPROFSYS_USE_MPIP=ON \
    -e ROCPROFSYS_FLAT_PROFILE=ON \
    -e ROCPROFSYS_COLLAPSE_PROCESSES=ON \
    -e ROCPROFSYS_COLLAPSE_THREADS=ON \
    -- ./mpi-example