derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
103 lines
3.0 KiB
Markdown
103 lines
3.0 KiB
Markdown
# MPI
|
|
|
|
## Overview
|
|
|
|
This example benchmarks various MPI communication patterns including point-to-point (`MPI_Send`/`MPI_Recv`) and collective operations (`MPI_Alltoall`, `MPI_Allgather`, `MPI_Bcast`, `MPI_Reduce`, `MPI_Scatter`/`MPI_Gather`, `MPI_Allreduce`). The main C++ program exercises send/recv and all-to-all with different data types across duplicated and split communicators, while individual C programs each demonstrate a specific MPI collective. This suite is useful for profiling MPI interposition (`ROCPROFSYS_USE_MPIP`) and understanding communication overhead in distributed workloads.
|
|
|
|
## Source Files
|
|
|
|
- `mpi.cpp` - Main benchmark with `all2all<Tp,N>()` and `send_recv<Tp,N>()` templated over data types, plus communicator duplication and splitting by prime/non-prime rank.
|
|
- `allgather.c` - MPI_Allgather example
|
|
- `allreduce.c` - MPI_Allreduce example
|
|
- `all2all.c` - MPI_Alltoall example
|
|
- `bcast.c` - MPI_Bcast example
|
|
- `reduce.c` - MPI_Reduce example
|
|
- `scatter-gather.c` - MPI_Scatter/MPI_Gather example
|
|
- `send-recv.c` - MPI_Send/MPI_Recv example
|
|
|
|
## Prerequisites
|
|
|
|
- CMake 3.25+
|
|
- MPI implementation (OpenMPI, MPICH, etc.)
|
|
- C and C++ compilers
|
|
|
|
## Building
|
|
|
|
**Standalone build:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/mpi
|
|
cmake --build <build_dir>
|
|
```
|
|
|
|
**As part of the examples suite:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/
|
|
cmake --build <build_dir> --target mpi-example
|
|
```
|
|
|
|
**Targets:**
|
|
|
|
| Target | Description |
|
|
| -------- | ------------- |
|
|
| `mpi-example` | C++ benchmark (send/recv + all2all with communicator ops) |
|
|
| `mpi-allgather` | C allgather example |
|
|
| `mpi-allreduce` | C allreduce example |
|
|
| `mpi-all2all` | C all-to-all example |
|
|
| `mpi-bcast` | C broadcast example |
|
|
| `mpi-reduce` | C reduce example |
|
|
| `mpi-scatter-gather` | C scatter/gather example |
|
|
| `mpi-send-recv` | C send/recv example |
|
|
|
|
## Running
|
|
|
|
```bash
|
|
# Run with 2 MPI processes
|
|
mpirun -np 2 ./mpi-example
|
|
|
|
# Run individual collective tests
|
|
mpirun -np 4 ./mpi-allreduce
|
|
mpirun -np 4 ./mpi-scatter-gather
|
|
```
|
|
|
|
**Arguments (mpi-example):**
|
|
|
|
| Position | Description | Default |
|
|
| ---------- | ------------- | --------- |
|
|
| 1 | Number of iterations | 1 |
|
|
|
|
## Profiling with rocprofiler-systems
|
|
|
|
```bash
|
|
mpirun -np 2 rocprof-sys-run -- ./mpi-example
|
|
```
|
|
|
|
### Recommended Configuration
|
|
|
|
| Variable | Value | Purpose |
|
|
| ---------- | ------- | --------- |
|
|
| `ROCPROFSYS_USE_MPIP` | `ON` | Enable MPI profiling interposition |
|
|
| `ROCPROFSYS_USE_MPI` | `ON` | Enable MPI-aware output merging |
|
|
| `ROCPROFSYS_TRACE` | `true` | Generate Perfetto trace |
|
|
| `ROCPROFSYS_PROFILE` | `true` | Generate call-stack profile |
|
|
| `ROCPROFSYS_USE_SAMPLING` | `ON` | Enable sampling |
|
|
|
|
```bash
|
|
mpirun -np 2 rocprof-sys-run \
|
|
-e ROCPROFSYS_USE_MPIP=ON \
|
|
-e ROCPROFSYS_TRACE=true \
|
|
-- ./mpi-example
|
|
```
|
|
|
|
For flat profiling with collapsed processes/threads:
|
|
|
|
```bash
|
|
mpirun -np 2 rocprof-sys-run \
|
|
-e ROCPROFSYS_USE_MPIP=ON \
|
|
-e ROCPROFSYS_FLAT_PROFILE=ON \
|
|
-e ROCPROFSYS_COLLAPSE_PROCESSES=ON \
|
|
-e ROCPROFSYS_COLLAPSE_THREADS=ON \
|
|
-- ./mpi-example
|
|
```
|