derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
88 lines
2.5 KiB
Markdown
88 lines
2.5 KiB
Markdown
# rocSHMEM Host-Stream API Tracing Example
|
|
|
|
## Overview
|
|
|
|
This example exercises all nine rocSHMEM host-stream APIs in a minimal two-PE
|
|
program. Every PE calls every API so that rocprofiler-systems tracing captures
|
|
all nine `rocm_rocshmem_api` spans:
|
|
|
|
| API | Description |
|
|
| --- | --- |
|
|
| `barrier_all_on_stream` | Global barrier across all PEs |
|
|
| `sync_all_on_stream` | Lightweight global synchronization |
|
|
| `putmem_on_stream` | One-sided write to a remote PE |
|
|
| `getmem_on_stream` | One-sided read from a remote PE |
|
|
| `putmem_signal_on_stream` | Write data and set a remote signal atomically |
|
|
| `signal_wait_until_on_stream` | Wait until a local signal word satisfies a condition |
|
|
| `broadcastmem_on_stream` | PE 0 broadcasts a buffer to all PEs |
|
|
| `alltoallmem_on_stream` | Each PE sends a buffer to every other PE |
|
|
| `quiet_on_stream` | Ensure all outstanding remote operations are complete |
|
|
|
|
## Source Files
|
|
|
|
- `rocshmem.cpp` — Minimal two-PE program that calls all nine host-stream APIs
|
|
in sequence using symmetric memory buffers on a single HIP stream.
|
|
|
|
## Prerequisites
|
|
|
|
- CMake 3.25+
|
|
- ROCm with HIP runtime
|
|
- rocSHMEM ≥ 3.6.0 (included in ROCm 10.0)
|
|
- MPI runtime (e.g. Open MPI)
|
|
|
|
## Building
|
|
|
|
**Standalone build:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/rocshmem \
|
|
-DCMAKE_PREFIX_PATH=/opt/rocm
|
|
cmake --build <build_dir>
|
|
```
|
|
|
|
**As part of the examples suite:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/
|
|
cmake --build <build_dir> --target rocshmem-test
|
|
```
|
|
|
|
## Running
|
|
|
|
```bash
|
|
mpirun -np 2 ./rocshmem-test
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```text
|
|
[PE 0/2] rocshmem: all 9 host-stream APIs completed
|
|
[PE 1/2] rocshmem: all 9 host-stream APIs completed
|
|
```
|
|
|
|
## Profiling with rocprofiler-systems
|
|
|
|
```bash
|
|
mpirun -np 2 rocprof-sys-run -- ./rocshmem-test
|
|
```
|
|
|
|
### Recommended Configuration
|
|
|
|
| Variable | Value | Purpose |
|
|
| --- | --- | --- |
|
|
| `ROCPROFSYS_ROCM_DOMAINS` | `hip_runtime_api,kernel_dispatch,rocshmem_api` | Enable HIP runtime API, kernel dispatch, and rocSHMEM API tracing |
|
|
| `ROCPROFSYS_TRACE` | `true` | Generate Perfetto trace as output |
|
|
| `ROCPROFSYS_ROCPD` | `true` | Generate rocpd database as output |
|
|
|
|
```bash
|
|
ROCPROFSYS_ROCM_DOMAINS="hip_runtime_api,kernel_dispatch,rocshmem_api" \
|
|
ROCPROFSYS_TRACE=true \
|
|
ROCPROFSYS_ROCPD=true \
|
|
mpirun -np 2 rocprof-sys-run \
|
|
-- ./rocshmem-test
|
|
```
|
|
|
|
The resulting rocpd database will contain `rocm_rocshmem_api` spans for each of
|
|
the nine host-stream API calls made by each PE. It can be viewed with
|
|
[ROCm Optiq](https://github.com/ROCm/roc-optiq) visualizer.
|