derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
75 lines
2.5 KiB
Markdown
75 lines
2.5 KiB
Markdown
# SHMEM
|
|
|
|
## Overview
|
|
|
|
This directory contains two OpenSHMEM examples. `shmem-hello` demonstrates basic SHMEM initialization, symmetric memory allocation, and remote read operations between processing elements (PEs). `shmem-pingpong` implements a latency microbenchmark that measures round-trip message passing time between two PEs using various SHMEM communication strategies (polling, `shmem_wait_until`, `shmem_fence`, `shmem_quiet`). These examples are useful for profiling one-sided communication patterns in PGAS (Partitioned Global Address Space) programming models.
|
|
|
|
## Source Files
|
|
|
|
- `shmem-hello.c` - Initializes SHMEM, allocates symmetric memory with `shmem_malloc()`, performs `shmem_int_get()` from the next PE (circular), and prints the result.
|
|
- `shmem-pingpong.c` - Ping-pong latency benchmark with configurable message size, iteration count, and communication mode. Supports polling, `shmem_wait_until()`, `shmem_fence()`, `shmem_quiet()`, and global vs. heap data placement.
|
|
|
|
## Prerequisites
|
|
|
|
- CMake 3.25+
|
|
- OpenSHMEM implementation with `oshcc` compiler wrapper
|
|
|
|
## Building
|
|
|
|
**Standalone build:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/shmem
|
|
cmake --build <build_dir>
|
|
```
|
|
|
|
**As part of the examples suite:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/
|
|
cmake --build <build_dir> --target shmem-hello shmem-pingpong
|
|
```
|
|
|
|
**Targets:**
|
|
|
|
| Target | Description |
|
|
| -------- | ------------- |
|
|
| `shmem-hello` | Basic SHMEM hello world |
|
|
| `shmem-pingpong` | Ping-pong latency benchmark |
|
|
|
|
## Running
|
|
|
|
```bash
|
|
# Hello world (any number of PEs)
|
|
oshrun -np 4 ./shmem-hello
|
|
|
|
# Ping-pong (requires exactly 2 PEs)
|
|
oshrun -np 2 ./shmem-pingpong
|
|
```
|
|
|
|
**shmem-pingpong flags:**
|
|
|
|
| Flag | Description | Default |
|
|
| ------ | ------------- | --------- |
|
|
| `-n` | Number of iterations | 10000 |
|
|
| `-s` | Message size in bytes | 4 |
|
|
| `-w` | Use `shmem_wait_until()` instead of polling | off |
|
|
| `-f` | Send data and flag separately with `shmem_fence()` | off |
|
|
| `-g` | Use global data instead of heap | off |
|
|
| `-q` | Call `shmem_quiet()` after each put | off |
|
|
|
|
## Profiling with rocprofiler-systems
|
|
|
|
```bash
|
|
oshrun -np 2 rocprof-sys-run -- ./shmem-pingpong -n 1000
|
|
```
|
|
|
|
### Recommended Configuration
|
|
|
|
| Variable | Value | Purpose |
|
|
| ---------- | ------- | --------- |
|
|
| `ROCPROFSYS_TRACE` | `true` | Generate Perfetto trace |
|
|
| `ROCPROFSYS_PROFILE` | `true` | Generate call-stack profile |
|
|
| `ROCPROFSYS_USE_SAMPLING` | `ON` | Enable statistical sampling |
|
|
| `ROCPROFSYS_SAMPLING_FREQ` | `50` | Sampling frequency (interrupts/sec) |
|