TheRock/rocm-systems/projects/rocshmem/scripts/functional_tests/README-perf_compare.md

rocSHMEM Performance Scripts

Scripts for building, running, and comparing rocSHMEM performance across configurations.

Quick-start: compare current branch vs develop

The baseline is automatically set to the merge-base of your current branch with origin/develop (i.e. the common ancestor, not develop HEAD), so results reflect only the changes on your branch.

# From the rocm-systems/projects/rocshmem directory:
./scripts/functional_tests/run_perf_compare.sh --suite heatmap --iterations 5

# Output:
#   plots-heatmap-<branch>/heatmap_summary.png   summary heatmap (green = faster)
#   plots-heatmap-<branch>/heatmap_data.csv       raw percentage data (CSV)
#   plots-heatmap-<branch>/heatmap_summary.txt     summary % table (text)
#   plots-heatmap-<branch>/per_test/*.png         per-test latency-vs-size curves
#   plots-heatmap-<branch>/per_test/*.txt         per-test salient metrics + IQR stats

Run --suite all for the full functional test suite (slower):

./scripts/functional_tests/run_perf_compare.sh --suite all --iterations 5

Skip rebuild on second run

After the first full run, use --skip-build to re-run tests without rebuilding:

./scripts/functional_tests/run_perf_compare.sh --suite heatmap --iterations 20 --skip-build

Skip only the baseline (build and test runs) when the baseline data is already available:

./scripts/functional_tests/run_perf_compare.sh --suite heatmap --skip-baseline

Skip everything and just regenerate plots from existing logs:

./scripts/functional_tests/run_perf_compare.sh --suite heatmap --skip-build --skip-baseline --skip-branch

Multiple named variants (e.g. feature flag comparison)

Use --variant-args NAME:ENV=VAL,...:extra-cmake-args to compare multiple configurations of the same branch. Each spec produces its own build directory and named column in the heatmap.

# Example: compare SDMA off/disabled/enabled against develop baseline
./scripts/functional_tests/run_perf_compare.sh \
  --branch-args "-DUSE_SDMA=ON" \
  --variant-args "sdma-off::-DUSE_SDMA=OFF" \
  --variant-args "sdma-disabled:ROCSHMEM_SDMA_ENABLED=0:" \
  --variant-args "sdma-enabled:ROCSHMEM_SDMA_ENABLED=1:"

Fields in each spec (separated by :):

1. Name — label in plots and build directory suffix
2. Env vars — comma-separated KEY=VALUE pairs set at runtime (empty = none)
3. CMake args — extra cmake flags for this variant's build (empty = none)

Compare a GitHub PR

./scripts/functional_tests/run_perf_compare.sh --pr 4574 --suite heatmap --iterations 5
# Fetches PR #4574, builds it vs origin/develop, outputs plots-heatmap-pr4574/

Generate the heatmap from existing logs (no rebuild, no re-run)

If you already have log directories from a previous run:

python3 scripts/functional_tests/perf_compare.py \
  --baseline "path/to/build-baseline/logs-heatmap-*" \
  --variants "mybranch:path/to/build-branch/logs-heatmap-*" \
  --outdir plots/

# With multiple variants:
python3 scripts/functional_tests/perf_compare.py \
  --baseline "build-develop/logs-heatmap-*" \
  --variants \
    "sdma-off:build-sdma-off/logs-heatmap-*" \
    "sdma-disabled:build-sdma-on/logs-heatmap-disabled-*" \
    "sdma-enabled:build-sdma-on/logs-heatmap-enabled-*" \
  --outdir plots/

The script auto-detects multiple iteration directories (glob with *) and aggregates them with IQR outlier removal before computing medians.

Docker-based PR comparison

Note: the Docker baseline is the HEAD of develop at the time the image was built, not the PR's merge-base. For a fresh image against a recent PR the difference is usually negligible. For a precise merge-base comparison, use the local workflow instead.

Option A: one-shot (build image, run, get results on host)

# From the projects/rocshmem directory:
# Build the image — compiles both develop (/app/build) and PR (/app/build4574)
docker build -f docker/Dockerfile.ubuntu --build-arg PR_NUM=4574 --tag $USER/rocshmem-pr4574 docker/

# Run comparison; results appear on the host at ./pr4574-results/
mkdir -p pr4574-results
docker run --rm \
  --shm-size 64G --network host --device /dev/dri --device /dev/kfd \
  --ipc host --group-add video --cap-add SYS_PTRACE \
  --security-opt seccomp=unconfined --privileged \
  -v "$(pwd)/pr4574-results:/results" \
  $USER/rocshmem-pr4574 perf-compare --suite heatmap --iterations 5
# Plots: ./pr4574-results/heatmap_summary.png

Option B: interactive session (build once, re-run quickly)

# Start a persistent container with a bind-mount for results
docker run -d --name rocshmem-pr4574 \
  --shm-size 64G --network host --device /dev/dri --device /dev/kfd \
  --ipc host --group-add video --cap-add SYS_PTRACE \
  --security-opt seccomp=unconfined --privileged \
  -v "$(pwd)/pr4574-results:/results" \
  $USER/rocshmem-pr4574 sleep infinity

# Jump interactive, do some modifications, rebuild
docker exec -it rocshmem-pr4574 bash

# Run as many times as needed (e.g. vary --iterations or --suite)
docker exec rocshmem-pr4574 perf-compare --suite heatmap --iterations 5

# Teardown
docker stop rocshmem-pr4574 && docker rm rocshmem-pr4574

CI integration (heatmap artifact generation)

functional_tests/driver.sh accepts --artifact-dir DIR. When run on a heatmap suite with this flag, it generates performance plots into DIR after the tests complete and prints CI ARTIFACT: lines for each output file.

Single-build CI job (per-test latency plots, no baseline comparison)

cd $BUILD_DIR
$ROCSHMEM_DIR/scripts/functional_tests/driver.sh \
  tests/functional_tests/rocshmem_functional_tests \
  heatmap \
  logs-heatmap \
  --artifact-dir $CI_ARTIFACTS_DIR
# Produces: $CI_ARTIFACTS_DIR/per_test/*.png

Two-build CI job (heatmap comparison: branch vs develop)

Set PERF_BASELINE_DIR and PERF_BRANCH_DIR so driver.sh knows where both pre-built binaries live:

cd $BRANCH_BUILD_DIR
PERF_BASELINE_DIR=$DEVELOP_BUILD_DIR \
PERF_BRANCH_DIR=$BRANCH_BUILD_DIR \
  $ROCSHMEM_DIR/scripts/functional_tests/driver.sh \
    tests/functional_tests/rocshmem_functional_tests \
    heatmap \
    logs-heatmap \
    --artifact-dir $CI_ARTIFACTS_DIR
# Produces: $CI_ARTIFACTS_DIR/heatmap_summary.png + heatmap_data.csv + per_test/

Alternatively, call run_perf_compare.sh directly which handles the full build+run+compare pipeline:

$ROCSHMEM_DIR/scripts/functional_tests/run_perf_compare.sh \
  --suite heatmap \
  --iterations 3 \
  --outdir $CI_ARTIFACTS_DIR

Reference: all run_perf_compare.sh options

Option	Default	Description
--iterations N	10	Iterations per config
--suite SUITE	heatmap	Test suite (heatmap, all, rma, …)
--build-config CFG	all_backends	Build config under scripts/build_configs/
--cmake-args ARGS	—	Extra cmake args for all builds
--branch-args ARGS	—	Extra cmake args for branch build only
--variant-args SPEC	—	Additional variant (repeatable)
--pr NUM	—	Compare GitHub PR NUM vs develop
--base-branch NAME	origin/develop	Branch to compare against
--baseline-dir PATH	auto	Pre-built baseline directory
--branch-dir PATH	auto	Pre-built branch directory
--skip-build	—	Skip all builds
--skip-baseline / --skip-develop	—	Skip baseline build and test runs (reuse existing logs)
--skip-branch	—	Skip branch/PR build and test runs (reuse existing logs)
--outdir DIR	auto	Output directory for plots