TheRock/rocm-systems/projects/rocshmem/scripts/debug/README.md

rocSHMEM Deadlock Analysis Tools

Scripts for identifying and diagnosing deadlocks in rocSHMEM applications using AMD's ROCm debugger (rocgdb).

Files

File	Purpose
rocgdb_deadlock_analysis.py	rocgdb Python script: coalesces GPU wavefront backtraces, identifies rocSHMEM API entry points, and provides deadlock hints
attach_deadlock_analysis.sh	Shell wrapper: finds all running instances of an executable on the local node, attaches to each, saves per-process output, prints a compact summary
mpiexec_deadlock_analysis.sh	Multi-node wrapper: deploys attach_deadlock_analysis.sh to every node via mpiexec, then runs cross-rank coalescing
cross_rank_deadlock_analysis.py	Cross-rank coalescer: reads per-rank output files, groups identical backtrace patterns across ranks, highlights patterns uncommon among ranks
deadlock_test.cc	Intentional-deadlock test program used to validate the scripts

---

Prerequisites

• rocgdb in PATH (provided by ROCm; override path with ROCGDB env var)
• timeout command (standard on Linux)
• Python support in rocgdb (built-in for ROCm 6+)

For meaningful backtraces the application and rocSHMEM library should be built with debug symbols. A RelWithDebInfo build (CMake -DCMAKE_BUILD_TYPE=RelWithDebInfo) gives full inlined call chains while keeping optimized performance. A Release build will still show the outermost non-inlined API frame but not the internal wait loops.

---

Quick Start

# 1. Start your (deadlocked) application
mpirun -np 4 ./my_rocshmem_app &

# 2. Attach to all instances, analyze, save output
./scripts/debug/attach_deadlock_analysis.sh my_rocshmem_app

# Output is written to ./rocshmem_deadlock_analysis_<timestamp>/
# A compact summary is printed to stdout.

---

attach_deadlock_analysis.sh

Usage

attach_deadlock_analysis.sh <executable_name> [--directory <dir>] [--cull[=p1,p2,...]]

Argument	Description
executable_name	Exact process name to search for
--directory <dir>	Directory for per-process output files (default: ./rocshmem_deadlock_analysis_<timestamp>/)
--cull	Cull groups stuck in GPU barriers / gridsync using the built-in default pattern list
--cull=p1,p2,...	Cull groups whose backtrace contains any of the comma-separated substrings

Environment Variables

Variable	Default	Description
ROCGDB	rocgdb	Path to the rocgdb binary
ROCSHMEM_GDB_TIMEOUT	120	Per-process timeout in seconds
ROCSHMEM_DEADLOCK_COLOR	always	Color mode for output files (see Color Support)

Output Files

Each analyzed process produces a file named:

<output_dir>/pe<PE_RANK>_pid_<PID>.txt

The MPI PE rank is read from the process's OMPI_COMM_WORLD_RANK environment variable. If the variable is not set (non-MPI process or already exited), the rank is shown as unknown.

Stdout Summary

After all processes are analyzed, a compact summary is printed:

=== Attach and Analyze Summary ===
Processes analyzed: 2 / 2

  PE 0     PID 12345   groups=?    (no GPU threads found)
  PE 1     PID 12346   groups=1    Waiting for remote PE memory update...

Full output in: ./rocshmem_deadlock_analysis_20260416_143022/

• groups — number of distinct backtrace groups found in that process
• dominant — the most frequent [HINT] line across all groups, or a status string if no deadlock was detected

Error Handling

Situation	Behavior
Process exits before attach	Logged as SKIP, counted as failure
rocgdb exceeds timeout	Logged as TIMEOUT; output file kept for inspection
rocgdb exits non-zero but produced analysis	Still counted as success
rocgdb not found in PATH	Fatal error with hint to set ROCGDB

---

rocgdb_deadlock_analysis.py

The Python script can be used in three modes.

Mode 1: Batch attach (used by attach_deadlock_analysis.sh)

Attach rocgdb to a running process and analyze immediately:

ROCSHMEM_DEADLOCK_AUTO_ANALYZE=1 \
    rocgdb -batch -p <pid> -x scripts/debug/rocgdb_deadlock_analysis.py

The ROCSHMEM_DEADLOCK_AUTO_ANALYZE=1 variable tells the script to run the analysis as soon as the process is stopped (which happens automatically on attach).

Mode 2: Interactive session

Attach rocgdb manually, then source and run the script:

rocgdb -p <pid>
(rocgdb) source scripts/debug/rocgdb_deadlock_analysis.py
(rocgdb) rocshmem-deadlock-analyze

Write the report to a file:

(rocgdb) rocshmem-deadlock-analyze /tmp/analysis.txt

Force colored output in the file:

(rocgdb) rocshmem-deadlock-analyze --color /tmp/analysis.txt

Mode 3: Launch mode

Run the application under rocgdb from the start; analysis triggers on first stop (e.g. pkill --signal SIGINT my_rocshmem_app), or when the program crashes:

mpirun -n 2 -x ROCSHMEM_DEADLOCK_AUTO_ANALYZE=1 rocgdb -batch \
    -x scripts/debug/rocgdb_deadlock_analysis.py \
    -ex run \
    --args ./my_rocshmem_app arg1 arg2

### GDB Command Reference

After sourcing the script, the `rocshmem-deadlock-analyze` command is available:

(rocgdb) rocshmem-deadlock-analyze [--color|--no-color] [--cull[=pat,...]] [--check-lanes] [output_file]

| Option | Description |
|---|---|
| `--color` | Force ANSI color codes on, even when writing to a file |
| `--no-color` | Disable color codes, even on a TTY |
| `--cull` | Cull groups stuck in GPU barriers / gridsync using the built-in default pattern list |
| `--cull=p1,p2,...` | Cull groups whose backtrace contains any of the comma-separated substrings |
| `--check-lanes` | Enable lane-level parameter mismatch detection for `_wg` and `_wave` collective calls (slow; see below) |
| `--stats` | Append a `=== rocSHMEM Stats ===` section with non-zero API call counters (requires `-DPROFILE=ON` build) |
| `output_file` | Write report to this file instead of stdout |

#### Cull option

When `--cull` is given without a pattern list, the following substrings are used to remove groups
that are simply waiting at a non-rocSHMEM GPU synchronization primitive (and are therefore not
rocSHMEM deadlocks):

| Pattern | Covers |
|---|---|
| `__syncthreads` | HIP workgroup barrier and reduction variants (`_count`, `_and`, `_or`) |
| `__builtin_amdgcn_s_barrier` | Direct AMDGCN scalar barrier intrinsic |
| `__work_group_barrier` | Workgroup barrier with memory-fence flags |
| `__named_sync` | Named barrier intrinsic |
| `__ockl_grid_sync` | OCKL grid-level synchronization |
| `__ockl_gws_barrier` | OCKL global-workspace barrier |
| `__ockl_multi_grid_sync` | OCKL multi-grid synchronization |
| `cooperative_groups` | HIP cooperative groups (`grid_group`, `thread_block`, `multi_grid_group`) |

Culled groups are not shown in the report; the header prints a count of how many
groups and wavefronts were removed:

Unique backtrace groups: 1 (+ 3 group(s) / 192 wavefront(s) culled — GPU barrier / gridsync)

### Environment Variables

| Variable | Default | Description |
|---|---|---|
| `ROCSHMEM_DEADLOCK_AUTO_ANALYZE` | `0` | Set to `1` to trigger analysis automatically on process stop |
| `ROCSHMEM_DEADLOCK_COLOR` | `always` | Color mode (see [Color Support](#color-support)) |

---

## Output Format

=== rocSHMEM Deadlock Analysis === Process(es): 12346 Total GPU wavefronts analyzed: 64 Unique backtrace groups: 2

--- Group 1 (63 wavefront(s)) --- WFs: (0,0,0)/0,(1,0,0)/0,(2,0,0)/0,... Backtrace: #0 rocshmem::uncached_load (...) at src/assembly.hpp:125 #1 rocshmem::Context::test (...) at src/context_tmpl_device.hpp:432 #2 rocshmem::Context::wait_until (...) at src/context_tmpl_device.hpp:224 #3 rocshmem::IPCContext::internal_direct_barrier (...) at ... ... #7 rocshmem::Context::barrier_all_wg (...) at src/context_device.cpp:151 #8 my_kernel () at my_app.cpp:42 [HINT] Waiting for remote PE memory update (barrier/sync or user rocshmem_wait_until). Check if the remote PE is alive and making progress.

--- Group 2 (1 wavefront(s)) --- WGs: (0,0,0) WFs: 0 Backtrace: #0 rocshmem::acquire_lock (...) at src/gda/mlx5/queue_pair_mlx5.cpp:53 #1 rocshmem::QueuePair::mlx5_post_wqe_rma (...) at ... #2 rocshmem::GDAContext::put (...) at ... #3 rocshmem::rocshmem_ctx_put (...) at ... <<< rocSHMEM API entry #4 my_kernel () at my_app.cpp:38 [rocSHMEM] Stuck in: rocshmem_ctx_put [HINT] Waiting for SQ spinlock held by another wavefront. The lock holder may itself be deadlocked in mlx5_poll_cq_until.

=== Summary === 1 wavefront(s) inside rocSHMEM 63 wavefront(s) outside rocSHMEM

### Reading the Output

**Groups** — Wavefronts with identical backtraces (after stripping runtime-variable
parts like addresses and argument values) are coalesced into a single group.
Each group shows which workgroups (WGs) and wavefronts (WFs) share that call
stack.  Template parameters and source file locations are kept in the key, so
`wait_until<int>` and `wait_until<long>` at different call sites produce
separate groups.

**`<<< rocSHMEM API entry`** — Marks the outermost rocSHMEM public API function
in the backtrace: the call the application made that led to the deadlock.

**`[rocSHMEM] Stuck in:`** — Names the matched API function for easy grepping.

**`[HINT]`** — A short diagnosis of the likely deadlock cause based on the
innermost recognized frame.

The innermost deadlock frame (the actual spinning loop) is printed in bold cyan.

**`=== Collective API Usage Issues ===`** — Appears after the per-group section
when incorrect `_wg` or `_wave` API usage is detected from the backtraces.
Three kinds of violations are reported:

| Issue | Description |
|---|---|
| `[BAD] … non-collective _wg call` | Wavefronts in the same workgroup are at different `_wg` API calls, or some are not in any `_wg` call. All wavefronts in a WG must call the same `_wg` primitive together. |
| `[BAD] … parameter mismatch in rocshmem_*_wg` | All wavefronts in a WG are at the same `_wg` call but with different argument values (e.g., different `pe` or `nelems`). |

`_wave` primitives are collective only within a single wavefront (all 64 lanes
must call with matching parameters). Different wavefronts — even in the same WG —
may independently call any `_wave` function with different parameters; that is valid.
Lane-level divergence within a wavefront is not visible from wavefront-level
backtraces and requires `--check-lanes` to detect.

#### Backend statistics (`--stats`)

When `--stats` is given, the script reads the rocSHMEM API call counters
directly from the stopped process's host memory without calling any function
in the inferior.  The counters live in `Backend::globalStats` (device-side,
written by GPU `atomicAdd`) and `Backend::globalHostStats` (host-side,
written by `std::atomic`).  Both arrays are in `hipHostMalloc`-pinned
memory, accessible to gdb without any cache flush.

Only non-zero counters are printed, grouped by device and host:

=== rocSHMEM Stats === PE 1 (counters reflect activity since process start) Device: put 8192 quiet 64 barrier_all_wg 4 Host: (all zero)

> **Requires `-DPROFILE=ON`** at CMake build time.  Without that flag
> `ROCStats` and `ROCHostStats` are empty `NullStats<N>` structs (no
> storage; all counters read as zero).  The section will note when this
> is the case.

The counters count operations since process start, not since the deadlock.
They show what the application was doing before it got stuck — e.g. a high
`barrier_all_wg` count with a non-zero `put` count suggests the collective
stalled after a put-heavy phase.

#### Lane-level parameter mismatch (`--check-lanes`)

When `--check-lanes` is added to the `rocshmem-deadlock-analyze` command, the
script iterates over the active lanes of every wavefront that is inside a `_wg`
or `_wave` collective call.  It switches rocgdb to each active lane, collects
the per-lane backtrace, and extracts the argument values for the collective
API frame.

If any two active lanes in the same wavefront disagree on a parameter, a
`LANE_PARAM_MISMATCH` issue is reported:

=== Collective API Usage Issues === [BAD] WG (0,0,0) WF 0 — lane parameter mismatch in rocshmem_putmem_wave (_wave contract) Differing parameter(s): pe Lane 0 [0,0,0]: pe=0 Lane 1 [1,0,0]: pe=1 Lane 2 [2,0,0]: pe=0 ...

This check covers both `_wave` (lanes diverging within one wavefront) and `_wg`
(lanes in a wavefront that should present unified parameters to the WG-wide
collective).

> **Performance note:** `--check-lanes` issues up to 64 additional `bt` calls per
> wavefront in a collective call.  On a busy kernel with many wavefronts this can
> be slow (several minutes).  Use it only when wavefront-level analysis has
> already identified a collective call of interest.

### Recognized Deadlock Patterns and Hints

#### mlx5 (Mellanox/ConnectX) backend

| Innermost frame | Hint |
|---|---|
| `mlx5_poll_cq_until` | Waiting for NIC completion (CQ polling). Check if NIC is responsive and if the remote PE is also stuck. |
| `acquire_lock` | Waiting for SQ spinlock held by another wavefront. The lock holder may itself be deadlocked in `mlx5_poll_cq_until`. |
| `mlx5_quiet` | Quiet operation waiting for all outstanding RMA ops to complete. Check NIC health. |

#### bnxt (Broadcom) backend

| Innermost frame | Hint |
|---|---|
| `bnxt_poll_cq_until` | Waiting for NIC completion (CQ polling). Check if the bnxt NIC is responsive and if the remote PE is also stuck. |
| `bnxt_post_wqe_rma` | Waiting for bnxt SQ spinlock held by another wavefront. The lock holder is likely itself deadlocked in `bnxt_poll_cq_until`. |
| `bnxt_quiet` | Quiet operation waiting for all outstanding RMA ops to complete (bnxt). Check NIC health. |

#### ionic (AMD/Pensando) backend

| Innermost frame | Hint |
|---|---|
| `ionic_quiet_internal_ccqe_single` | Waiting for NIC completion in CCQE mode (single-thread path). Check if the ionic NIC is responsive and if the remote PE is also stuck. |
| `ionic_quiet_internal_ccqe` | Waiting for NIC completion in CCQE mode. Check if the ionic NIC is responsive and if the remote PE is also stuck. |
| `ionic_quiet_internal` | Waiting for NIC completion (CQ polling). Check if the ionic NIC is responsive and if the remote PE is also stuck. |
| `spin_lock_acquire_unique` | Waiting for ionic SQ doorbell spinlock (exclusive) held by another wavefront. The lock holder may itself be stuck in `ionic_quiet_internal`. |
| `spin_lock_acquire_shared` | Waiting for ionic CQ spinlock (shared) held by another wavefront. The lock holder may itself be stuck in `ionic_quiet_internal`. |
| `ionic_quiet` | Quiet operation waiting for all outstanding RMA ops to complete (ionic). Check NIC health. |

#### Shared / IPC / user-level waits (all backends)

| Innermost frame | Hint |
|---|---|
| `wait_until_any` | Waiting for any element of a multi-element condition. Check if the remote PE is alive. |
| `wait_until_all` | Waiting for all elements of a multi-element condition. Check if the remote PE is alive. |
| `wait_until_some` | Waiting for some elements of a multi-element condition. Check if the remote PE is alive. |
| `wait_until` | Waiting for remote PE memory update (barrier/sync or user wait). Check if the remote PE is alive. |

---

## Color Support

Color is controlled by the `ROCSHMEM_DEADLOCK_COLOR` environment variable:

| Value | Behavior |
|---|---|
| `always` (default) | Always enable ANSI color codes |
| `auto` | Enable color when the output is a TTY; disable for files |
| `1`, `yes`, `true` | Always enable ANSI color codes |
| `0`, `no`, `false`, `never` | Always disable color codes |

Color can also be forced per-invocation with the `--color` / `--no-color`
flags of the interactive `rocshmem-deadlock-analyze` command.

The shell script's stdout summary is always plain text; ANSI codes are
stripped from the per-process output files before extracting the dominant hint.

### Color Scheme

| Element | Color |
|---|---|
| Section headers (`=== ... ===`) | Bold blue |
| Group headers (`--- Group N ---`) | Bold bright blue |
| Innermost deadlock frame | Bold cyan |
| `<<< rocSHMEM API entry` annotation | Bold green |
| `[rocSHMEM] Stuck in:` line | Bold bright red |
| `[HINT]` line | Bold cyan |
| Stuck-wavefront count in summary | Bold bright red |
| Not-stuck count in summary | Bold green |
| "No GPU threads found" notice | Red |

---

## Building with Debug Symbols

Full call-chain visibility requires debug symbols in the rocSHMEM library and
the application.  Use `RelWithDebInfo` for the best balance of performance and
debuggability:

```bash
# Configure the rocSHMEM build tree
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo <source_dir>
cmake --build . --target rocshmem --parallel 8

# Compile the application with matching flags
hipcc -O2 -g -fgpu-rdc -x hip my_app.cc \
      --offload-arch=<target> \
      -I<rocshmem_include> -I<mpi_include> \
      -c -o my_app.o

hipcc -fgpu-rdc --hip-link my_app.o \
      --offload-arch=<target> \
      <rocshmem_lib>/librocshmem.a <mpi_lib>/libmpi.so \
      -L/opt/rocm/lib -lamdhip64 -lhsa-runtime64 \
      -o my_app

Without debug symbols only the outermost non-inlined frame is visible. The <<< rocSHMEM API entry annotation still works (those functions are marked __attribute__((noinline))), but the [HINT] requires the inner frames (wait_until, mlx5_poll_cq_until, etc.) to be present.

---

Known Limitations

• IPC backend: rocshmem_barrier_all_wg and related functions are fully inlined in the IPC backend even at RelWithDebInfo. The [rocSHMEM] Stuck in: line will not appear, but the [HINT] fires from the inner wait_until frame.

• Release builds: Without debug symbols, internal frames (wait_until, mlx5_poll_cq_until, etc.) are invisible and no [HINT] is generated. The API entry frame is still detected when it is non-inlined.

• Lane-level analysis performance: --check-lanes issues up to 64 bt calls per wavefront in a collective call. On large kernels this can take several minutes.

• Single GPU per process: The script handles one GPU device per MPI rank. Multi-GPU-per-rank configurations are untested.

• GPU thread format: Tested with rocgdb from ROCm 7.2. Earlier ROCm versions use a different thread name format (AMDGPU Thread X.Y (GPU, WG (...), WF (N))); the script includes a regex for this legacy format but it is untested.