derek.holloway
7502bf64a4
llx_latency_bench
Measures T(GPU0 -> GPU1) and one-way bandwidth for four RCCL-style LL
flag/data signaling protocols between two GPUs on the same node, sweeping
message sizes from 8 B to 128 MB.
The send is one-way: GPU0 streams data, GPU1 polls each iteration's flag.
A single 8-byte ACK per iteration from GPU1 back to GPU0 provides the
minimum backpressure needed to keep GPU0 from overwriting unread flags.
Per-iter cost ≈ T(A->B) + ~1 µs for the ACK round trip.
Protocols
| Protocol | Line size | Wire efficiency | Ordering mechanism |
|---|---|---|---|
LL |
16 B | 50% | Data and flag packed into each 64-bit store; receiver checks both flag halves |
LL<N>-sc |
N bytes | (N−8)/N | s_waitcnt vmcnt(0) drains the wavefront's data stores before the flag lane writes the flag; receiver uses __any() ballot |
LL<N>-sc+wb |
N bytes | (N−8)/N | Same as sc but adds __threadfence_system() per line between s_waitcnt and the flag store (extra cache writeback) |
LL<N>-tf |
N bytes | (N−8)/N | One __threadfence_system() per iteration between a bulk data pass and a bulk flag pass; receiver polls flag then re-reads data |
Wire efficiencies for supported line sizes:
| LINE_BYTES | Data per line | Wire efficiency |
|---|---|---|
| 64 | 56 B (7 × 8 B) | 87.5% |
| 128 | 120 B (15 × 8 B) | 93.75% |
| 256 | 248 B (31 × 8 B) | 96.9% |
Build
# All three line sizes
make -j
# Single line size
make llx_latency_bench_64
make llx_latency_bench_128
make llx_latency_bench_256
Requires ROCm in $ROCM_PATH (default /opt/rocm). The arch is auto-detected
via --offload-arch=native. The .c source is compiled as HIP via -x hip.
Run
./llx_latency_bench_64 [warmup [iters [timeout_s [max_bytes]]]]
./llx_latency_bench_128 [warmup [iters [timeout_s [max_bytes]]]]
./llx_latency_bench_256 [warmup [iters [timeout_s [max_bytes]]]]
Defaults: warmup=10, iters=100, timeout=30 s, max_bytes=128M.
max_bytes accepts K/M/G suffixes (e.g. 16M, 1G). iters and
warmup must be ≤ 4096 (size of the per-iter ACK counter array).
Output
LL128-sc 16777216 B nlines=139811 nb=64 T(A->B)= 377.65 us BW= 44.426 GB/s [OK]
nlines— number of wire lines transferred per iterationnb— thread blocks launched (scales with message size, capped at 64)T(A->B)— average GPU0 -> GPU1 transfer time per iteration in microseconds (includes the per-iter 8-byte ACK round trip; ≈1 µs of overhead, negligible beyond ~1 KB)BW—data_bytes / T(A->B)in GB/s. Bounded by the per-link unidirectional rate by construction.[OK]/[ERRORS: N]— result of the correctness verification pass
Correctness verification
Between warmup and timing, one iteration runs with a non-zero deterministic payload (a hash of group index, lane, and flag value). The receiver checks every recovered data word and reports any mismatch. This catches stale or zeroed data that a zero-initialized payload would miss.