TheRock/rocm-libraries/shared/stinkytofu/docs/developer/cluster-barrier.md

Insert Cluster Barrier Pass

createInsertClusterBarrierPass builds a pass that inserts cluster-barrier instructions at five well-defined rules covering the main and tail loops. This document describes each rule, the emitted code shapes, and the pass parameters.

The pass is created via:

STINKYTOFU_EXPORT std::unique_ptr<Pass> createInsertClusterBarrierPass(
    bool isKernelScope = true,
    int  pgrValue      = 1,
    int  plrValue      = 1);

Overview

Rules are numbered in kernel-execution order -- the rule with the lowest number is the first to fire when the kernel runs.

The cluster handshake uses signal/wait pairs at two scopes:

• Workgroup scope: s_barrier_signal -1 / s_barrier_wait -1
• Cluster scope: s_barrier_signal -3 / s_barrier_wait -3

<HASH> in the emitted labels is a fresh 16-character alphanumeric identifier generated per insertion. Only the first wave (WaveIdx == 0) executes the signal; the other waves fall through to the label.

Idempotency: each rule has its own skip check, so re-running the pass is a no-op when the handshake is already present.

---

Rule 1 -- Post-GSU==1 signal-only

Signal-only (no leading cluster wait), emitted immediately after each label_GSU_1: label (Tensile's post-GSU==1-guard label), wrapped in an outer LoopCounterL != 0 gate so the cluster-barrier signal only fires on non-zero iterations.

A workgroup-scope s_barrier_signal -1 / s_barrier_wait -1 pair sits inside the outer LCL skip region (and before the inner WaveIdx gate) so every wave in the workgroup has reached the post-GSU==1 join before any wave issues the cluster signal:

    s_cmp_eq_u32 s[sgprLoopCounterL], 0
    s_cbranch_scc1 label_skipCBPreSignal_LCL_<HASH_OUTER>
    s_barrier_signal -1                                          // workgroup signal
    s_barrier_wait -1                                            // workgroup wait
    s_cmp_eq_u32 s[sgprWaveIdx], 0
    s_cbranch_scc0 label_skipCBPreSignal_<HASH_INNER>
    s_barrier_signal -3
  label_skipCBPreSignal_<HASH_INNER>:
  label_skipCBPreSignal_LCL_<HASH_OUTER>:

---

Rule 2 -- First kernel load wait (kernel scope only)

A single s_barrier_wait -3 immediately before the first tensor_load_to_lds of the whole kernel.

---

Rule 3 -- LDS-publication handshake (currently disabled)

Rule 3 is currently disabled via the kRule3Enabled master switch in the .cpp. The description below is retained for when it is re-enabled.

A LoopCounterL-gated signal-only handshake at the LDS publication point that precedes label_openLoopL:. It has the same shape as Rule 1 but with the outer gate set to s_cmp_le_u32 s[sgprLoopCounterL], pgrValue (the cbranch skips the signal when there are too few iterations left, so the producer is not needed).

The gate mirrors Tensile's own s_cmp_le_u32 LCL, pgrValue / s_cbranch_scc1 LoopEndL loop-entry guard, so the cluster signal is suppressed on the exact same control-flow paths where the corresponding s_barrier_wait -3 inside the unrolled loop body is skipped -- keeping signal -3 / wait -3 paired everywhere.

Two anchor modes (backward scan from label_openLoopL:)

(a) Publication point already exists (typical for PrefetchLocalRead > 0 schedules)

An s_barrier_wait -1 is already present. Anchor at the successor of that wait; no new workgroup sync is synthesized. The scan stops as soon as a tensor_load_to_lds is reached: that instruction marks the prefetch section, before any workgroup sync could sit, so an earlier workgroup wait would be unrelated. Defers to Rule 4 if the same wait would also be a Rule-4 trigger.

(b) No publication point (typical for PrefetchLocalRead == 0 schedules)

No s_barrier_wait -1 between the prefetch tail and label_openLoopL: (the prologue has no local-read preamble barrier). Only active when plrValue == 0; anchor at the label and synthesize an s_barrier_signal -1 / s_barrier_wait -1 pair inside the LCL skip region (between the outer LCL skip-branch and the inner WaveIdx gate) so the workgroup sync sits on the same control-flow path as the cluster signal -- both are bypassed together on the LCL <= pgrValue skip path (matching Tensile's loop-entry guard: when the unrolled loop body is skipped, the LDS reads inside it are skipped too, so no LDS publication is needed). Emitted shape immediately before the label_openLoopL: label:

    s_cmp_le_u32 s[sgprLoopCounterL], <pgrValue>          // outer LCL gate
    s_cbranch_scc1 label_skipCBPreSignal_LCL_<HASH_OUTER> // skip when LCL <= pgr
    s_barrier_signal -1                                  // workgroup signal
    s_barrier_wait -1                                    // workgroup sync
    s_cmp_eq_u32 s[sgprWaveIdx], 0                        // inner wave gate
    s_cbranch_scc0 label_skipCBPreSignal_<HASH_INNER>
    s_barrier_signal -3
  label_skipCBPreSignal_<HASH_INNER>:
  label_skipCBPreSignal_LCL_<HASH_OUTER>:

Internal control-flow labels inside the prefetch prologue (e.g. label_skipPGR2_*) do not match label_openLoopL by exact-name comparison and are walked through.

Section-level idempotency: the backward scan also flags whether a cluster-scope signal/wait already sits in the section; if so (e.g. a prior pass run already emitted Rule 3), Rule 3 self-disables. Unlike a TEXTBLOCK anchor, the label/instruction-based scan survives ScopeAdaptor::moveIRToBlock, so Rule 3 keeps working whenever Gfx1250Backend::buildGfx1250Pipeline runs this pass at kernel scope when moduleOptions.ClusterBarrier == true.

---

Rule 4 -- Cluster handshake before loop loads

A cluster handshake after each workgroup-scope wait that precedes a tensor_load_to_lds. For every load in a label-/branch-delimited segment, the pass walks backward to the nearest preceding s_barrier_wait -1; triggers are deduplicated by identity so multiple loads sharing the same anchor wait yield exactly one handshake.

The emission is selected by the kRule4ForceUngatedSignalMode master switch in the .cpp (default on = mode (c)):

(c) always-ungated mode (active) -- for every trigger emit a WaveIdx-gated s_barrier_signal -3 then a bare s_barrier_wait -3; the cluster signal is never wrapped in an LCL skip branch. findLiveLoopCounterLCmpUpstream is still consulted: if SIA hoisted a live loop-exit s_cmp_eq LCL, imm whose SCC a downstream s_cbranch_scc0 LoopBeginL consumes, a clone of it is re-emitted after the bare wait (which has no SCC side effect) to restore that SCC.

When the switch is off, two fallback modes are selected by findLiveLoopCounterLCmpUpstream:

(a) inherited-SCC mode -- when SIA (typically ScheduleIterAlg=4) hoists the loop-exit s_cmp_eq_{u32,i32} LCL, imm above the anchor, a downstream cbranch still consumes its SCC. Emit an ungated leading s_barrier_wait -3 first, then the inherited-SCC signal block (the signal is single-iter skipped via the inherited SCC; a clone of the upstream cmp is re-emitted between the inner and outer skip labels to rebuild SCC for that downstream cbranch).

(b) drain-gated mode -- when no such upstream cmp is live, gate the handshake with ASYMMETRIC LCL thresholds: skip the WAIT at LCL <= pgrValue and the SIGNAL one stage earlier at LCL <= pgrValue+1 (both lowered by any hoisted LCL pre-decrement). The drain iterations -- where the paired tensor_load_to_lds is disabled -- drop the handshake while keeping signal -3 / wait -3 balanced.

Shape (mode (c); the <clone of upstream LCL cmp> line is emitted only when a live upstream cmp exists):

    s_cmp_eq_u32 s[sgprWaveIdx], 0                               // inner wave gate
    s_cbranch_scc0 label_skipCBPreSignal_<HASH_INNER>
    s_barrier_signal -3
  label_skipCBPreSignal_<HASH_INNER>:
    s_barrier_wait -3                                            // bare cluster wait
    <clone of upstream LCL cmp>                                  // restore SCC (if any)

---

Rule 5 -- Tail-loop cluster handshake (kernel scope only, paired)

Anchors on the first tensor_load_to_lds that follows the /* Tail Loop */ TEXTBLOCK marker. The wait and signal are emitted at two distinct sites because the load and the preceding workgroup wait sit in different label/branch-delimited segments (the tail TDM-reset block between them is not synchronization-critical, so collapsing both into a single site would unnecessarily serialize the cluster).

• 5a -- signal-only handshake (no LoopCounterL gate) immediately after the workgroup-scope wait (s_barrier_wait -1) that precedes the tail load (searched backward from the load, bounded by the /* Tail Loop */ marker). Defers to Rule 4 if that wait is already a Rule-4 trigger.
• 5b -- a single s_barrier_wait -3 immediately before the tail tensor_load itself.

Each half has its own idempotency check so re-runs are no-ops.

---

Parameters

isKernelScope (default true)

Must be true for the GFX1250 backend pipeline (whole-kernel insertion). When false, the pass is intended for region-scoped invocation via createKernelToRegionsPassAdaptor (not used by the backend today). Rule 2 only fires when this is true because the "first tensor_load of the whole kernel" anchor is meaningful only at kernel scope.

pgrValue (default 1, i.e. PrefetchGlobalRead=1)

Tensile's PrefetchGlobalRead setting. It is consulted only by Rule 4's drain-gated fallback mode (b) (the LCL <= pgrValue / LCL <= pgrValue+1 thresholds), i.e. when kRule4ForceUngatedSignalMode is off. The active mode (c) does not use it, and Rule 3 (whose LoopCounterL <= pgrValue gate would also use it) is disabled via kRule3Enabled. It is retained in the signature both for mode (b)'s drain gate and so the Rule 3 gate can be reinstated without an API change.

plrValue (default 1)

Tensile's PrefetchLocalRead setting. It only takes effect while Rule 3 is enabled (see kRule3Enabled), where it selects Rule 3's anchor mode (b): when plrValue == 0 and the backward scan from label_openLoopL: finds no s_barrier_wait -1 before reaching the prefetch boundary (tensor_load_to_lds), the rule synthesizes the missing publication point (workgroup s_barrier_signal -1 / s_barrier_wait -1) followed by the same LCL <= pgrValue gated cluster signal as anchor mode (a). Any non-zero value disables mode (b) (default).

---

Analysis invalidation

This pass mutates the CFG (new branches and a new label), so dependent CFG / dominance analyses are invalidated.

---

See Also

• Architecture Overview -- system architecture and pass pipeline
• src/transforms/asm/InsertClusterBarrierPass.cpp -- implementation
• include/stinkytofu/transforms/asm/InsertClusterBarrierPass.hpp -- public API