TheRock/rocm-systems/experimental/python/perfxpert/docs/architecture/agent-hierarchy.md

# Agent Hierarchy

PerfXpert's agentic runtime (v0.2.0+) is a three-tier hierarchy of
OpenAI Agents SDK agents plus deterministic middleware. This document
is the authoritative reference for how the agents relate, how the
fence-slice pattern keeps each agent's system prompt narrow, and where
to look in source.

**Each of the 8 agents is callable from two surfaces:**

- **MCP:** `perfxpert_agent_root`, `perfxpert_agent_analysis`,
  `perfxpert_agent_recommendation`, `perfxpert_agent_correctness`,
  `perfxpert_agent_compute_specialist`,
  `perfxpert_agent_memory_specialist`,
  `perfxpert_agent_latency_specialist`,
  `perfxpert_agent_diff_specialist` (READ_ONLY, auto-registered
  from `perfxpert.tools.agents.*`).
- **Python API:** `perfxpert.api.agent_root(...)`,
  `perfxpert.api.agent_analysis(...)`, etc. — **1:1 mirror** of the
  MCP tools (same function, same schema).

The two surfaces resolve to the same underlying Python function; the
MCP registration is a thin wrapper around the `perfxpert.api.*`
callable. Agents are reachable from any entry point — the Root is NOT
the only way into the hierarchy. Backend TUIs (Claude Code, Gemini
CLI, Codex CLI, opencode) pick whichever agent matches the user's
intent based on the `AGENTS.md` reference.

Cross-links:
- [Gate cascade](gate-cascade.md) — the 5 deterministic correctness
  gates that sit between agents
- [MCP server](../integration/mcp-server.md) — how READ_ONLY tools are
  re-exposed to external clients
- [Python API](../guides/python-api.md) — in-process embedding with
  the same schemas as the MCP tools
- [Agentic mode guide](../guides/agentic-mode.md) — end-user provider
  ladder and air-gap behavior

## Three tiers

```mermaid
flowchart TD
  classDef root fill:#e8f3ff,stroke:#1664ad,color:#0f2f4a,stroke-width:1px
  classDef tier1 fill:#fff4d8,stroke:#a66a00,color:#4a3100,stroke-width:1px
  classDef tier2 fill:#eaf8ef,stroke:#227343,color:#143b26,stroke-width:1px
  classDef brain fill:#fff8c7,stroke:#9a6b00,color:#3d2b00,stroke-width:2px

  subgraph brainSection["🧠 PerfXpert agent brain"]
    root["Root<br/>Tier 0: intent classify"]
    analysis["Analysis<br/>Tier 1: read traces + artifacts"]
    correctness["Correctness<br/>Tier 1: verdicts and gate summaries"]
    recommendation["Recommendation<br/>Tier 1: optimization strategy"]
  end

  compute["Compute specialist<br/>Tier 2"]
  memory["Memory specialist<br/>Tier 2"]
  latency["Latency specialist<br/>Tier 2"]
  diff["Diff specialist<br/>Tier 2"]

  root --> analysis
  root --> correctness
  root --> recommendation
  analysis --> compute
  analysis --> memory
  analysis --> latency
  analysis --> diff
  recommendation --> compute
  recommendation --> memory
  recommendation --> latency
  recommendation --> diff

  class root,analysis,correctness,recommendation brain
  class compute,memory,latency,diff tier2
```

- **Tier 0 (Root)** — classifies the user's natural-language query
  into a routing decision (profile, analyze, recommend, diagnose).
  Owns no tools; its only job is handoff selection.
- **Tier 1 (Layer-1 agents)** — three siblings:
  - **Analysis** — reads trace/source artifacts via READ_ONLY tools,
    emits a structured finding (primary bottleneck, metrics,
    kernel_runtimes).
  - **Correctness** — receives a `GateVerdict` from the gate-cascade
    middleware and narrates it (never runs gates itself). The tier-0
    diagram above shows Correctness as a sibling of the other two
    tier-1 agents, but the data flow is: Recommendation proposes an
    edit → middleware runs the gate cascade → the resulting
    `GateVerdict` is handed to Correctness. See
    [gate-cascade.md](gate-cascade.md).
  - **Recommendation** — converts findings into proposed code changes,
    delegates to specialists when needed.
- **Tier 2 (Specialists)** — four narrow experts invoked by
  Recommendation (or directly from a TUI backend) when the user's
  intent falls in their domain:
  - `compute_specialist` — VALU occupancy, waves-per-EU, VGPR pressure.
    Also consumes `kernel_fusion.find_fusion_candidates` to surface
    adjacent-short-kernel fusion recipes (elementwise+elementwise,
    GEMM+bias+act, ...) drawn from `knowledge/fusion_patterns.yaml`.
    MFMA vs VALU ratio classifier (`mfma_vs_valu_ratio_classification`
    in `knowledge/sol_metrics.yaml`) derived from existing `metrics.*`
    helpers. Attention-scope patterns (Flash / naive SDPA / KV-cache
    amplification) from `knowledge/attention_patterns.yaml`. Pragma
    suggestions (allowlisted LLVM loop hints) gated behind `--advanced`.
  - `memory_specialist` — HBM bandwidth, cache miss rate, LDS conflicts.
    Binds `unified_memory.analyze_paging` for MI300X cross-die + paging
    detection. Multi-level bandwidth chain (HBM → L2 → L1 → LDS) with
    per-level classifier + recommendation in
    `knowledge/memory_patterns.yaml`. Attaches change-impact
    predictions to each surfaced technique via
    `predict_impact.predict_change_impact`.
  - `latency_specialist` — dependency chains, kernel launch overhead,
    async-stream gaps.
    Binds `dependency_graph.reconstruct_dag` for critical-path +
    GPU-bubble attribution. Binds `rccl_analysis.analyze_collectives`
    + `interconnect.lookup_peaks` for per-collective bus-bandwidth +
    NIC efficiency ratios. Reads `gpu_runtime_monitor.*` outputs from
    the user-supplied `PERFXPERT_GPU_MONITOR_LOG` to flag thermal /
    power throttle (MCP surface only — not in the cap-5 allowlist
    because thermal analysis is diagnostic / out-of-band).
  - `diff_specialist` — run-to-run comparison; wraps
    `trace_diff.diff_runs` + `regression.compare_runs` +
    `roofline.classify` and returns an
    `improved` / `regressed` / `neutral` verdict with per-kernel deltas.

## Fence-slice pattern

Each agent's system prompt lives in `perfxpert/agents/fence/<agent>.md`
and is loaded verbatim at runtime — no runtime string concatenation of
prose. Slices are enforced at ≤ 400 lines by CI.

Slice files (one per agent, plus `always.md` loaded into every agent):

```
perfxpert/agents/fence/
├── always.md              # safety preamble loaded into every agent
├── root.md                # tier 0
├── analysis.md            # tier 1
├── correctness.md         # tier 1
├── recommendation.md      # tier 1
├── compute_specialist.md  # tier 2
├── memory_specialist.md   # tier 2
├── latency_specialist.md  # tier 2
└── diff_specialist.md     # tier 2
```

Why fence-slicing:

1. **Scope containment** — each agent only sees rules relevant to
   its job. Cross-agent leakage is impossible because the other slices
   are never loaded into that agent's context.
2. **Length bound** — the 400-line cap forces focused guidance. Content
   that grows past the cap is a signal to split agents or move facts
   into `perfxpert/knowledge/*.yaml`.
3. **Reviewability** — every slice change shows as a small diff. The
   prior monolithic reference guide (removed in Phase 7.1) made
   cross-agent behavior hard to reason about in a single review pass.

## Invoking the hierarchy

From Python, you interact with the hierarchy through the session handle
returned by `build_session()`:

```python
from perfxpert.agents.runtime import build_session
from perfxpert.agents.schemas import RootInput

session = build_session(airgap=True)  # or provider='anthropic'
output = session.run_root(RootInput(
    user_query="Identify the hot kernel and suggest a first fix.",
    airgap=True,
))
print(output.primary_bottleneck)
print(output.recommendations)
```

The session also exposes `run_analysis`, `run_correctness`, and
`run_recommendation` for direct tier-1 invocation — useful for
integrations that already know the routing decision.

## Where to find each piece in source

| Concern | Path |
|---------|------|
| Session handle + agent runner | `perfxpert/agents/runtime.py` |
| Schemas (Root/Analysis/Correctness/Recommendation I/O) | `perfxpert/agents/schemas.py` |
| Per-agent definitions | `perfxpert/agents/{root,analysis,correctness,recommendation}.py` |
| Specialists | `perfxpert/agents/{compute,memory,latency,diff}_specialist.py` |
| Fence slices (system prompts) | `perfxpert/agents/fence/*.md` |
| Agent framework + framework-level helpers | `perfxpert/agents/framework.py` |
| Gate cascade (middleware between agents) | `perfxpert/runtime/gate_cascade.py` |
| Recursion guard (prevents opencode→opencode loops) | `perfxpert/runtime/recursion_guard.py` — `ensure_not_recursive` is also re-exported from `perfxpert.runtime` and imported from there by `agents/runtime.py` |
| Intent classifier (deterministic pre-route) | `perfxpert/runtime/intent_classifier.py` |

## Adding a new agent

The contributor walkthrough is in
[contributing/agents.md](../contributing/agents.md). In brief:

1. Create `perfxpert/agents/<name>.py` with an `AgentSpec` + handoff
   dataclass.
2. Add the fence slice `perfxpert/agents/fence/<name>.md` (≤ 400 lines).
3. Wire it into the parent agent's tools list (Recommendation →
   specialist allowlist; Root → tier-1 allowlist).
4. Add tests under `tests/test_agents/test_<name>.py`.
5. Update this page's diagram.

Every agent addition is a small diff: one Python module, one fence
slice, two tests. No architectural review required for agents that fit
within an existing tier.