TheRock/rocm-systems/emulation/rocjitsu/docs/architecture.md

# Architecture

rocjitsu is a full-system GPU simulator organized into layered components.
From bottom up:

```
┌─────────────────────────────────────────────┐
│  HIP / ROCR / RCCL  (unmodified binaries)   │
├─────────────────────────────────────────────┤
│  KMD Layer (kmd/linux/)                     │  ┌──────────────────────────┐
│    Interposer ── RemoteDriver ── RPC ───────┼──┤  Daemon (tools/rocjitsu) │
│    SimulatedDriver ── EventState            │  │  Unix socket RPC         │
├─────────────────────────────────────────────┤  └──────────────────────────┘
│  VM Layer (vm/amdgpu/)                      │
│    GPU SOC and component blocks             │
│    GpuMemory (VMID page tables)             │
│    Cache and memory models                  │
├─────────────────────────────────────────────┤
│  ISA Layer (isa/)                           │
│    Decoder ── Instruction ── Execute        │
│    GFX architecture targets                 │
├─────────────────────────────────────────────┤
│  Code Layer (code/)                         │
│    Executable ── AmdGpuCodeObject           │
│    BasicBlock ── DBT (binary translator)    │
├─────────────────────────────────────────────┤
│  Simdojo Engine (simdojo/)                  │
│    PDES ── Components ── Topology           │
│    Events ── Ports ── Links                 │
├─────────────────────────────────────────────┤
│  Config (config/)                           │
│    JSON ── FlatBuffers ── load_config       │
└─────────────────────────────────────────────┘
```

## Components

### Simdojo (`lib/simdojo/`)

Parallel Discrete Event Simulation (PDES) framework. Provides the
component model, topology builder, event queue, and multi-threaded
execution engine with barrier-based LBTS synchronization.

See [simdojo.md](simdojo.md) for the full design.

### VM — GPU Hardware Model (`vm/amdgpu/`)

See [vm-design.md](vm-design.md) for the full hardware model design.
Models the GPU hardware pipeline:

- **CommandProcessor** — Monitors doorbells, fetches AQL packets from
  ring buffers, parses kernel descriptors, dispatches workgroups to CUs.
  Handles SDMA packets (copy, fill, GCR, HDP flush).
- **ComputeUnit** — Executes wavefronts. Manages SGPR/VGPR register
  files, LDS, and scratch memory. Supports functional and cycle-accurate
  modes.
- **CompletionTracker** — Tracks per-dispatch workgroup retirement.
  Fires completion signals in submission order. Writes queue-inactive
  signal on HQD idle.
- **ShaderEngine / XCD / IOD** — Hierarchical GPU topology matching
  real hardware (shader engines contain CU arrays, XCDs contain SEs).
- **GpuMemory** — VRAM model with per-process VMID page tables.
  Supports passthrough mode (GPU VA == host VA) and daemon mode
  (shared memfd mappings).
- **Cache hierarchy** — L1 vector cache, L1 scalar cache, L2 cache,
  memory-side cache. MTYPE-aware (UC, CC, RW).
- **Execution plugins** — Pluggable hooks for race detection, kernel
  logging, and more. See [plugins.md](plugins.md).

### ISA — Instruction Set Architecture (`isa/`)

Instruction decoding and execution for AMD GPU architectures plus
RISC-V. Most files are autogenerated from the MR ISA XML spec by the
`amdisa` codegen pipeline.

- **Decoder** — Decodes variable-length instructions from code objects.
- **Instruction** — Per-encoding instruction structs with typed fields.
- **Execute** — Instruction semantics (shared templates across ISAs
  where possible).
- **Operand resolution** — SGPR, VGPR, literal, inline constant,
  FLAT_SCRATCH encoding.

Hand-written files handle address calculation (`addr_calc_flat.h`),
matrix math execution (`mma_exec.h`), and ISA-specific traits.

See [codegen.md](codegen.md) for the full codegen pipeline and
regeneration commands. See [isa-target-providers.md](isa-target-providers.md)
for target identity, static provider composition, and model-only linkage.

### KMD — Kernel Mode Driver Emulation (`kmd/linux/`)

Emulates the AMDKFD kernel driver via LD_PRELOAD interposition:

- **Interposer** — Intercepts `open`, `close`, `ioctl`, `mmap`,
  `munmap`, `fopen`, `dup2`, `fork` to route `/dev/kfd` operations.
- **SimulatedDriver** — Handles KFD ioctls (create_queue,
  alloc_memory, map_memory, wait_events, etc.). Manages per-process
  state (page tables, doorbells, events).
- **RemoteDriver** — Client-side RPC stub for daemon mode. Forwards
  ioctls over a Unix socket to the daemon's SimulatedDriver.
- **EventState** — KFD event lifecycle (create, set, reset, wait,
  destroy). Signal page shared via memfd between daemon and client.

### Code — Executable Loading & Analysis (`code/`)

- **Executable** — Loads x86 HIP fat binaries, extracts Clang offload
  bundles, parses AMD GPU HSA device ELFs.
- **BasicBlock** — Control-flow basic block construction from decoded
  instruction streams.
- **DBT** — Dynamic Binary Translation between ISA pairs. Uses
  autogenerated legalization tables and encoding translators.
  See [dbt-design.md](dbt-design.md) for the full design.
- **DBI** — Dynamic Binary Instrumentation. Instruments existing
  binaries with probe functions for tracing, software counters,
  and more. See [dbi-design.md](dbi-design.md) for the current state.

#### Loading and inspecting a code object (C API)

```c
rj_code_executable_t *exec = NULL;
rj_code_executable_create("kernels/matmul_naive.o", &exec);

rj_code_object_t *obj = NULL;
rj_code_executable_get_code_object(exec, ROCJITSU_CODE_TARGET_GFX942, 0, &obj);

rj_code_basic_block_list_t *bbs = NULL;
rj_code_basic_block_list_create(obj, ROCJITSU_CODE_TARGET_GFX942, &bbs);

uint32_t num_blocks = rj_code_basic_block_list_size(bbs);
for (uint32_t i = 0; i < num_blocks; ++i) {
    rj_code_basic_block_t *bb = NULL;
    rj_code_basic_block_list_get(bbs, i, &bb);

    for (const rj_code_inst_t *inst = rj_code_basic_block_first_inst(bb);
         inst != NULL; inst = rj_code_inst_next(inst)) {
        char buf[256];
        rj_code_inst_disassemble(inst, buf, sizeof(buf));
        printf("%s\n", buf);
    }

    rj_code_basic_block_destroy(bb);
    rj_code_basic_block_release(bb);
}

rj_code_basic_block_list_destroy(bbs);
rj_code_object_destroy(obj);
rj_code_object_release(obj);
rj_code_executable_destroy(exec);
```

### Analysis (`analysis/`)

Register liveness and def-use chain analysis over GPU kernel CFGs.
Shared by DBT (for temporary register allocation during instruction
expansion) and DBI (for spill slot planning).

- **LivenessAnalysis** — Backward dataflow over `BasicBlock` CFG scoped
  to a single kernel. Provides `live_before(inst)`, `find_free_run()`,
  `find_free_sgpr_pair()` for safe register allocation in injected code.
- **DefUseChain** — Per-instruction def-use relationships including
  implicit operands (e.g., FLAT `saddr` SGPR pairs).

### Config (`config/`)

JSON configuration validated against FlatBuffers schemas. The
`load_config` functions build the full GPU topology (SoC, XCDs, SEs,
CUs, caches, memory) from a single JSON file.

See [configuration.md](configuration.md) for the config format.

### CLI (`tools/rocjitsu/`)

Three execution modes:

- **Local** — In-process simulation via LD_PRELOAD
- **Daemon** — Fork a daemon server, then exec the application
- **Attach** — Connect to a running daemon

See [rocjitsu-cli.md](rocjitsu-cli.md) for the daemon RPC protocol.