TheRock/rocm-systems/emulation/README.md

ROCM Emulation

work board

Components

• Rocjitsu — JIT-based AMD GPU emulator. The core engine that interprets/translates AMDGPU ISA and models GPU blocks so unmodified ROCm apps can run without a physical GPU.
  • KMD Interposer — Intercepts kernel-mode driver calls and routes them to the simulated KMD. Per-OS backends in linux/ and windows/.
  • Config Layer — Loads and composes topology/profile configs (e.g. MI350X) on top of the modular VM. See config_loader.cpp and checkpoint.cpp.
  • VM Layer — Virtual machine modeling GPU blocks, SoC, and thread context; drives instruction execution. Plugin harness via execution_plugin.h. Per-arch backends in amdgpu/ and risc_v/. Design notes: DESIGN.md.
  • Code Layer — Code object APIs: ELF parse/patch, basic blocks, kernel descriptors, executables. See amdgpu_code_object.cpp, basic_block.cpp, executable.cpp.
    • DBT — Dynamic binary translation: encoding and semantic translation.
    • Patch / DBI — Patch points and (planned) instrumentation, spill/restore.
  • Analysis Layer — Register liveness, def-use chains, hazard scheduling, lane permutation. See liveness.cpp and def_use_chain.cpp.
  • ISA Layer — Per-ISA decoders, operand types, machine instruction structs, encoding formats (CDNA1–4, RDNA1–4, RDNA3.5). Entry points: decoder.cpp, rj_decode.cpp; per-arch tables in arch/.
  • simdojo — Decoupled parallel discrete-event simulation engine that orchestrates composable models. See DESIGN.md and src/ (simulation.cpp, topology.cpp, component.cpp).
  • amdisa — Python codegen that consumes AMD machine-readable ISA specs to auto-generate decoders, semantics, encoding/legalization tables, and cross-ISA translators. Key modules: parser.py, semantics.py, encoding_translator_codegen.py, legalization_codegen.py, cross_isa.py, codegen/.
  • rocjitsu_vllm — Python integration glue for running vLLM on top of rocjitsu.
  • configs, schemas, scripts, tests — Topology/profile configs, schema definitions, build/dev scripts, and test suites.
• Mirage — iOS-simulator-inspired UX and CLI that drives rocjitsu (named profiles, persistent sessions, containers, topology editor, session dashboard, interactive terminal). A Rust workspace whose top-level mirage binary composes the crates below. See the Mirage section for the architecture.
  • core — Shared library: profile/session/topology/agent stores, the control-plane trait, KMD/library discovery, and container env wiring. See discovery.rs, profile.rs, ctl.rs, container.rs.
  • ctl — User-facing control plane: parses and dispatches every mirage subcommand (profile/topology/agent/session/exec/run/…). Entry point lib.rs.
  • host — Per-session host process that boots a session, brings up containers, hosts the emulator daemon, and runs/streams executions across nodes and ranks. See lib.rs and main.rs.
  • container — Drives the docker/podman CLI to realise the containerised parts of a session: image build/pull, networks, node launch. See lib.rs.
  • builtin — Built-in agents and topologies (e.g. MI300X/MI350X) preloaded into the user config. See agents.rs and profiles.rs.
  • Emulator backends — pluggable engines registered via inventory, selected per profile by kind:
    • rocjitsu — rocjitsu integration: synthesizes the sim config, hosts the in-process emulator daemon, and the rocjitsu-dbt translation backend. See lib.rs, daemon.rs, dbt.rs.
    • rocjitsu_sys — Runtime (dlopen) FFI bindings to the rocjitsu VM C API (rj_vm_*). See lib.rs.
    • hotswap — HotSwap backend: load-time ISA rewriter that runs a guest-built workload on the host GPU. See lib.rs.
    • noop — Pass-through backend that runs the workload directly with no GPU emulation. See lib.rs.
  • daemon — Optional HTTP/WebSocket server exposing the control plane as a REST API (off by default). See api.rs and server.rs.
  • dashboard — React + Vite single-page app (topology editor, session dashboard, interactive terminals) bundled into the daemon. Source under web/src/.
  • scripts, tests, docs, cmake — Build/dev and Dockerized-build scripts, end-to-end and ML test fixtures, design docs, and the CMake wrapper over cargo.

Rocjitsu

The heart and core of the emulation effort.

flowchart TB
    subgraph APPS["Applications & Frameworks (unmodified)"]
        A1["PyTorch"]
        A2["vLLM"]
        A3["rocBLAS"]
        A4["HIP kernels"]
    end

    subgraph ROCM["ROCm Stack (unmodified)"]
        R1["HIP<br/>(High-level GPU API)"]
        R2["HSA<br/>(Heterogeneous System Architecture)"]
        R3["ROCR<br/>(ROCm Runtime)"]
        R1 --> R2 --> R3
    end
    A1 --> R1
    A2 --> R1
    A3 --> R1
    A4 --> R1

    subgraph CORE["Rocjitsu"]

        subgraph KMD["KMD Interposer"]
            K1["Intercepts driver calls, routes to simulated kernel-mode driver.<br/>Runs unmodified apps."]
        end

        subgraph CFG["Config Layer"]
            C1["Compose specific topologies<br/>(e.g., MI350X) on top of<br/>the modular VM."]
        end
        subgraph SIM["simdojo (PDES engine)"]
            direction LR
            S1["Decoupled parallel discrete event simulation engine<br/>orchestrates scalable simulation and composable models"]
            subgraph VM["VM Layer"]
                direction TB
                V1["Virtual machine models GPU blocks<br/>and context, drives instruction execution."]
                subgraph PLUGINS["Plugins"]
                    direction TB
                    P1["loading of tools"]
                    P2["Hazard detection"]
                    PM["Metrics"]
                    P3["Timing Model"]
                    P1 --> P2 --> PM
                    P1 --> P3 --> PM
                end
            end
            subgraph CODE["Code Layer"]
                direction TB
                subgraph CODEAPI["Code Object APIs"]
                    CO1["ELF parse/patch"]
                    CO2["Basic blocks"]
                    CO3["Kernel descriptors"]
                end
                subgraph DBT["DBT"]
                    D1["Encoding translation"]
                    D1b["Semantic translation"]
                end
                subgraph DBI["DBI (planned)"]
                    D2["Instrumentation"]
                    D3["Spill/restore"]
                    D4["Patch points"]
                end
                subgraph ANA["Analysis Layer"]
                    AN1["Register liveness"]
                    AN2["Hazard scheduling"]
                    AN3["Lane permutation"]
                end
                CODEAPI --> DBT --> ANA
                CODEAPI --> DBI --> ANA
            end
            subgraph ISA["ISA Layer"]
                I1["Per-ISA: decoders, execution, operand types, machine instr structs, encoding formats<br/>9 ISAs: CDNA1–4, RDNA1–4, RDNA3.5"]
                subgraph AMDISA["amdisa (Python codegen)"]
                    AM1["Leverages the AMD machine-readable ISA specs to auto-generate<br/>ISA description and semantics for simulation, instrumentation, and translation"]
                end
            end
            CFG --> VM --> CODE --> ISA
        end
    end


    R3 --> KMD --> SIM

    classDef apps fill:#e8eef7,stroke:#5b7aa8
    classDef kmd fill:#f3e2ec,stroke:#a8567a
    classDef cfg fill:#eeeae0,stroke:#8a7a55
    classDef vm fill:#e6e0f0,stroke:#6b5ba8
    classDef code fill:#f7dede,stroke:#a85b5b
    classDef ana fill:#dceee6,stroke:#4f8a72
    classDef isa fill:#e3efd9,stroke:#6f9a52
    classDef sim fill:#f7e6cf,stroke:#b58440
    classDef amdisa fill:#d9ebe5,stroke:#3f8a78

    class APPS,ROCM apps
    class KMD kmd
    class CFG cfg
    class VM vm
    class CODE,DBT,DBI code
    class ANA ana
    class ISA isa
    class SIM sim
    class AMDISA amdisa

Mirage

An iOS-simulator-inspired UX and CLI that makes rocjitsu easy to drive. Where rocjitsu is the raw emulation engine, Mirage is the control plane around it: it turns a one-off LD_PRELOAD invocation into a managed, reproducible workflow with named configs, persistent sessions, containers, and a dashboard.

Mirage exists to:

• Drive the emulator. A single mirage run boots an emulated GPU and launches your unmodified ROCm workload on it — Mirage discovers the rocjitsu KMD interposer, synthesizes the simulation config from your profile, hosts the emulator daemon, and injects the right environment so the app "sees" the virtual hardware.
• Support complex topologies. Describe multi-GPU, multi-node systems (e.g. MI300X/MI350X/MI450X) declaratively. The topology editor and --num-nodes / --gpus-per-node / --nproc_per_node flags let you stand up large virtual clusters, and Mirage wires up rendezvous (MASTER_ADDR, WORLD_SIZE, RANK, per-node NCCL_HOSTID, …) so distributed frameworks like torchrun and RCCL run unchanged.
• Support containers and reproducible runs. A profile is a self-contained, on-disk description of a workload. Executions run inside containers with pinned images so a run can be replayed identically on another machine, and Mirage handles image bring-up, mounts, and host/glibc compatibility automatically.
• Make it easier to debug and run. Persistent sessions, a session dashboard with live metrics, interactive terminals into running sessions, and streamed build/boot logs replace the manual dance of preloading libraries and hand-editing JSON configs.

flowchart TB
    subgraph MIRAGE["Mirage"]
        subgraph UI["UI"]
            TOPOE["Topology Editor"]
            TERM["Interactive Terminal"]
            DASHBOARD["Session Dashboard"]
        end

        TOPO["Topology"]

        SIM["Emulator"]
        PLUGINS["Plugins"]
        TOPO --> PROFILE
        SIM --> PROFILE
        PLUGINS --> PROFILE
        PROFILE["Profile"]
        SESSION["Session"]
        METRICS["Metrics"]
        SESSION --> METRICS
        METRICS --> DASHBOARD
        EXECUTION["Execution "]
        PROFILE --> SESSION --> EXECUTION
        SESSION --> DASHBOARD
        CONTAINERS["Containers"]
        EXECUTION --> CONTAINERS
        EXECUTION --> SIM
        EXECUTION --> TERM
        TERM --> EXECUTION
        SIM --> METRICS
        TOPOE --> TOPO
    end

Concepts

Mirage models a workload as a small set of layered, persistent objects:

term	definition
profile	a config: emulator + topology + container/image + plugins
session	a booted, running profile (an emulated machine)
execution	running a program on a session
workload	a profile + execution