TheRock/rocm-systems/emulation/mirage/docs/state-layout.md

mirage on-disk state layout

All mirage state lives on disk in standard XDG Base Directory locations. This document is the authoritative reference for the layout and the file formats; tools that interoperate with mirage may read and write these files directly.

Base directories

Resource	Override	XDG fallback	Subpath
Config	MIRAGE_CONFIG	$XDG_CONFIG_HOME	mirage/ (profiles, agents, topologies)
Sessions (runtime)	MIRAGE_RUNTIME	$XDG_RUNTIME_DIR	mirage/session/<id>/...
Persistent state	MIRAGE_STATE	$XDG_STATE_HOME	mirage/

Each MIRAGE_* variable, when set, fully overrides the corresponding directory; otherwise the XDG variable (or its standard default) is used. mirage paths prints the resolved directories. $XDG_RUNTIME_DIR is preferred for sessions because it is per-user, writable only by its owner, and cleared on logout — exactly the lifetime sessions want.

The config directory holds three resource trees:

<config>/mirage/
├── profile/<name>.json     # ProfileDef
├── agent/<name>.json       # AgentDef   (hardware GPU definition)
└── topology/<name>.json    # TopologyDef (rack/node/GPU layout)

Profiles

A profile is a JSON ProfileDef named by its filename:

<config>/mirage/profile/<name>.json

{
  "name": "cdna4",
  "description": "Single-node rocjitsu targeting MI350X.",
  "emulator": {
    "emulator": "rocjitsu",
    "plugins": {},
    "exec_mode": "Functional",
    "options": {},
    "topology": {
      "num_nodes": 1,
      "gpus_per_node": 1,
      "agent": "MI350X"
    }
  }
}

• emulator.topology may be an inline object (as above) or a string naming a topology in <config>/mirage/topology/.
• topology.agent may likewise be an inline object or a string naming an agent in <config>/mirage/agent/.
• A containerised profile additionally carries a containerize object (image, optional provider, and mounts).

Use mirage profile show <name> to print an existing profile.

Sessions

A session is a directory under <runtime>/mirage/session/<id>:

<session>/
├── def.json              # SessionDef (immutable after create)
├── health.json           # SessionHealth (rewritten by the host)
├── node/                 # per-node host runtime state (one dir per rank)
│   └── <rank>/
│       ├── pid           # pid of the node's host process
│       └── host.log      # the node host's stderr (tracing output)
└── exec/
    └── <exec-id>/        # one directory per exec

def.json

A SessionDef:

{
  "id": "s-20260616-191636-f6c1",
  "profile": "cdna4",
  "workdir": "/home/me/work",
  "daemon": false,
  "created_at": "2026-06-16T19:16:36.951820571Z"
}

• profile may be a string (a profile name under <config>/mirage/profile) or an inline ProfileDef object (used when CLI overrides are applied).
• daemon selects out-of-process emulation when true.

health.json

A SessionHealth, rewritten by the host as the session progresses:

{
  "timestamp": "2026-06-16T19:16:36.972372050Z",
  "healthy": true,
  "state": "ready",
  "terminal": false
}

state is one of starting, pulling, ready, degraded, stopping, stopped, error. When terminal=true the host has given up and the session must be discarded. An optional message carries detail (image-pull progress, node bring-up status, crash diagnostics). mirage session wait polls this file until healthy=true or terminal=true.

Host pid and log

The per-session (node 0) host writes its pid to node/0/pid on startup and appends tracing output to node/0/host.log. Each additional node rank has its own node/<rank>/ directory with the same two files. mirage session stop signals the recorded host pids.

Execs

Each exec gets a directory under <session>/exec/<exec-id>/:

<exec>/
├── def.json              # ExecDef (immutable)
├── status.json           # ExecStatus (rewritten as nodes start/finish)
└── node/
    └── <node-id>/
        ├── stdin         # FIFO (named pipe), bridged to the child's PTY
        ├── stdout        # plain file, merged child stdout+stderr (PTY)
        ├── pid           # pid of the child process
        └── exit_code     # exit code, written after the child exits

Exec id format

<exec-id> is e-<n> with a zero-padded counter (e.g. e-000000). Ids are allocated monotonically from the existing entries.

def.json

{
  "timestamp": "2026-06-16T19:31:00.000Z",
  "session": "s-20260616-191636-f6c1",
  "exec": {
    "command": "/bin/sh",
    "args": ["-c", "echo hi"],
    "env": { "EXTRA": "1" },
    "workdir": null
  },
  "worker_exec": null,
  "keep": true
}

The host watches for exec/<id>/def.json directories with no status.json yet; when one appears it creates the per-node structure and spawns the child.

status.json

{
  "started": true,
  "ended": true,
  "exit_code": 0,
  "started_at": "2026-06-16T19:31:00.100Z",
  "ended_at": "2026-06-16T19:31:00.250Z",
  "nodes": {
    "0": { "pid": 110891, "exit_code": 0 }
  }
}

The top-level exit_code summarises across nodes; consult the per-node entries for full detail.

Per-node files

• stdin is a Unix FIFO created with mkfifo(2). Writing to it forwards to the running exec's stdin: printf 'data\n' > <node>/stdin. The CLI exposes this via MirageCtl::session_stdin (used by interactive attach). It is created when the exec starts.
• stdout is a plain file opened O_APPEND; the PTY merges the child's stderr into it, so concurrent readers can tail -f safely.
• pid is written before the child runs.
• exit_code is written atomically after the child exits; its presence marks the child done.

Atomicity guarantees

• All JSON writes are atomic (<path>.tmp.<pid> then rename).
• stdin and stdout exist before the child is spawned, so readers can attach as soon as the per-node directory appears.
• status.json is rewritten many times; each rewrite is atomic.
• The host writes its pid before publishing healthy=true, so any client that sees healthy=true can rely on the recorded pid.

Cleanup

• When keep=false (the default for exec start without --keep, and for run), the host removes the entire <exec>/ directory once exit_code has been written for every node. Attaching to such an exec after it finishes is therefore racy; use --keep if you need its logs afterward.
• mirage session stop removes the entire <session>/ directory after the host exits.
• mirage state purge removes the runtime and state directories (and, with --all, the config directory too).
• $XDG_RUNTIME_DIR is cleared on logout, so leaked sessions do not survive across reboots.