TheRock/rocm-systems/emulation/rocjitsu/docs/rocjitsu-cli.md

Rocjitsu CLI and Transport Layer Design

Overview

The rocjitsu CLI is the primary entry point for running applications on the simulated GPU. It operates in two modes:

• Local mode: Sets LD_PRELOAD and execves the target application. The simulator runs in-process via the interposer.
• Daemon mode: Spawns a daemon child process that hosts the simulation engine, then LD_PRELOAD + execves the target. Client processes communicate with the daemon via RPC over a Unix domain socket. GPU memory is shared using memfds passed via SCM_RIGHTS.

Daemon mode supports vLLM's multiprocessing spawn, torchrun, torch.distributed, and NCCL --- workloads where multiple processes share a single simulated GPU.

Command-Line Options

Usage: rocjitsu --config <config.json> [--daemon|--attach] -- <app> [args...]

Option	Description
--config <path>	Path to simulation config JSON (required)
--daemon	Run in daemon mode: fork a daemon process hosting the simulation engine, then launch the application with the interposer. Without -- <app>, runs the daemon server only.
--attach	Attach to a running daemon. The socket path is resolved as $ROCJITSU_RUNTIME_DIR/daemon.sock, then $XDG_RUNTIME_DIR/rocjitsu/daemon.sock, falling back to /tmp/rocjitsu-<uid>/daemon.sock.
--help, -h	Print usage and exit
--version, -v	Print version and exit
--	Separator between rocjitsu options and the target application command line

Usage Examples

# Local mode: in-process simulation
rocjitsu --config configs/gfx950_cdna4_kmd.json -- ./app

# Daemon mode: fork daemon + launch app
rocjitsu --daemon --config configs/gfx950_cdna4_kmd.json -- ./app args...

# Daemon-only: run server (no app launched)
rocjitsu --daemon --config configs/gfx950_cdna4_kmd.json

# Attach to running daemon
rocjitsu --attach --config configs/gfx950_cdna4_kmd.json -- ./app

Architecture

Client Process (HIP application)          Daemon Process
+-----------------------------------+     +-----------------------------------+
| HIP Runtime                       |     | SimulationEngine (background)     |
| ROCR (HSA Runtime)                |     |   VirtualMachine (topology root)  |
| libhsakmt (KFD thunk)             |     |     SoC -> XCD -> CP, CUs, L2     |
|   |                               |     |   SimulatedDriver (KFD ioctls)    |
|   v                               |     |     EventState, allocations       |
| LD_PRELOAD interposer             |     +-----------------------------------+
|   |-- open("/dev/kfd")            |                    |
|   |     RemoteDriver (RPC stub) --+----- Unix socket --+
|   |-- ioctl(kfd_fd, ...)          |     (RPC_IOCTL)
|   |-- mmap(drm_fd, offset)        |     (RPC_MMAP + SCM_RIGHTS)
|   |-- close(kfd_fd)               |     (RPC_CLOSE)
|   `-- madvise/mprotect/munmap     |
+-----------------------------------+

Ownership Chain

rj_vm_t (C API handle)
  |-- engine (unique_ptr<SimulationEngine>)
  |     `-- topology
  |           `-- root (unique_ptr<VirtualMachine>)
  |                 |-- SoC (adopted children: XCD, GpuMemory, ...)
  |                 `-- driver_ (unique_ptr<SimulatedDriver>)
  `-- engine_config

No circular ownership. Destroying rj_vm_t destroys the engine, which destroys the topology root (VirtualMachine), which destroys the driver.

Process Modes

Mode	Entry Point	Driver	Engine
Local	rocjitsu -- ./app	SimulatedDriver (in-process)	Background thread
Daemon (server)	rocjitsu --daemon -- ./app	SimulatedDriver (in-process)	Background thread
Daemon (client)	LD_PRELOAD interposer	RemoteDriver (RPC stub)	In daemon process

RPC Protocol

Message Format

All messages use a fixed 16-byte header followed by an opcode-specific payload:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    opcode     |   reserved    |          request_id           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        payload_bytes                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           result                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Opcodes

Opcode	Direction	Payload	Memfd
RPC_HANDSHAKE	C->D->C	RpcHandshakeResponse + topology path	No
RPC_CLOSE	C->D->C	None	No
RPC_MMAP	C->D->C	RpcMmapRequest / RpcMmapResponse	Optional
RPC_MUNMAP	C->D->C	RpcMunmapRequest	No
RPC_IOCTL	C->D->C	RpcIoctlRequest + raw args + inlined arrays	Optional

Embedded Pointer Serialization

KFD ioctls like WAIT_EVENTS, MAP_MEMORY_TO_GPU, and GET_PROCESS_APERTURES_NEW contain pointer fields that reference inline arrays. The client serializes these by inlining the pointed-to array after the struct. The daemon reconstructs the pointer to its local buffer. The client saves and restores original pointers across the RPC round-trip.

File Descriptor Passing

Memfds are passed via sendmsg()/recvmsg() with SCM_RIGHTS ancillary data. This is the standard Unix mechanism for cross-process file descriptor sharing (same pattern as QEMU vhost-user and CRIU).

Two RPC operations carry memfds:

• RPC_IOCTL (ALLOC_MEMORY response) --- allocation backing memfd
• RPC_MMAP (response) --- doorbell and event page memfds

Thread Safety

ROCR is heavily multithreaded. The rpc_mutex_ in RemoteDriver serializes all send+recv pairs. WAIT_EVENTS uses client-side polling with ppoll on a shutdown eventfd between daemon timeout=0 polls, avoiding mutex starvation.

Memory Sharing (Same-VA Architecture)

Both client and daemon map every memfd at the same virtual address using MAP_FIXED. GPU VAs are in the GPUVM aperture (0x1000000000 - 0x3FFFFFFFFFFF), which doesn't collide with either process's normal address space.

This means reinterpret_cast<void*>(gpu_va) works in both processes --- AQL packets, kernarg pointers, signal handles, ring buffers, and rptr/wptr all resolve correctly without VA translation.

Allocation Flow

Client                                     Daemon
------                                     ------
1. ROCR: fmm_allocate()
   mmap(MAP_PRIVATE|MAP_NORESERVE)

2. ROCR: hsaKmtAllocMemory()
   RemoteDriver::send_ioctl(ALLOC)  ---->  alloc_memory_ioctl()
                                            memfd_create + ftruncate + fallocate
                                            MAP_FIXED memfd at VA (proactive)
                                    <----  response + memfd (SCM_RIGHTS)
   [USERPTR: mincore+copy, MAP_FIXED]

3. ROCR: fmm_map_to_cpu()
   mmap(drm_fd, MAP_FIXED, offset)
   interposer routes DRM fd     ---->     mmap() -> already mapped, return
                                    <----  response

Proactive Mapping

The daemon MAP_FIXED maps all allocations at ALLOC_MEMORY time, not at mmap time. This prevents a race where the CP thread dereferences kernarg pointers before the client's fmm_map_to_cpu RPC arrives.

USERPTR Sharing

For USERPTR allocations (ring buffer, rptr/wptr), the client receives the daemon's memfd, copies committed pages via mincore, then MAP_FIXED replaces the anonymous pages with the shared memfd. The interposer suppresses MADV_HUGEPAGE and MADV_DONTFORK on GPUVM addresses to prevent kernel 6.17 shmem fault failures.

Trust Model

The RPC protocol assumes a trusted, version-matched client (i.e., the same rocjitsu build on both sides). The handshake rejects version mismatches, but payload contents are not validated beyond size checks. Untrusted or adversarial clients are out of scope.

Transport Layer

Abstract Interface

class Transport {
public:
  virtual bool send(const void *data, size_t len) = 0;
  virtual bool recv(void *data, size_t len) = 0;
  virtual ssize_t send_with_handle(const void *data, size_t len, int handle) = 0;
  virtual ssize_t recv_with_handle(void *data, size_t len,
                                   int *handle_out, size_t *num_handles) = 0;
  virtual void close() = 0;
};

Unix Implementation

UnixTransport wraps a Unix domain socket with SCM_RIGHTS for handle passing. Static factory methods:

• UnixTransport::listen(endpoint) --- create a listening socket
• UnixTransport::connect(endpoint) --- connect to a daemon
• transport->accept() --- accept a client connection

The endpoint is $XDG_RUNTIME_DIR/rocjitsu/daemon.sock by default.

Platform Portability

The transport abstraction enables future Windows support:

• Unix: Unix domain socket + SCM_RIGHTS
• Windows: Named pipe + DuplicateHandle

The RPC message format (header + payload) is platform-independent. Only the transport and shared-memory handle mechanism change per platform.

Event System

Auto-Reset Semantics in Daemon Mode

In daemon mode, WAIT_EVENTS uses timeout=0 polling via RPC. Multiple client threads poll the same event independently. Auto-reset is skipped on timeout=0 polls to prevent one poller from "stealing" the event signal from others. This matches the real kernel's behavior where wake_up_all() wakes all waiters simultaneously.

Shutdown Sequence

1. ROCR calls SET_EVENT on wake signals during AsyncEventsControl::Shutdown()
2. All client-side pollers see the event (no auto-reset on polls)
3. ROCR's async threads exit their loops, thread joins succeed
4. ROCR calls DESTROY_QUEUE, DESTROY_EVENT, close(kfd_fd)
5. RemoteDriver::close() sends RPC_CLOSE to daemon
6. Daemon calls SimulatedDriver::close() -> notify_closing()

Fd Management

Fd Reservation (DMTCP Pattern)

The driver reserves a contiguous fd range (4000-4255) for internal memfds. claim_fd(real_fd) moves a real fd into this range via dup2. The interposer's close() checks owns_reserved_fd(fd) to protect internal fds from ROCR's normal fd lifecycle.

Fd Protection

ROCR opens and closes many fds during init (topology files, /proc, DRM render nodes). Without protection, a transient file's fd number could collide with an allocation memfd. The owned_fds_ set in SimulatedDriver tracks all internal memfds, and the interposer suppresses close() on protected fds.

Limitations

Kernel 6.17 shmem Bug

MAP_SHARED at GPUVM addresses can trigger SIGBUS on kernel 6.17 due to a shmem large folio regression. The interposer suppresses MADV_HUGEPAGE and MADV_DONTFORK on GPUVM addresses as a workaround. Upstream fixes are in kernel 6.12.59+ and 6.15+.

File Map

File	Role
rpc.h	Protocol types, send/recv helpers
transport.h/.cpp	Abstract transport + Unix socket implementation
remote_driver.h/.cpp	Client-side RPC stub
simulated_kfd.h/.cpp	KFD ioctl dispatch, allocation table, events
interposer.cpp	LD_PRELOAD syscall intercepts
events.h/.cpp	KFD event subsystem (create, set, wait, destroy)
tools/rocjitsu/main.cpp	CLI entry point, RPC dispatch loop (daemon mode)
tests/daemon_test.cpp	Gtest fixture for daemon end-to-end tests
virtual_machine.h/.cpp	VM component owning SoC and driver

Future Work

• Windows: Named pipe transport, KMD escape driver backend
• RDMA: Rack-scale transport for distributed simulation
• Platform-independent commands: Abstract command enum for cross-platform