TheRock/rocm-systems/projects/rocprofiler-systems/examples/unified-memory/README.md

# Unified Memory

## Overview

This example exercises HIP managed-memory (`hipMallocManaged`) access patterns
that intentionally trigger KFD page-fault and page-migration events. It runs a
sequence of workloads -- single-allocation host/device alternation, ping-pong
access, prefetch-driven migration, memory-pressure scenarios, and
concurrent-kernel access. On systems that expose KFD migrations, these
workloads populate the host-to-device and device-to-host direction buckets. They
also populate every fault trigger (`gpu_page_fault`, `cpu_page_fault`,
`prefetch`) in the unified-memory report. It is the reference workload for
validating `ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING` in the pytest/CTest
validation suite.

## Source Files

- `unified-memory.cpp` - Implements 19 managed-memory test scenarios (basic
  faulting, ping-pong, prefetch, pressure, concurrent kernels, fine-grain
  access), CLI argument parsing, and a configurable subset runner (defaults to
  the first 9 tests, which is what the e2e validator expects).

## Prerequisites

- CMake 3.25+
- HIP runtime and `hipcc` compiler
- XNACK-capable AMD GPU (for example, MI200/MI300 or newer Instinct accelerators) with `HSA_XNACK=1`
- ROCm 7.13 or later, or ROCProfiler-SDK 1.2.2 or later for standalone SDK installs (only required when running under `rocprof-sys-run`)

## Get the Example Source

If you don't already have a `rocm-systems` checkout, use Git sparse checkout to
download only this example and the helper files needed for a standalone example
build:

```bash
git clone --filter=blob:none --sparse https://github.com/ROCm/rocm-systems.git
git -C rocm-systems sparse-checkout set \
    projects/rocprofiler-systems/examples/cmake \
    projects/rocprofiler-systems/examples/unified-memory \
    projects/rocprofiler-systems/scripts
```

Then use the ROCm Systems Profiler project directory as the project root:

```bash
cd rocm-systems/projects/rocprofiler-systems
```

## Building

**Standalone build:**

```bash
cmake -B <build_dir> -S <project_root>/examples/unified-memory -DCMAKE_PREFIX_PATH=/opt/rocm
cmake --build <build_dir>
```

**As part of the examples suite:**

```bash
cmake -B <build_dir> -S <project_root>/examples/ -DCMAKE_PREFIX_PATH=/opt/rocm
cmake --build <build_dir> --target unified-memory
```

## Running

```bash
# Default: 64 MB per allocation, 512 MB pressure pool, 8 ping-pong iterations, first 9 tests
./unified-memory

# Smaller workload matching the e2e validation path
./unified-memory -s 32 -p 256 -i 4

# Run the full set of 19 tests
./unified-memory -a
```

**Flags:**

| Flag | Description | Default |
| ------ | ------------- | --------- |
| `-s, --size <MB>` | Per-allocation size in MB | 64 |
| `-p, --pressure <MB>` | Managed memory for the pressure test | 512 |
| `-d, --device <ID>` | GPU device ID | 0 |
| `-i, --iterations <N>` | Ping-pong iterations | 8 |
| `-a, --all` | Run all 19 tests (default: first 9 only) | off |

Running on a host without XNACK support, or without `HSA_XNACK=1`, will not
crash, but no KFD page-fault / page-migration events will be captured under
`rocprof-sys-run`, so the unified-memory report will be empty.

## Profiling with rocprofiler-systems

```bash
HSA_XNACK=1 \
ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING=ON \
rocprof-sys-run -- ./unified-memory -s 32 -p 256 -i 4
```

Output files (written to the standard rocprofiler-systems output directory):

| File | Contents |
| ------ | ---------- |
| `unified_memory.txt` | Human-readable per-GPU summary: total page faults, trigger breakdown (`gpu_page_fault`, `cpu_page_fault`, `prefetch`), and Host-to-Device / Device-to-Host effective migration throughput when migration events are present. |
| `unified_memory.json` | Machine-readable equivalent with the same fields, plus an `xnack_enabled` flag and an always-present `device_to_device` bucket for schema stability. Migration buckets can remain at zero on systems that do not generate KFD migration events. |
| `perfetto-trace.proto` | Standard Perfetto trace, including KFD page-fault tracks and migration tracks when migration events are present. |
| `rocpd.db` *(optional)* | Standard ROCpd database; KFD events are recorded in the `kfd_page_fault` / `kfd_page_migrate` tables. |

### Shared-HBM Systems

On MI300A and other systems where CPU and GPU agents point to the same physical
HBM, page faults can occur without page migrations because there is no separate
CPU memory and GPU memory to migrate between. In that topology, a valid
unified-memory profile can be fault-only: page-fault totals and trigger
breakdowns populate, migration counters remain zero, and the Perfetto
migration-throughput track is not shown.

### Recommended Configuration

| Variable | Value | Purpose |
| ---------- | ------- | --------- |
| `HSA_XNACK` | `1` | Required runtime prerequisite for any KFD page-fault / migration events. |
| `ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING` | `ON` | Enable the aggregate `unified_memory.{txt,json}` reports and automatically opt-in to the required KFD tracing domains. |
| `ROCPROFSYS_ROCM_DOMAINS` | `hip_runtime_api,kernel_dispatch,kfd_events` | Capture HIP API calls and kernel dispatches alongside the KFD events. `kfd_events` is auto-enabled by the setting above; listing it explicitly is harmless. |
| `ROCPROFSYS_TRACE` | `ON` | Generate the Perfetto trace alongside the unified-memory reports. |

```bash
HSA_XNACK=1 \
ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING=ON \
ROCPROFSYS_ROCM_DOMAINS=hip_runtime_api,kernel_dispatch,kfd_events \
ROCPROFSYS_TRACE=ON \
rocprof-sys-run -- ./unified-memory -s 32 -p 256 -i 4
```

## See Also

- [`examples/README.md`](../README.md) -- the unified-memory entry in the top-level examples catalog (GPU Compute table).
- `docs/how-to/configuring-runtime-options.rst` -- the `ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING` reference entry.
- `tests/pytest/test_unified_memory.py` -- the pytest harness that drives unified-memory output validation against this example.