TheRock/rocm-systems/projects/rocprofiler-systems/examples/unified-memory

Unified Memory

Overview

This example exercises HIP managed-memory (hipMallocManaged) access patterns that intentionally trigger KFD page-fault and page-migration events. It runs a sequence of workloads -- single-allocation host/device alternation, ping-pong access, prefetch-driven migration, memory-pressure scenarios, and concurrent-kernel access. On systems that expose KFD migrations, these workloads populate the host-to-device and device-to-host direction buckets. They also populate every fault trigger (gpu_page_fault, cpu_page_fault, prefetch) in the unified-memory report. It is the reference workload for validating ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING in the pytest/CTest validation suite.

Source Files

• unified-memory.cpp - Implements 19 managed-memory test scenarios (basic faulting, ping-pong, prefetch, pressure, concurrent kernels, fine-grain access), CLI argument parsing, and a configurable subset runner (defaults to the first 9 tests, which is what the e2e validator expects).

Prerequisites

• CMake 3.25+
• HIP runtime and hipcc compiler
• XNACK-capable AMD GPU (for example, MI200/MI300 or newer Instinct accelerators) with HSA_XNACK=1
• ROCm 7.13 or later, or ROCProfiler-SDK 1.2.2 or later for standalone SDK installs (only required when running under rocprof-sys-run)

Get the Example Source

If you don't already have a rocm-systems checkout, use Git sparse checkout to download only this example and the helper files needed for a standalone example build:

git clone --filter=blob:none --sparse https://github.com/ROCm/rocm-systems.git
git -C rocm-systems sparse-checkout set \
    projects/rocprofiler-systems/examples/cmake \
    projects/rocprofiler-systems/examples/unified-memory \
    projects/rocprofiler-systems/scripts

Then use the ROCm Systems Profiler project directory as the project root:

cd rocm-systems/projects/rocprofiler-systems

Building

Standalone build:

cmake -B <build_dir> -S <project_root>/examples/unified-memory -DCMAKE_PREFIX_PATH=/opt/rocm
cmake --build <build_dir>

As part of the examples suite:

cmake -B <build_dir> -S <project_root>/examples/ -DCMAKE_PREFIX_PATH=/opt/rocm
cmake --build <build_dir> --target unified-memory

Running

# Default: 64 MB per allocation, 512 MB pressure pool, 8 ping-pong iterations, first 9 tests
./unified-memory

# Smaller workload matching the e2e validation path
./unified-memory -s 32 -p 256 -i 4

# Run the full set of 19 tests
./unified-memory -a

Flags:

Flag	Description	Default
-s, --size <MB>	Per-allocation size in MB	64
-p, --pressure <MB>	Managed memory for the pressure test	512
-d, --device <ID>	GPU device ID	0
-i, --iterations <N>	Ping-pong iterations	8
-a, --all	Run all 19 tests (default: first 9 only)	off

Running on a host without XNACK support, or without HSA_XNACK=1, will not crash, but no KFD page-fault / page-migration events will be captured under rocprof-sys-run, so the unified-memory report will be empty.

Profiling with rocprofiler-systems

HSA_XNACK=1 \
ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING=ON \
rocprof-sys-run -- ./unified-memory -s 32 -p 256 -i 4

Output files (written to the standard rocprofiler-systems output directory):

File	Contents
unified_memory.txt	Human-readable per-GPU summary: total page faults, trigger breakdown (gpu_page_fault, cpu_page_fault, prefetch), and Host-to-Device / Device-to-Host effective migration throughput when migration events are present.
unified_memory.json	Machine-readable equivalent with the same fields, plus an xnack_enabled flag and an always-present device_to_device bucket for schema stability. Migration buckets can remain at zero on systems that do not generate KFD migration events.
perfetto-trace.proto	Standard Perfetto trace, including KFD page-fault tracks and migration tracks when migration events are present.
rocpd.db (optional)	Standard ROCpd database; KFD events are recorded in the kfd_page_fault / kfd_page_migrate tables.

Shared-HBM Systems

On MI300A and other systems where CPU and GPU agents point to the same physical HBM, page faults can occur without page migrations because there is no separate CPU memory and GPU memory to migrate between. In that topology, a valid unified-memory profile can be fault-only: page-fault totals and trigger breakdowns populate, migration counters remain zero, and the Perfetto migration-throughput track is not shown.

Recommended Configuration

Variable	Value	Purpose
HSA_XNACK	1	Required runtime prerequisite for any KFD page-fault / migration events.
ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING	ON	Enable the aggregate unified_memory.{txt,json} reports and automatically opt-in to the required KFD tracing domains.
ROCPROFSYS_ROCM_DOMAINS	hip_runtime_api,kernel_dispatch,kfd_events	Capture HIP API calls and kernel dispatches alongside the KFD events. kfd_events is auto-enabled by the setting above; listing it explicitly is harmless.
ROCPROFSYS_TRACE	ON	Generate the Perfetto trace alongside the unified-memory reports.

HSA_XNACK=1 \
ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING=ON \
ROCPROFSYS_ROCM_DOMAINS=hip_runtime_api,kernel_dispatch,kfd_events \
ROCPROFSYS_TRACE=ON \
rocprof-sys-run -- ./unified-memory -s 32 -p 256 -i 4

See Also

• examples/README.md -- the unified-memory entry in the top-level examples catalog (GPU Compute table).
• docs/how-to/configuring-runtime-options.rst -- the ROCPROFSYS_USE_UNIFIED_MEMORY_PROFILING reference entry.
• tests/pytest/test_unified_memory.py -- the pytest harness that drives unified-memory output validation against this example.