derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
78 lines
2.4 KiB
Markdown
78 lines
2.4 KiB
Markdown
# SDMA Test
|
|
|
|
## Overview
|
|
|
|
This example benchmarks SDMA (System DMA) engine performance by measuring bandwidth for Host-to-Device (H2D), Device-to-Device (D2D), and Device-to-Host (D2H) memory transfers. It runs configurable iterations of async copy operations using pinned host memory and HIP streams, reports per-transfer-type bandwidth in GB/s, and verifies data integrity via `memcmp`. An infinite-loop mode allows sustained profiling of SDMA utilization over extended periods.
|
|
|
|
## Source Files
|
|
|
|
- `sdma-test.cpp` - Implements transfer benchmarking with `hipMemcpyAsync`, CLI argument parsing with size suffix support (K/M/G), bandwidth calculation, and signal handling for graceful shutdown in infinite mode.
|
|
|
|
## Prerequisites
|
|
|
|
- CMake 3.25+
|
|
- HIP runtime and `hipcc` compiler
|
|
|
|
## Building
|
|
|
|
**Standalone build:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/sdma-test -DCMAKE_PREFIX_PATH=/opt/rocm
|
|
cmake --build <build_dir>
|
|
```
|
|
|
|
**As part of the examples suite:**
|
|
|
|
```bash
|
|
cmake -B <build_dir> -S <project_root>/examples/ -DCMAKE_PREFIX_PATH=/opt/rocm
|
|
cmake --build <build_dir> --target sdma-test
|
|
```
|
|
|
|
## Running
|
|
|
|
```bash
|
|
# Default: 512 MB, 10 iterations, 10 copies/iteration, device 0
|
|
./sdma-test
|
|
|
|
# Custom: 1 GB transfers, 5 iterations
|
|
./sdma-test -s 1024 -n 5
|
|
|
|
# Infinite mode (Ctrl+C to stop)
|
|
./sdma-test -s 256 -n 0
|
|
```
|
|
|
|
**Flags:**
|
|
|
|
| Flag | Description | Default |
|
|
| ------ | ------------- | --------- |
|
|
| `-s, --size` | Transfer size in MB (supports K/M/G suffixes) | 512 |
|
|
| `-n, --iterations` | Number of iterations (0 = infinite) | 10 |
|
|
| `-c, --copies` | Number of copies per iteration | 10 |
|
|
| `-d, --device` | GPU device ID | 0 |
|
|
|
|
## Profiling with rocprofiler-systems
|
|
|
|
```bash
|
|
rocprof-sys-run -- ./sdma-test -s 256 -n 5
|
|
```
|
|
|
|
### Recommended Configuration
|
|
|
|
| Variable | Value | Purpose |
|
|
| ---------- | ------- | --------- |
|
|
| `ROCPROFSYS_ROCM_DOMAINS` | `hip_runtime_api,kernel_dispatch,memory_copy` | Trace HIP API and memory copy operations |
|
|
| `ROCPROFSYS_ROCM_EVENTS` | `SQ_WAVES,GRBM_COUNT` | Sample GPU hardware counters |
|
|
| `ROCPROFSYS_AMD_SMI_METRICS` | `sdma_usage` | Collect SDMA engine usage via AMD-SMI |
|
|
| `ROCPROFSYS_TRACE` | `true` | Generate Perfetto trace for timeline analysis |
|
|
|
|
To capture SDMA-specific metrics:
|
|
|
|
```bash
|
|
rocprof-sys-run \
|
|
-e ROCPROFSYS_ROCM_DOMAINS=hip_runtime_api,memory_copy \
|
|
-e ROCPROFSYS_AMD_SMI_METRICS=sdma_usage \
|
|
-e ROCPROFSYS_TRACE=true \
|
|
-- ./sdma-test -s 512 -n 5
|
|
```
|