TheRock/rocm-systems/experimental/perf-dkms

AMD GPU Performance Monitoring - Perf PMU Stub

⚠️ EXPERIMENTAL - NOT FOR PRODUCTION USE

This is a research and development project. The module is not ready for general use and may cause system instability.

Requirements:

• GFX12/RDNA4 GPUs only (RX 9070 series)

Use at your own risk in development/testing environments only.

A Linux kernel module implementing AMD GPU performance counter integration with the perf subsystem. This module provides AQL (Asynchronous Queuing Language) packet generation for GPU performance monitoring on GFX12/RDNA4 architectures.

Overview

This kernel module bridges AMD GPU hardware performance counters with the Linux perf interface, enabling standard perf tools to monitor GPU performance events. It includes:

• AQL C Library: Pure C implementation of PM4 packet generation for GPU counter programming
• Perf PMU Driver: Complete Linux perf subsystem integration
• Hardware Support: GFX12/RDNA4 GPU architecture support
• Counter Registry: Architecture-specific performance event definitions

Features

• ✅ Linux perf subsystem integration
• ✅ AQL/PM4 packet generation for GPU counter control
• ✅ GFX12/RDNA4 hardware support
• ✅ Multi-GPU architecture support
• ✅ Hardware block counter definitions (SQ, CPC, GL2C, etc.)
• ✅ Dimension-Specific Monitoring - Monitor specific SE/SA/WGP hardware units
• ✅ Standard perf tool compatibility
• ✅ DKMS-compatible installation

Quick Start

Prerequisites

# Install build tools and kernel headers
sudo apt update
sudo apt install build-essential linux-headers-$(uname -r)

# Verify perf tools are available
perf --version

Building the Module

# Clone and enter directory
cd perf-dkms

# Build kernel module and tools
cmake -B build && cmake --build build

# The build produces:
# - build/src/amdgpu_pmu.ko (kernel module)
# - build/src/aql_c/tools/packet_gen_tool (PM4 packet generator)
# - build/src/aql_c/tools/pm4_decoder (PM4 packet decoder)
# - build/src/aql_c/tests/* (validation tests)

Loading the Module

# Load the module
sudo insmod build/src/amdgpu_pmu.ko

# Or if using .o file (development kernels):
sudo insmod build/src/amdgpu_pmu.o

# Verify module is loaded
lsmod | grep amdgpu_pmu

# Check kernel messages
dmesg | tail -20

Verification

# Check PMU device registration
ls -la /sys/bus/event_source/devices/ | grep pmu

# List available GPU performance events
perf list | grep amdgpu_pmu

# Or directly check sysfs
ls /sys/bus/event_source/devices/amdgpu_pmu/events/

Testing with Perf

Basic GPU Counter Test

Test GPU shader wave counting with detailed verbose output:

# Monitor SQ_WAVES counter with 800ms intervals
perf stat -I 800 -vv -C 0 -a --no-inherit -e amdgpu_pmu/sq_waves/ sleep 4

Command breakdown:

• -I 800: Print counter deltas every 800ms
• -vv: Very verbose output (shows counter reads and updates)
• -C 0: Monitor CPU 0 (or use -a for all CPUs)
• -a: System-wide monitoring
• --no-inherit: Don't inherit counters to child processes
• -e amdgpu_pmu/sq_waves/: Monitor the SQ_WAVES GPU counter

Available Hardware Counters

The module exposes GPU hardware performance counters through the perf interface:

# Shader Processor (SQ) counters
perf stat -e amdgpu_pmu/sq_waves/ <command>           # Active waves
perf stat -e amdgpu_pmu/sq_busy_cycles/ <command>     # SQ busy cycles
perf stat -e amdgpu_pmu/sq_instructions/ <command>    # Instructions executed

# L2 Cache (GL2C) counters
perf stat -e amdgpu_pmu/gl2c_hit/ <command>           # L2 cache hits
perf stat -e amdgpu_pmu/gl2c_miss/ <command>          # L2 cache misses

# Command Processor (CPC) counters
perf stat -e amdgpu_pmu/cpc_busy/ <command>           # CP busy cycles
perf stat -e amdgpu_pmu/cpc_stall/ <command>          # CP stall cycles

Multi-Event Monitoring

# Monitor multiple GPU counters simultaneously
perf stat -e amdgpu_pmu/sq_waves/,pmu_stub/gl2c_hit/,pmu_stub/gl2c_miss/ sleep 5

# System-wide GPU monitoring
sudo perf stat -a -e amdgpu_pmu/sq_waves/ sleep 10

# Per-process GPU monitoring (when supported)
perf stat -e amdgpu_pmu/sq_waves/ ./gpu_workload

Advanced Perf Usage

# Record GPU events with sampling
sudo perf record -e amdgpu_pmu/sq_waves/ -a sleep 5
sudo perf report

# Monitor with custom intervals
perf stat -I 1000 -e amdgpu_pmu/sq_waves/ sleep 10

# Export to JSON
perf stat -j -e amdgpu_pmu/sq_waves/ sleep 2

Dimension-Specific Monitoring

The amdgpu_pmu driver supports monitoring specific GPU hardware dimensions (Shader Engines, Shader Arrays, Work Group Processors) for fine-grained performance analysis.

Quick Examples

# Monitor specific Shader Engine (SE)
perf stat -e amdgpu_pmu/sq_waves,se=0/ -a sleep 1

# Monitor SE and Shader Array (SA)
perf stat -e amdgpu_pmu/sq_waves,se=1,sa=0/ -a sleep 1

# Monitor full dimension hierarchy
perf stat -e amdgpu_pmu/sq_waves,se=2,sa=1,wgp=3/ -a sleep 1

# Compare activity across all Shader Engines
perf stat -e amdgpu_pmu/sq_waves,se=0/ \
          -e amdgpu_pmu/sq_waves,se=1/ \
          -e amdgpu_pmu/sq_waves,se=2/ \
          -e amdgpu_pmu/sq_waves,se=3/ \
          -a ./my_gpu_app

Hardware Dimensions

AMD GPUs are organized hierarchically:

• SE (Shader Engine): Top-level parallel units, typically 4 per GPU
• SA (Shader Array): Subdivisions within SE, typically 2 per SE
• WGP (Work Group Processor): Compute units, typically 4 per SA
• CU (Compute Unit): Individual SIMD units

For GFX12: 4 SE × 2 SA × 4 WGP = 32 unique addressable locations

Available Parameters

• se=N - Shader Engine index (0-3 typical)
• sa=N - Shader Array index (0-1 typical)
• wgp=N - Work Group Processor index (0-3 typical)
• cu=N - Compute Unit index (0-63 typical)
• xcc=N - XCC index (usually 0)

Supported Counters

Not all counters support dimension targeting:

• SQ Counters (SE/SA/WGP): sq_waves, sq_insts_valu, sq_busy_cycles
• TA Counters (SE/SA/WGP): ta_busy, ta_total_wavefronts
• GL2C Counters (SE/SA): gl2c_hit, gl2c_miss
• Global Counters (no dimensions): grbm_count, grbm_busy

Use Cases

Load Balancing Analysis: Check if work is evenly distributed across SEs

perf stat -e amdgpu_pmu/sq_waves,se=0/ \
          -e amdgpu_pmu/sq_waves,se=1/ \
          -e amdgpu_pmu/sq_waves,se=2/ \
          -e amdgpu_pmu/sq_waves,se=3/ \
          -a ./workload

Hotspot Detection: Identify which hardware unit is most active

perf record -e amdgpu_pmu/sq_busy_cycles,se=0/ \
            -e amdgpu_pmu/sq_busy_cycles,se=1/ \
            -a sleep 10

Memory Analysis: Monitor cache behavior per Shader Array

perf stat -e amdgpu_pmu/gl2c_hit,se=0,sa=0/ \
          -e amdgpu_pmu/gl2c_miss,se=0,sa=0/ \
          -a ./memory_test

Documentation

For detailed information, see:

• User Guide: docs/user_guide_dimensions.md - Complete usage guide with examples
• Design Doc: docs/design.md - Technical implementation details
• Tests: test/load_test.sh, test/perf_test.sh - Integration tests

Architecture Support

Supported GPUs

• GFX12 (RDNA4): ✅ Currently Supported

  • Radeon RX 9070 series
  • All hardware blocks (SQ, CPC, CPF, GL1, GL2, CHA, CHC, etc.)

• GFX9 (Vega/MI): ✅ Supported

  • MI100, MI200, Vega series
  • SQ, GL2C, TA, GRBM blocks

• GFX10/11: ❌ Not Currently Supported

  • Framework exists but event mappings incomplete

Hardware Blocks

The module provides counters for these GPU hardware blocks:

Block	Description	Example Counters
SQ	Shader Processor (Sequencer)	waves, instructions, busy_cycles
GL2C	Graphics L2 Cache	hit, miss, requests
GL1A/C	Graphics L1 Cache	hit, miss, latency
CPC	Command Processor Compute	busy, stall, packets
CPF	Command Processor Fetcher	fetch_stall, me_busy
CHA/CHC	Cache Hierarchy Arbiters	requests, conflicts
TCC	Texture Cache Controller	hit, miss, write
TCP	Texture Cache Processor	read, write, stall

Development

Building Components

# Build only kernel module
cmake -B build -DBUILD_KERNEL_MODULE=ON -DBUILD_USERSPACE_TOOLS=OFF -DBUILD_TESTS=OFF && cmake --build build --target kernel_module

# Build only userspace tools
cmake -B build -DBUILD_KERNEL_MODULE=OFF -DBUILD_USERSPACE_TOOLS=ON -DBUILD_TESTS=OFF && cmake --build build

# Build only tests
cmake -B build -DBUILD_KERNEL_MODULE=OFF -DBUILD_USERSPACE_TOOLS=OFF -DBUILD_TESTS=ON && cmake --build build

# Clean everything
rm -rf build

Running Tests

# Build and run tests
cmake -B build -DBUILD_TESTS=ON && cmake --build build

# Run PM4 packet validation
build/src/aql_c/tests/test_pm4

# Run counter registry tests
build/src/aql_c/tests/test_counter_registry

# Run packet generation tests
build/src/aql_c/tests/test_packet_generation

Debug Output

# Enable kernel debug messages
echo 'module amdgpu_pmu +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

# View trace output
sudo cat /sys/kernel/debug/tracing/trace

# Check kernel log
sudo dmesg -w | grep -i pmu

Tools

PM4 Packet Generator:

build/src/aql_c/tools/packet_gen_tool <gpu_arch> <block> <event_id>

PM4 Decoder:

build/src/aql_c/tools/pm4_decoder <hex_packet_data>

File Structure

perf-dkms/
├── src/
│   ├── pmu_main.c              # Perf PMU driver
│   ├── pmu_events.c            # Event definitions
│   ├── pmu_dimension.h         # Dimension encoding/validation
│   ├── aql_perf.c/h            # AQL session & GPU discovery
│   ├── aql_packet_ops.c        # PM4 packet operations
│   ├── aql_pmu_integration.c   # PMU-AQL bridge
│   ├── aql_error_recovery.c    # Error handling & recovery
│   ├── aql_queue_manager.c/h   # AQL queue lifecycle
│   ├── kfd_ioctl_bridge.c/h    # KFD ioctl bridge (vm_mmap)
│   └── aql_c/                  # AQL C library (kernel/userspace)
│       ├── packet_generation.c # PM4 packet generation
│       ├── pm4_packets.c       # Low-level PM4 primitives
│       ├── counter_registry.c  # Counter definitions
│       ├── arch_creator.c      # Architecture factory
│       ├── gfx12_creator.c     # GFX12 implementation
│       ├── gfx12_events.c      # GFX12 event definitions
│       ├── gfx9_creator.c      # GFX9 implementation
│       ├── gfx9_events.c       # GFX9 event definitions
│       ├── aql_queue.c/h       # Ring buffer & IB pool
│       ├── tests/              # Validation tests
│       └── tools/              # Packet tools
├── docs/                       # Documentation
├── test/                       # Integration tests
└── README.md                   # This file

How It Works

Architecture Overview

┌─────────────┐
│  perf tool  │  (userspace)
└──────┬──────┘
       │ syscall
┌──────▼──────────────────────┐
│   Linux Perf Subsystem      │
└──────┬──────────────────────┘
       │ PMU callbacks
┌──────▼──────────────────────┐
│   amdgpu_pmu (this module)    │  (kernel)
│  ┌─────────────────────┐    │
│  │ AQL Packet Gen (C)  │    │
│  │ - PM4 packets       │    │
│  │ - Counter registry  │    │
│  │ - Arch detection    │    │
│  └─────────────────────┘    │
└──────┬──────────────────────┘
       │ PM4 commands
┌──────▼──────────────────────┐
│   AMD GPU Hardware          │
│   (GFX12/RDNA4)             │
└─────────────────────────────┘

Event Flow

1. User runs perf: perf stat -e amdgpu_pmu/sq_waves/ <cmd>
2. Perf subsystem: Calls PMU driver add() callback
3. PMU driver: Creates AQL measurement session
4. AQL library: Generates PM4 START packet
5. Hardware: GPU enables SQ_WAVES counter
6. Monitoring: Counter accumulates during execution
7. Read: PMU driver generates PM4 READ packet
8. Hardware: GPU returns counter value
9. Perf: Displays counter to user

Troubleshooting

Module won't load

# Check kernel version compatibility
uname -r
ls /lib/modules/$(uname -r)/build

# Check for errors
sudo dmesg | grep -i error

No events visible

# Verify sysfs registration
ls /sys/bus/event_source/devices/amdgpu_pmu/

# Check event files
cat /sys/bus/event_source/devices/amdgpu_pmu/events/*

Perf can't find events

# Refresh perf cache
sudo rm -rf ~/.debug/
perf list | grep amdgpu_pmu

# Check permissions
sudo chmod -R 755 /sys/bus/event_source/devices/amdgpu_pmu/

Build failures

# Ensure kernel headers match running kernel
sudo apt install linux-headers-$(uname -r)

# Clean and rebuild
rm -rf build && cmake -B build && cmake --build build

Unloading

# Remove module
sudo rmmod amdgpu_pmu

# Verify removal
lsmod | grep amdgpu_pmu

# Check for errors
dmesg | tail -20

Limitations

Known Bugs

⚠️ Critical Issues:

1. First Dimension Only: The module currently only supports returning counter values for the first hardware dimension (e.g., first shader engine, first compute unit). Multi-dimensional counter reads are not yet fully implemented.

2. Memory Leaks on Stop: The stop operation may not properly clean up resources in certain scenarios, leading to memory leaks. This occurs when:

   • Multiple events are stopped in quick succession
   • System shutdown interrupts the cleanup sequence
   • GPU state transitions happen during stop operations

3. GFX12 Only: This module currently only supports GFX12/RDNA4 GPUs. Attempting to use on GFX9/10/11 architectures will fail. The architecture detection code exists but event mappings and register definitions are incomplete for older generations.

Current Restrictions

1. Hardware Integration: Uses real GPU hardware access via:

   • KFD (Kernel Fusion Driver) integration
   • AMDGPU driver coordination
   • GPU memory allocation for PM4 buffers

2. Architecture Support:

   • GFX12/RDNA4: Full support (RX 9070 series)
   • GFX9/Vega/MI: Supported (MI100, MI200, Vega series)
   • GFX10/11: Not supported (framework only, missing event definitions)

3. Kernel Compatibility:

   • Development kernels may produce .o instead of .ko files
   • DKMS requires stable kernel versions

Future Work

• Complete GFX9/10/11 support
• Interrupt-driven counter collection
• SPM (Streaming Performance Monitor) support
• Multi-GPU enumeration improvements

References

AMD Documentation

• AMD GPU Performance Counters
• PM4 Packet Format Specifications
• GFX12/RDNA4 Architecture Documentation

Linux Kernel

• Perf Subsystem Documentation
• PMU Driver Development

Related Projects

• AQLProfile (C++ implementation)
• ROCm Profiler SDK
• ROCProfiler

License

See LICENSE file for details.

Contributing

This is a research/development module. Contributions welcome for:

• Additional GPU architecture support
• Hardware integration improvements
• Performance optimizations
• Documentation enhancements

Contact

For issues and questions, please file a GitHub issue.