TheRock/tests/extended_tests/benchmark

Benchmark Testing Framework

Automated benchmark testing framework for ROCm libraries with system detection, results collection, and performance tracking.

Prerequisites: See Extended Tests Framework Overview for environment setup, CI/CD integration, and general architecture.

Table of Contents

• Features
• Project Structure
• Quick Start
• Architecture
• Adding New Benchmarks

Features

• Automated Benchmark Execution - ROCfft, ROCrand, ROCsolver, hipBLASLt, rocBLAS, RCCL
• System Auto-Detection - Hardware, OS, GPU, and ROCm version detection
• Distributed Testing - Multi-GPU RCCL benchmarks (requires OpenMPI in Docker image)
• Results Management - Local storage (JSON) and API upload with retry logic
• Performance Tracking - LKG (Last Known Good) comparison
• Comprehensive Logging - File rotation and configurable log levels
• Modular Architecture - Extensible design for adding new benchmarks
• CI/CD Integration - Parallel execution with regular tests in nightly CI

Project Structure

extended_tests/benchmark/                       # Benchmark test directory
├── scripts/                     # Benchmark test implementations
│   ├── benchmark_base.py        # Base class for all benchmarks
│   ├── test_hipblaslt_benchmark.py
│   ├── test_rccl_benchmark.py
│   ├── test_rocblas_benchmark.py
│   ├── test_rocfft_benchmark.py
│   ├── test_rocrand_benchmark.py
│   └── test_rocsolver_benchmark.py
│
├── configs/                     # Benchmark-specific configs
│   ├── hipblaslt.json           # hipBLASLt benchmark config
│   ├── rccl.json                # RCCL benchmark config
│   ├── rocblas.json             # rocBLAS benchmark config
│   └── rocfft.json              # ROCfft benchmark config
│
├── benchmark_test_matrix.py     # Benchmark matrix definitions
└── README.md                    # This file

Note: For the overall extended_tests structure including shared utilities (utils/), configuration (configs/), and other test types, see the main extended_tests README.

Quick Start

Available Benchmarks

The following benchmark tests are defined in tests/extended_tests/benchmark/benchmark_test_matrix.py:

Test Name	Library	Platform	Timeout	Shards
hipblaslt_bench	hipBLASLt	Linux, Windows	60 min	1
rccl_bench	RCCL	Linux	60 min	1
rocblas_bench	rocBLAS	Linux	60 min	1
rocfft_bench	ROCfft	Linux, Windows	60 min	1
rocrand_bench	ROCrand	Linux, Windows	60 min	1
rocsolver_bench	ROCsolver	Linux, Windows	60 min	1

GPU Family Support:

GPU Family	Platform	Architecture	Benchmark Supported	Benchmark CI Status
gfx94x	Linux	MI300X/MI325X (CDNA3)	Yes	Enabled (nightly CI)
gfx1151	Windows	RDNA 3.5	Yes	Enabled (nightly CI)
gfx950	Linux	MI355X (CDNA4)	Yes	Not enabled
gfx110x	Windows	RDNA 2	Yes	Not enabled
gfx110x	Linux	RDNA 2	Yes	Not enabled
gfx120x	Linux	RDNA 3	Yes	Not enabled
gfx120x	Windows	RDNA 3	Yes	Not enabled
gfx90x	Linux	MI200 (CDNA2)	Yes	Not enabled
gfx1151	Linux	RDNA 3.5	Yes	Not enabled

Note: All benchmarks are architecture-agnostic and support any ROCm-compatible GPU. The table above lists GPU families actively used in CI testing. To add support for additional GPU families, update amdgpu_family_matrix.py with appropriate benchmark-runs-on runners.

Architecture

Benchmark Script Execution Flow

1. Initialize ExtendedTestClient
   ↓ Auto-detect system (GPU, OS, ROCm version)
   ↓ Load configuration from config.yml

2. Run Benchmarks
   ↓ Execute benchmark binary
   ↓ Capture output to log file

3. Parse Results
   ↓ Extract metrics from log file
   ↓ Structure data according to schema

4. Upload Results
   ↓ Submit to API (with retry)
   ↓ Save JSON locally

5. Compare with LKG
   ↓ Fetch last known good results
   ↓ Calculate performance delta

6. Report Results
   ↓ Display formatted table
   ↓ Append to GitHub Actions step summary
   ↓ Return exit code (0=success, 1=failure)

Adding New Benchmarks

To add a new benchmark test to the nightly CI:

1. Create Benchmark Script

Create extended_tests/benchmark/scripts/test_your_benchmark.py. Reference existing benchmarks like test_rocfft_benchmark.py as a template.

Key components:

• Inherit from BenchmarkBase class
• Implement run_benchmarks() - executes binary and logs output
• Implement parse_results() - parses logs and returns structured data
• Results are automatically uploaded to API via base class

Example:

import sys
from pathlib import Path
from typing import Dict, List, Tuple, Any
from prettytable import PrettyTable

sys.path.insert(0, str(Path(__file__).resolve().parents[2]))  # For extended_tests/utils
sys.path.insert(0, str(Path(__file__).parent))  # For benchmark_base
from benchmark_base import BenchmarkBase, run_benchmark_main
from utils.logger import log


class YourBenchmark(BenchmarkBase):
    def __init__(self):
        super().__init__(benchmark_name="your_lib", display_name="YourLib")
        self.log_file = self.script_dir / "your_lib_bench.log"

    def run_benchmarks(self) -> None:
        """Execute benchmark binary and log output."""
        # Load config if needed
        config_file = self.script_dir.parent / "configs" / "your_lib.json"

        # Your benchmark execution logic here
        pass

    def parse_results(self) -> Tuple[List[Dict[str, Any]], PrettyTable]:
        """Parse log file and return (test_results, table)."""
        # Your parsing logic here
        # Use self.create_test_result() to build result dictionaries
        pass


if __name__ == "__main__":
    run_benchmark_main(YourBenchmark())  # Handles sys.exit() internally

2. Add to Benchmark Test Matrix

Edit tests/extended_tests/benchmark/benchmark_test_matrix.py:

"your_benchmark": {
    "job_name": "your_benchmark",
    "fetch_artifact_args": "--your-lib --tests",
    "timeout_minutes": 60,
    "test_script": f"python {_get_benchmark_script_path('test_your_benchmark.py')}",
    "platform": ["linux", "windows"],  # Supported platforms
    "total_shards": 1,
    # TODO: Remove xfail once dedicated performance servers are added
    "expect_failure": True,
},

The benchmark will automatically be included in nightly CI runs. See CI/CD Integration for details.

3. Test Locally

# Set environment variables
export THEROCK_BIN_DIR=/path/to/build/bin
export ARTIFACT_RUN_ID=local-test
export AMDGPU_FAMILIES=gfx950-dcgpu

# Run the benchmark
python3 tests/extended_tests/benchmark/scripts/test_your_benchmark.py