TheRock/tests/extended_tests/benchmark/README.md

# Benchmark Testing Framework

Automated benchmark testing framework for ROCm libraries with system detection, results collection, and performance tracking.

> **Prerequisites:** See [Extended Tests Framework Overview](../README.md) for environment setup, CI/CD integration, and general architecture.

## Table of Contents

- [Features](#features)
- [Project Structure](#project-structure)
- [Quick Start](#quick-start)
- [Architecture](#architecture)
- [Adding New Benchmarks](#adding-new-benchmarks)

## Features

- **Automated Benchmark Execution** - ROCfft, ROCrand, ROCsolver, hipBLASLt, rocBLAS, RCCL
- **System Auto-Detection** - Hardware, OS, GPU, and ROCm version detection
- **Distributed Testing** - Multi-GPU RCCL benchmarks (requires OpenMPI in Docker image)
- **Results Management** - Local storage (JSON) and API upload with retry logic
- **Performance Tracking** - LKG (Last Known Good) comparison
- **Comprehensive Logging** - File rotation and configurable log levels
- **Modular Architecture** - Extensible design for adding new benchmarks
- **CI/CD Integration** - Parallel execution with regular tests in nightly CI

## Project Structure

```
extended_tests/benchmark/                       # Benchmark test directory
├── scripts/                     # Benchmark test implementations
│   ├── benchmark_base.py        # Base class for all benchmarks
│   ├── test_hipblaslt_benchmark.py
│   ├── test_rccl_benchmark.py
│   ├── test_rocblas_benchmark.py
│   ├── test_rocfft_benchmark.py
│   ├── test_rocrand_benchmark.py
│   └── test_rocsolver_benchmark.py
│
├── configs/                     # Benchmark-specific configs
│   ├── hipblaslt.json           # hipBLASLt benchmark config
│   ├── rccl.json                # RCCL benchmark config
│   ├── rocblas.json             # rocBLAS benchmark config
│   └── rocfft.json              # ROCfft benchmark config
│
├── benchmark_test_matrix.py     # Benchmark matrix definitions
└── README.md                    # This file
```

> **Note:** For the overall extended_tests structure including shared utilities (`utils/`), configuration (`configs/`), and other test types, see the [main extended_tests README](../README.md).

## Quick Start

### Available Benchmarks

The following benchmark tests are defined in `tests/extended_tests/benchmark/benchmark_test_matrix.py`:

| Test Name         | Library   | Platform       | Timeout | Shards |
| ----------------- | --------- | -------------- | ------- | ------ |
| `hipblaslt_bench` | hipBLASLt | Linux, Windows | 60 min  | 1      |
| `rccl_bench`      | RCCL      | Linux          | 60 min  | 1      |
| `rocblas_bench`   | rocBLAS   | Linux          | 60 min  | 1      |
| `rocfft_bench`    | ROCfft    | Linux, Windows | 60 min  | 1      |
| `rocrand_bench`   | ROCrand   | Linux, Windows | 60 min  | 1      |
| `rocsolver_bench` | ROCsolver | Linux, Windows | 60 min  | 1      |

**GPU Family Support:**

| GPU Family | Platform | Architecture          | Benchmark Supported | Benchmark CI Status  |
| ---------- | -------- | --------------------- | ------------------- | -------------------- |
| `gfx94x`   | Linux    | MI300X/MI325X (CDNA3) | Yes                 | Enabled (nightly CI) |
| `gfx1151`  | Windows  | RDNA 3.5              | Yes                 | Enabled (nightly CI) |
| `gfx950`   | Linux    | MI355X (CDNA4)        | Yes                 | Not enabled          |
| `gfx110x`  | Windows  | RDNA 2                | Yes                 | Not enabled          |
| `gfx110x`  | Linux    | RDNA 2                | Yes                 | Not enabled          |
| `gfx120x`  | Linux    | RDNA 3                | Yes                 | Not enabled          |
| `gfx120x`  | Windows  | RDNA 3                | Yes                 | Not enabled          |
| `gfx90x`   | Linux    | MI200 (CDNA2)         | Yes                 | Not enabled          |
| `gfx1151`  | Linux    | RDNA 3.5              | Yes                 | Not enabled          |

> **Note:** All benchmarks are **architecture-agnostic** and support any ROCm-compatible GPU. The table above lists GPU families actively used in CI testing. To add support for additional GPU families, update [`amdgpu_family_matrix.py`](../amdgpu_family_matrix.py) with appropriate `benchmark-runs-on` runners.

## Architecture

### Benchmark Script Execution Flow

```
1. Initialize ExtendedTestClient
   ↓ Auto-detect system (GPU, OS, ROCm version)
   ↓ Load configuration from config.yml

2. Run Benchmarks
   ↓ Execute benchmark binary
   ↓ Capture output to log file

3. Parse Results
   ↓ Extract metrics from log file
   ↓ Structure data according to schema

4. Upload Results
   ↓ Submit to API (with retry)
   ↓ Save JSON locally

5. Compare with LKG
   ↓ Fetch last known good results
   ↓ Calculate performance delta

6. Report Results
   ↓ Display formatted table
   ↓ Append to GitHub Actions step summary
   ↓ Return exit code (0=success, 1=failure)
```

## Adding New Benchmarks

To add a new benchmark test to the nightly CI:

### 1. Create Benchmark Script

Create `extended_tests/benchmark/scripts/test_your_benchmark.py`. Reference existing benchmarks like `test_rocfft_benchmark.py` as a template.

Key components:

- Inherit from `BenchmarkBase` class
- Implement `run_benchmarks()` - executes binary and logs output
- Implement `parse_results()` - parses logs and returns structured data
- Results are automatically uploaded to API via base class

Example:

```python
import sys
from pathlib import Path
from typing import Dict, List, Tuple, Any
from prettytable import PrettyTable

sys.path.insert(0, str(Path(__file__).resolve().parents[2]))  # For extended_tests/utils
sys.path.insert(0, str(Path(__file__).parent))  # For benchmark_base
from benchmark_base import BenchmarkBase, run_benchmark_main
from utils.logger import log


class YourBenchmark(BenchmarkBase):
    def __init__(self):
        super().__init__(benchmark_name="your_lib", display_name="YourLib")
        self.log_file = self.script_dir / "your_lib_bench.log"

    def run_benchmarks(self) -> None:
        """Execute benchmark binary and log output."""
        # Load config if needed
        config_file = self.script_dir.parent / "configs" / "your_lib.json"

        # Your benchmark execution logic here
        pass

    def parse_results(self) -> Tuple[List[Dict[str, Any]], PrettyTable]:
        """Parse log file and return (test_results, table)."""
        # Your parsing logic here
        # Use self.create_test_result() to build result dictionaries
        pass


if __name__ == "__main__":
    run_benchmark_main(YourBenchmark())  # Handles sys.exit() internally
```

### 2. Add to Benchmark Test Matrix

Edit `tests/extended_tests/benchmark/benchmark_test_matrix.py`:

```python
"your_benchmark": {
    "job_name": "your_benchmark",
    "fetch_artifact_args": "--your-lib --tests",
    "timeout_minutes": 60,
    "test_script": f"python {_get_benchmark_script_path('test_your_benchmark.py')}",
    "platform": ["linux", "windows"],  # Supported platforms
    "total_shards": 1,
    # TODO: Remove xfail once dedicated performance servers are added
    "expect_failure": True,
},
```

The benchmark will automatically be included in nightly CI runs. See [CI/CD Integration](../README.md#cicd-integration) for details.

### 3. Test Locally

```bash
# Set environment variables
export THEROCK_BIN_DIR=/path/to/build/bin
export ARTIFACT_RUN_ID=local-test
export AMDGPU_FAMILIES=gfx950-dcgpu

# Run the benchmark
python3 tests/extended_tests/benchmark/scripts/test_your_benchmark.py
```