TheRock/tests/extended_tests/README.md

TheRock Extended Tests Framework

Unified testing framework for TheRock ROCm distribution, supporting benchmark and functional testing with automated execution, system detection, and results management.

Table of Contents

• Overview
• Quick Start
• Project Structure
• CI/CD Integration
• Architecture

Overview

The test framework provides infrastructure for two test types:

Test Type	Purpose	Result Types	When to Use
Benchmark	Performance regression detection	PASS/FAIL/UNKNOWN	Prevent performance degradation (nightly CI)
Functional	Correctness validation	PASS/FAIL/ERROR/SKIP	Verify expected behavior (nightly CI)

Key Features

• Shared Infrastructure - Common utilities, configuration, and results management
• System Auto-Detection - Hardware, OS, GPU, and ROCm version detection
• Results Management - Local storage (JSON) and API upload with retry logic
• Comprehensive Logging - File rotation and configurable log levels
• Error Handling - Custom exceptions with clear, actionable messages
• Modular Architecture - Extensible design for adding new test types
• CI/CD Integration - Parallel execution in nightly CI

Quick Start

Environment Setup

All tests require these environment variables. Note: These are automatically configured in CI runs. For local testing, adjust values based on your setup.

Setting Up Environment: See Test Environment Reproduction for instructions on reproducing the CI test environment, including how to obtain TheRock build artifacts from a CI run and get the ARTIFACT_RUN_ID (same as CI_RUN_ID in the document).

# Required: Path to TheRock bin/ directory (see above for how to obtain)
export THEROCK_BIN_DIR=/path/to/therock/build/bin

# Required: GitHub Actions run ID (from CI run URL, e.g., https://github.com/ROCm/TheRock/actions/runs/16948046392)
export ARTIFACT_RUN_ID=16948046392

# Required: Update to match your GPU family (e.g., gfx908, gfx90a, gfx942, gfx950-dcgpu)
export AMDGPU_FAMILIES=gfx950-dcgpu

# Optional: Control GPU visibility on multi-GPU nodes (e.g., ROCR_VISIBLE_DEVICES=0)
# export ROCR_VISIBLE_DEVICES=0

Running Tests

See test-specific READMEs for detailed instructions and examples:

• Benchmark Tests - Performance regression testing
• Functional Tests - Correctness validation testing

Project Structure

extended_tests/
├── __init__.py
├── README.md                       # This file
│
├── configs/                        # SHARED configuration
│   └── config.yml                 # Framework config (logging, API, execution)
│
├── benchmark/                      # Benchmark tests (LKG comparison)
│   ├── scripts/                   # Test implementations
│   │   ├── benchmark_base.py      # Base class with LKG logic
│   │   └── test_*_benchmark.py    # Individual benchmark tests
│   ├── configs/                   # Test-specific configurations
│   │   ├── hipblaslt.json
│   │   └── rocfft.json
│   ├── benchmark_test_matrix.py   # Benchmark test matrix
│   └── README.md                  # Benchmark-specific docs
│
├── functional/                    # Functional/correctness tests
│   ├── scripts/                   # Test implementations
│   │   ├── functional_base.py     # Base class for functional tests
│   │   └── test_*.py              # Individual tests
│   ├── configs/                   # Test-specific configurations
│   │   └── *.json
│   ├── functional_test_matrix.py  # Functional test matrix
│   └── README.md                  # Functional-specific docs
│
└── utils/                          # SHARED utilities for all test types
    ├── extended_test_base.py      # ExtendedTestBase - shared base class for all tests
    ├── extended_test_client.py    # ExtendedTestClient - system detection & result reporting
    ├── exceptions.py              # Custom exception classes
    │   ├── BenchmarkExecutionError   # Execution/parsing failures
    │   ├── BenchmarkResultError      # Result validation failures
    │   └── FrameworkException        # Base exception
    │
    ├── logger.py                  # Logging utilities
    ├── constants.py               # Global constants
    │
    ├── config/                    # Configuration parsers
    │   ├── config_parser.py
    │   ├── config_validator.py
    │   └── config_helper.py
    │
    ├── results/                   # Results handling & LKG
    │   ├── results_api.py        # API for storing/retrieving results
    │   └── results_handler.py    # Process and format results
    │
    └── system/                    # Hardware & ROCm detection
        ├── hardware.py           # GPU detection and capabilities
        ├── platform.py           # Platform-specific utilities
        └── rocm_detector.py      # ROCm version detection

CI/CD Integration

Test Execution Schedule

Workflow Trigger	Benchmark Tests	Functional Tests
Pull Request (PR)	Skipped	Skipped
Nightly CI (scheduled)	Run (parallel)	Run (parallel)
Push to main	Skipped	Skipped

Parallel Execution Architecture

Tests run in parallel for faster CI execution:

ci_nightly.yml
  └─ ci_linux.yml / ci_windows.yml
      ├─ build_artifacts
      │
      └─ test_artifacts
          └─ Component + Functional + Benchmark tests
              (runner auto-selected per component)

Workflow Files:

• .github/workflows/ci_nightly.yml - Nightly CI orchestration
• .github/workflows/ci_linux.yml / ci_windows.yml - Platform-specific CI logic
• .github/workflows/test_artifacts.yml - All test execution (component, functional, benchmark)

Key Differences:

• Component Tests: Run on all PRs (smoke) and nightly (full), use regular runners (test_runs_on)
• Benchmark Tests: Run only when globally enabled via run_extended_tests == true AND a dedicated performance runner is available (benchmark-runs-on in amdgpu_family_matrix.py). Runner is auto-selected per component.
• Functional Tests: Run only when globally enabled via run_extended_tests == true, use regular runners (test_runs_on)

Architecture

Common Test Execution Flow

All tests follow this pattern:

1. Initialize - Auto-detect system (GPU, ROCm), load configuration, setup logging
2. Execute - Run test binaries/scripts, capture output to log files
3. Parse - Extract metrics/results from logs, structure data
4. Process - Type-specific validation (LKG comparison or correctness check)
5. Report - Display results, upload to API, update GitHub Actions summary

Implementation Details

See test-specific READMEs for detailed implementation guides:

• Benchmark Tests - LKG comparison logic and adding new benchmarks
• Functional Tests - Correctness validation and adding new tests
• Shared Utils - Common utilities, exceptions, and helpers