TheRock/tests/extended_tests/functional/README.md

Functional Testing Framework

Functional tests validate correctness and verify expected behavior without performance measurements.

Prerequisites: See Extended Tests Framework Overview for environment setup, CI/CD integration, and general architecture.

Table of Contents

• Available Tests
• CI Configuration
• Project Structure
• How Functional Tests Work
• Adding New Functional Tests

Available Tests

Test Script	Library	Platform	Timeout	Description
test_miopendriver_conv.py	MIOpen	Linux	30 min	Convolution forward/backward tests

CI Configuration

• Test Matrix: See functional_test_matrix.py for complete test definitions (platforms, timeouts, artifacts needed)
• Execution: All functional tests run in nightly CI builds only
• Architecture Support: Tests are architecture-agnostic and support any ROCm-compatible GPU
• Architecture Exclusions: If a specific GPU architecture is not supported for a test, use the exclude_family field in the test matrix to skip that architecture/platform combination (see fetch_test_configurations.py for filtering logic)

Project Structure

functional/
├── scripts/
│   ├── functional_base.py        # Base class for all functional tests
│   └── test_miopendriver_conv.py # MIOpen convolution test
│
├── configs/
│   └── miopendriver_conv.json    # Test configuration
│
├── functional_test_matrix.py     # CI test matrix definitions
└── README.md                     # This file

How Functional Tests Work

Result Tables

Functional tests generate two tables:

Detailed Table: One row per test case

+--------------+--------------------+--------+
| TestSuite    | TestCase           | Status |
+--------------+--------------------+--------+
| Forward_Conv | Forward_Conv_case1 | PASS   |
| Forward_Conv | Forward_Conv_case2 | PASS   |
+--------------+--------------------+--------+

Summary Table: Overall statistics

+-------------------+-------------------+--------+--------+---------+---------+-------------------+
| Total TestSuites  | Total TestCases   | Passed | Failed | Errored | Skipped |   Final Result    |
+-------------------+-------------------+--------+--------+---------+---------+-------------------+
|         2         |         18        |   18   |   0    |    0    |    0    |        PASS       |
+-------------------+-------------------+--------+--------+---------+---------+-------------------+

Adding New Functional Tests

1. Create Test Script

Create scripts/test_your_test.py:

"""YourTest Functional Test"""

import sys
from pathlib import Path
from typing import Any, Dict, List

sys.path.insert(0, str(Path(__file__).resolve().parents[2]))  # For utils
sys.path.insert(0, str(Path(__file__).resolve().parent))  # For functional_base
from functional_base import FunctionalBase, run_functional_main
from utils.logger import log
from utils.exceptions import TestExecutionError


class YourTest(FunctionalBase):
    """YourTest functional test."""

    def __init__(self):
        super().__init__(test_name="your_test", display_name="YourTest Functional")
        self.results_json = self.script_dir / "your_test_results.json"

        # Load test configuration from JSON
        config = self.load_config("your_test.json")
        self.test_cases = config.get("test_cases", [])

    def run_tests(self) -> None:
        """Run functional tests and save results to JSON."""
        log.info(f"Running {self.display_name}")

        # Optional: Get GPU architecture for GPU-specific behavior
        from github_actions_api import get_first_gpu_architecture

        gfx_id = get_first_gpu_architecture(therock_bin_dir=self.therock_bin_dir)

        all_results = []
        for test_case in self.test_cases:
            # Execute test, capture result
            status = "PASS"  # or "FAIL" based on return code
            all_results.append(
                {
                    "test_case": test_case["name"],
                    "status": status,
                }
            )

        # Save results to JSON
        with open(self.results_json, "w") as f:
            json.dump(all_results, f, indent=2)

    def parse_results(self) -> List[Dict[str, Any]]:
        """Parse results and return test_results list."""
        log.info("Parsing Results")

        test_results = []

        with open(self.results_json, "r") as f:
            json_results = json.load(f)

        for result in json_results:
            test_results.append(
                self.create_test_result(
                    test_name=self.test_name,
                    subtest_name=result["test_case"],
                    status=result["status"],
                    suite=result.get("test_suite", "default"),
                )
            )

        return test_results


if __name__ == "__main__":
    run_functional_main(YourTest())

Required Methods:

• run_tests() → Execute tests and save results to JSON
• parse_results() → Returns List[Dict] of test results
  • Must use self.create_test_result(test_name, subtest_name, status, suite, **kwargs)
  • Status must be: "PASS", "FAIL", "ERROR", or "SKIP"
  • Base class generates detailed table and calculates num_suites automatically

Available Helper Methods (inherited from ExtendedTestBase):

• self.load_config(filename) → Load JSON config from configs/ directory
• self.execute_command(cmd, env=env) → Execute command with streaming output
• self.create_test_result(...) → Create standardized result dictionary
• self.get_rocm_env() → Get environment with LD_LIBRARY_PATH for ROCm libraries

2. Create Configuration File

Create configs/your_test.json:

{
  "description": "YourTest Configuration",
  "test_cases": [
    {"name": "test_case_1", "command": "..."},
    {"name": "test_case_2", "command": "..."}
  ],
  "gpu_specific_flags": {
    "gfx906": {"flags": "-V 0"}
  }
}

3. Add to Test Matrix

Edit functional_test_matrix.py:

"test_your_test": {
    "job_name": "test_your_test",
    "fetch_artifact_args": "--your-lib --tests",
    "timeout_minutes": 30,
    "test_script": f"python {_get_functional_script_path('test_your_test.py')}",
    "platform": ["linux"],
    "total_shards": 1,
},

4. Test Locally

Environment Setup: See Extended Tests Framework Overview for required environment variables and setup instructions.

# Run the test
python3 tests/extended_tests/functional/scripts/test_your_test.py