TheRock/build_tools/github_actions/test_executable_scripts/README.md

Test Runner

test_runner.py is a generic test runner used by GitHub Actions to run component tests (e.g. MIOpen). It relies on a standardized CTest naming and labeling scheme so that only tests valid for the current scenario are run.

This document describes how the mechanism works, what the component must provide, and how it ties into CI.

---

Overview

1. The script is invoked by the Test component workflow with env vars set (component name, GPU arch, test type, sharding).
2. The script expects the component under test to have a ctest based interface, with labels corresponding to test categories like quick, standard, comprehensive and full(which can be run for scenarios like quick test, pre-commit etc)
3. The component can opt to always exclude some tests or based on condition like OS and GPU where the test is getting run
4. The script discovers which GPU-specific test suites exist by running ctest --print-labels and parsing labels of the form ex_gpu_{gpu_arch}.
5. It builds and runs a ctest command with the right labels and options (category, parallelism, sharding, etc.).

No component-specific logic is required beyond using the expected test names and CTest labels.

---

Test Naming and Labels (Component Contract)

For this runner to work, the component’s CTest configuration must follow this convention.

CTest labels (discovery)

The script runs ctest --print-labels --test-dir <component> and collects every label that starts with ex_gpu_. The suffix is treated as the GPU architecture (e.g. ex_gpu_gfx110X → gfx110X, ex_gpu_gfx950 → gfx950). Only labels whose suffix starts with gfx are used.

CTest labels (selection)

• Category: Tests are selected by label for the run (e.g. “run all tests with label quick” or standard). The script uses -L <category>.
• GPU-specific suites: For a given gpu_arch, the script looks for a label ex_gpu_{gpu_arch} (e.g. ex_gpu_gfx1150, ex_gpu_gfx11X). It uses -L ex_gpu_<arch> when it finds a matching label, or -LE ex_gpu to exclude all GPU-specific tests when no match is found or when no GPU is specified.

So the component must assign GPU-specific tests the label ex_gpu_{gpu_arch} (and typically a category label like quick or standard).

---

Execution Flow

1. Resolve component directory Map TEST_COMPONENT (e.g. miopen) to the test directory name (e.g. MIOpen) via COMPONENT_DIR_MAPPING. Fail if the test directory does not exist under THEROCK_BIN_DIR.
2. Discover GPU suites Run ctest --print-labels --test-dir {THEROCK_BIN_DIR}/{TEST_COMPONENT}. Collect every label that starts with ex_gpu_ and whose suffix starts with gfx (e.g. ex_gpu_gfx110X → gfx110X). This yields the set of available GPU architectures for which a label exists. These are the set of GPU's for which some test exclusions apply - i.e there are some tests which should not be run on these GPU models. The exp_gpu_ labels with these gpu models points to the ctest entries where the tests are excluded.
3. Choose category From TEST_TYPE: quick → full, else → standard.
4. Resolve GPU arch Parse AMDGPU_FAMILIES for the first gfx... token (e.g. gfx1151). If missing or generic, the script will exclude all GPU-specific tests (-LE ex_gpu).
5. Match GPU to suite Using find_matching_gpu_arch() (defined in test_runner.py):

• Prefer exact match in the discovered set (e.g. gfx1151).
• Else try wildcard patterns from most to least specific (e.g. for gfx1151: gfx115X, then gfx11X).
• If a match is found, add -L ex_gpu_{matching_arch}; otherwise add -LE ex_gpu.

6. Build ctest command

• ctest -L <category> (and optionally -L ex_gpu_<arch> or -LE ex_gpu).
• Common options: --output-on-failure, --parallel <N>, --test-dir, -V, --tests-information <SHARD_INDEX>,<TOTAL_SHARDS>.
• Parallelism: default 1 (serial); can be adjusted according to AMDGPU_FAMILIES

7. Run ctest Execute the command in THEROCK_DIR with the environment that includes ROCM_PATH, GTEST_SHARD_INDEX, and GTEST_TOTAL_SHARDS.

---

CI Integration

• Workflow: .github/workflows/test_component.yml runs the test step with env vars such as TEST_COMPONENT, TEST_TYPE, AMDGPU_FAMILIES, SHARD_INDEX, TOTAL_SHARDS.
• Test script: For components that use this mechanism, the test script is set to python .../test_runner.py in build_tools/github_actions/fetch_test_configurations.py (e.g. MIOpen).
• Sharding: The workflow matrix uses shard_arr from the same config; the script passes sharding to ctest via --tests-information and to GTest via GTEST_SHARD_INDEX / GTEST_TOTAL_SHARDS.

---

Adding a New Component to This Runner

1. In the component (e.g. CMake/CTest):

• Name GPU-specific tests {target}_{category}_{gpu_arch}_suite.
• Assign labels ex_gpu_{gpu_arch} and the category label (quick / standard or equivalent).

2. In TheRock:

• In test_filters.py, add the job name → directory mapping in COMPONENT_DIR_MAPPING.
• In fetch_test_configurations.py, set the component’s test_script to python .../test_filters.py and set job_name (and shards, timeout, etc.) as needed.

After that, the generic flow (discovery → match GPU → run ctest with the right labels) applies without further changes to test_filters.py for that component.