TheRock/rocm-systems/shared/ctest/README.md

ROCm Systems CTest Integration Architecture

This directory contains the shared CTest integration files for organizing and executing tests across ROCm system projects using YAML-based test categorization.

Directory Structure

shared/ctest/
├── README.md                      # This file - architecture documentation
├── TestCategories.cmake           # CMake module for test category integration
└── parse_ctest_categories.py      # Python parser for YAML to CMake conversion

Files:

• TestCategories.cmake - CMake module with apply_ctest_category_labels() function
• parse_ctest_categories.py - Python parser for YAML to CMake conversion

Architecture Overview

The CTest integration provides a label-based system for organizing pre-existing CTest tests into tiered categories. Unlike gtest-filter approaches used in rocm-libraries that create new test targets, this system applies CTest LABELS to tests already registered via add_test(), enabling flexible filtering with ctest -L.

Core Components

1. test_categories.yaml (Project-specific)

Located in each project's test directory (e.g., projects/rocprofiler-compute/tests/test_categories.yaml).

Defines test organization:

• Test categories with tiered label inheritance
• Per-category exclusions
• Custom labels per category
• Timeout settings

2. parse_ctest_categories.py (Shared)

Python script that:

• Parses YAML configuration files
• Applies tiered label inheritance (quick → standard → comprehensive → full)(so that a test added for quick category will be executed for other categories as well)
• Supports exact test name matching and CMake regex patterns
• Supports matching tests by existing labels (test_labels)
• Generates CMake set_property() code to apply labels
• Deduplicates labels across all tests
• Optionally appends generated code to an install-time test file

3. TestCategories.cmake (Shared)

CMake module providing:

• apply_ctest_category_labels() function for projects
• Python3 interpreter detection
• Input validation and error handling
• Optional install_test_file parameter for install-time label support

Execution Flow

flowchart TD
    A[Project CMakeLists.txt] -->|"add_test() calls<br/>define tests"| B[Tests Registered in CTest]
    A -->|include| C[TestCategories.cmake]
    A -->|call| D["apply_ctest_category_labels(yaml)"]

    C -->|find_package| E[Python3]
    D -->|execute_process| F[parse_ctest_categories.py]

    F -->|read| G[test_categories.yaml]
    F -->|apply| H[Tiered Label Inheritance]
    F -->|generate| I["CMake set_property() Code"]

    I -->|"file(WRITE)"| J["build/test_categories.cmake"]
    J -->|include| K["Labels Applied to Existing Tests"]

    B --> K
    K -->|"ctest -L quick"| L[Run Quick Tests]
    K -->|"ctest -L comprehensive"| M[Run Comprehensive Tests]

    style A fill:#e1f5ff,color:#000
    style F fill:#fff4e1,color:#000
    style J fill:#e8f5e9,color:#000
    style K fill:#f3e5f5,color:#000

Tiered Label Inheritance

Categories follow a tiered inheritance model where lower tiers automatically include higher-tier labels:

Category	Labels Applied
quick	quick, standard, comprehensive, full
standard	standard, comprehensive, full
comprehensive	comprehensive, full
full	full

This means a test in the quick category is also included when running ctest -L standard, ctest -L comprehensive, or ctest -L full. Any custom labels defined in the YAML are appended after the tier labels.

YAML Configuration Format

Basic Structure

test_categories:
  category_name:
    description: "Human-readable description"
    test_patterns:
      - test_exact_name        # Exact CTest test name
      - test_regex.*pattern    # CMake regex pattern (auto-detected)
    test_labels:
      - "existing_label"       # Match tests that already have this label
    exclude:
      - test_to_exclude        # Applied with "{category}_exclude" label
    labels:
      - "custom_label"         # Additional labels beyond tier labels

general_exclude:
  exclude_section_name:
    test_patterns:
      - test_to_exclude
    labels:
      - "exclude_label"

execution_settings:
  default_timeout: 300
  timeout_multiplier: 1
  category_timeouts:
    quick: 300
    standard: 1800
    comprehensive: 7200

Example (rocprofiler-compute)

test_categories:
  quick:
    description: "Fast sanity checks - Run pre-commit (target: < 5 min)"
    test_patterns:
      - test_autogen_config
      - test_roofline_calc_ai_analyze
      - test_utils
      - test_tui_components
    exclude:
      - test_profile_live_attach_detach
    labels:
      - "quick"
      - "pre-commit"
      - "smoke"

  standard:
    description: "Core functionality - Run on every PR (target: < 30 min)"
    test_patterns:
      - test_autogen_config
      - test_utils
      - test_analyze_workloads
      - test_metric_validation
    exclude:
      - test_profile_live_attach_detach
    labels:
      - "standard"
      - "pr"

  comprehensive:
    description: "Extended testing - Run nightly (target: < 2 hours)"
    test_patterns:
      - test_profile_dispatch
      - test_profile_kernel_execution
      - test_profile_roofline_1
      # ... all tests
    exclude:
      - test_profile_live_attach_detach
    labels:
      - "comprehensive"
      - "nightly"

execution_settings:
  default_timeout: 300
  timeout_multiplier: 1
  category_timeouts:
    quick: 300
    standard: 1800
    comprehensive: 7200

Field Reference

Field	Required	Description
test_patterns	Yes	List of CTest test names or CMake regex patterns to match
test_labels	No	List of existing labels to match — tests carrying any of these labels get the category's labels
exclude	No	Test names/patterns to tag with {category}_exclude label
labels	No	Custom labels appended after tier-inherited labels
description	No	Human-readable description (ignored by parser)

Test Matching

The parser supports two matching strategies, auto-detected per pattern:

Exact match — test name contains no regex metacharacters:

test_patterns:
  - test_utils              # Matches the CTest test named exactly "test_utils"

Generated CMake:

foreach(_test test_utils)
    if(TEST ${_test})
        set_property(TEST ${_test} APPEND PROPERTY LABELS "quick;standard;comprehensive;full")
    endif()
endforeach()

Regex match — test name contains regex metacharacters (.*+?^${}()|[]\):

test_patterns:
  - test_profile_.*         # Matches all tests starting with "test_profile_"

Generated CMake:

get_property(_all_tests DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY TESTS)
foreach(_test ${_all_tests})
    if(_test MATCHES "^test_profile_.*$")
        set_property(TEST ${_test} APPEND PROPERTY LABELS "quick;standard;comprehensive;full")
    endif()
endforeach()

Label-Based Matching (test_labels)

In addition to matching by test name, categories can match tests by their existing labels:

test_categories:
  quick:
    test_labels:
      - "unit"              # All tests already labeled "unit" get quick-tier labels
    test_patterns:
      - test_extra_quick

This is useful when tests are already labeled in CMakeLists.txt via set_tests_properties() and you want to include them in a category without listing each name.

General Excludes

The general_exclude section applies labels directly to test patterns without tier inheritance:

general_exclude:
  exclude_flaky:
    test_patterns:
      - test_intermittent_failure
    labels:
      - "flaky"
      - "exclude_flaky"

This can be used with ctest -LE exclude_flaky to skip these tests.

Integration Guide

Step 1: Create test_categories.yaml

Create test_categories.yaml in your project's test directory:

projects/myproject/tests/test_categories.yaml

Step 2: Include in CMakeLists.txt

In your project's CMakeLists.txt, after all add_test() calls:

# Set path to the YAML file
set(_test_categories_yaml "${CMAKE_CURRENT_SOURCE_DIR}/tests/test_categories.yaml")

# Include the shared CTest module
include(${ROCM_SYSTEMS_ROOT}/shared/ctest/TestCategories.cmake)

# Apply category labels to existing tests
if(EXISTS "${_test_categories_yaml}")
    message(STATUS "YAML-based test categorization")
    apply_ctest_category_labels("${_test_categories_yaml}")
else()
    message(STATUS "No test categorization (YAML missing)")
endif()

Important: apply_ctest_category_labels() must be called after all add_test() calls, since it uses if(TEST ...) and get_property(... PROPERTY TESTS) to find existing tests.

Step 3: Build and Test

# Configure
cmake -B build ..

# Run quick sanity checks
ctest --test-dir build -L quick

# Run standard PR tests
ctest --test-dir build -L standard

# Run comprehensive nightly tests
ctest --test-dir build -L comprehensive

# Exclude specific tests
ctest --test-dir build -L quick -LE quick_exclude

# List all tests with their labels
ctest --test-dir build -N --show-only=json-v1

# Verbose output
ctest --test-dir build -L quick -V

Optional: Install-Time Test File

apply_ctest_category_labels() accepts an optional second argument to append the generated label code to a file for install-time use:

apply_ctest_category_labels(
    "${_test_categories_yaml}"
    "${CMAKE_CURRENT_BINARY_DIR}/install_test_labels.cmake"  # appended to this file
)

Integrations

• rocprofiler-compute - test_categories.yaml | CMakeLists.txt