TheRock/rocm-systems/projects/rocprofiler-systems/tests/pytest/DEVELOPMENT.md

rocprofiler-systems Test Infrastructure

rocprofsys Package

This package contains the code infrastructure required to run pytests. It contains 7 main components:

cache.py

A persistent, cross-process cache for expensive test-setup probes. CTest spawns many short-lived pytest processes; without caching, each would re-run the same rocminfo / amd-smi / mpicc / ldd / python --version subprocesses. cache.py stores probe results in a single per-build-tree JSON file so each probe runs once per CTest session. See Caching Expensive Probes for how (and when) to use it.

capabilities.py

Contains functions that allow pytests to discover what the current system supports. Everything here is lazy-initialized and, where the value is session-constant, cached across processes (see Caching Expensive Probes). This contains information on the system that is not required for every test. Should only be accessed through an instance of a RocprofsysConfig (see config.py).

config.py

Contains configuration information used by every test. This includes the location of core binaries, an instance of the SystemCapabilities class, the computed LD_LIBRARY_PATH and some core helper functions.

environment.py

Contains TestEnvironment, which models the per-run environment as three layers ordered from lowest to highest precedence: base (framework defaults for the test type), test (settings supplied by the test, plus framework-injected values like ROCPROFSYS_OUTPUT_PATH), and user (the inherited shell environment). A runner builds these layers and merges them (highest layer wins) into the environment handed to the subprocess.

gpu.py

Contains information about the GPU on the system. Also finds rocminfo and llvm-objdump/roc-obj-ls binaries used to check if a test binary was compiled for the proper architecture.

runners.py

Contains classes corresponding to each type of test. For example, a runtime instrumentation test would run through the RuntimeInstrumentRunner class. These runner classes handle test execution and return an instance of TestResult, which contains the information one would need to evaluate whether a test has passed or failed.

validators.py

Contains wrappers for the validation scripts and other helper functions that can be used during a test. They return a ValidationResult which can be used to determine if a test has failed or passed. In practice, the assert_ fixtures in conftest.py wrap these validators and should be used instead.

Caching Expensive Probes

cache.py exists for one reason: CTest runs each test in its own short-lived pytest process, so any value computed by shelling out to a subprocess would otherwise be recomputed hundreds of times. The rocprofiler-systems-pytest-config setup fixture regenerates the cache once at the start of a ctest run (writing every value to a shared JSON file under /tmp/$USER/), the cleanup fixture deletes it at the end, and every test process in between reads the values back instead of re-probing.

Rule of thumb: if your code launches a subprocess (or does an expensive filesystem scan) to compute a value that is constant for the whole build tree, cache it here. In-process memoization alone (functools.lru_cache / cached_property) does not help across CTest's many single-test processes — the persistent cache is what makes the probe run once per session instead of once per process.

The API

Two decorators, both in cache.py:

Decorator	Use for	Keyed on
persistent_cache("ns.name")	a function/method you call	namespace + call args
persistent_cached_property	a computed attribute (obj.x)	class + attribute name

• System capabilities (most common): add the probe as a persistent_cached_property on SystemCapabilities (capabilities.py). Config setup introspects the class and warms it automatically, so a new capability needs no extra wiring.
• Standalone functions: decorate with @persistent_cache("ns.name"); arguments become part of the key. For an instance method that also takes arguments, add method=True so self is dropped from the key (e.g. target_support_mpi(target_path)).
• Return types: None/bool/int/float/str/Path/tuple/set/list/dict, and any dataclass whose fields are (recursively) those, serialize automatically. Supply to_cache/from_cache only for something exotic — a non-dataclass that isn't JSON-native.

When NOT to persist

A persistent_cached_property key ignores per-process inputs, so never use it for a value that varies with a CLI flag or other per-process state (e.g. --python-versions) — one process's result would leak into another. Pass such inputs as function arguments (so they enter the key) or don't cache. See supported_python_versions in capabilities.py.

Safety

Reads and I/O are fail-open: a corrupt/stale entry or an I/O error just recomputes, so the cache can't break a test for environmental reasons. But an unserializable value raises SerializationError, surfacing the bug at config-setup instead of silently re-probing in every process. Set ROCPROFSYS_DISABLE_TEST_CACHE=1 to disable caching entirely.

conftest.py

conftest.py contains all the logic required for pytest to parse the test_*.py code and generate tests.

Core Classes

• RocprofsysTest: Base class for all test classes. It auto-injects common fixtures (run_test, assert_regex, test_output_dir, etc.) onto self via its _setup fixture, so test methods can call self.run_test(...), self.assert_regex(...), etc. directly.

Key Hooks

• pytest_configure: Registers all custom markers and sets up CLI options.
• pytest_collection_modifyitems: Handles test skipping based on marker conditions.
• _generate_ctest_definitions: Generates a CTestTestfile.cmake from collected pytest items.

Subtests (Validation Fixtures)

These fixtures run as subtests, meaning multiple validations within a single test are independently reported (one can fail without blocking others). They are automatically injected into RocprofsysTest via its _setup fixture, so test methods access them as self.assert_regex(...), self.assert_perfetto(...), etc.

Fixture	Description
assert_regex	Validates test output against pass/fail regex patterns. Patterns can be per-mode (e.g., different patterns for binary_rewrite vs sampling).
assert_file_regex	Like assert_regex but validates against a file's contents.
assert_perfetto	Validates that a Perfetto trace was generated and optionally checks its contents.
assert_rocpd	Validates that a ROCpd database was created. Requires @pytest.mark.rocpd("env_name").
assert_timemory	Validates timemory JSON output files.
assert_file_exists	Validates that a specific file exists in the output directory.
assert_unified_memory_output	Validates unified-memory text and JSON outputs (unified_memory*.txt / unified_memory*.json) under the test output directory.
assert_causal_json	Validates causal profiling JSON output.

For precise, one-off checks, plain assert statements can also be used. They are reported as a single test failure (not a subtest), but the same runner output is included in the failure report.

See the docstrings in conftest.py for full argument details.

Writing a Test

General Rules

At the minimum, the following rules should always be followed. If you have a reason not to, a comment should be left explaining the reasoning behind your choice.

• Test methods should always be inside a test class.
• Every test class should inherit from RocprofsysTest.
• At minimum, a test should use both run_test and assert_regex. They have complementary responsibilities: run_test performs process-level validation (subprocess launch, exit code, signals, timeout), while assert_regex performs content-level validation (matching against pass/fail patterns in the captured output). A test that only calls run_test will detect a crash but not a silent regression where the binary runs to completion without emitting the expected instrumentation output.

Runner Modes

When parametrizing a test to run against different runner modes (e.g., sampling, sys_run), use "mode" as the parameter name:

class TestExample(RocprofsysTest):
    @pytest.mark.parametrize("mode", ["sampling", "sys_run"])
    def test(self, mode, ...):
        result = self.run_test(mode, ...)
        self.assert_regex(result)

If a test is hard-coded to a single mode (no @pytest.mark.parametrize("mode", ...) decorator), tag it with @pytest.mark.<mode> (e.g. @pytest.mark.sampling) so _generate_ctest_definitions records the correct CTest mode label. When parametrize("mode", [...]) is used — even with a single-element list — this is handled automatically.

Test Parametrization

Parametrizing a test affects the generated CTest name. The order of @pytest.mark.parametrize decorators determines parameter order in the name. The runner mode should always come last so that tests group naturally when sorted:

# Good: parameters first, mode last
@pytest.mark.class_name("transpose")
class TestTranspose(RocprofsysTest):
    @pytest.mark.parametrize("mode", ["sampling", "sys_run"])
    @pytest.mark.parametrize("iterations,tile_dim,block_rows", [(1, 16, 16), (2, 32, 32)])
    def test_parametrized(self, mode, iterations, tile_dim, block_rows):
        ...

This produces names like:

transpose-parametrized-1-16-16-sampling
transpose-parametrized-1-16-16-sys_run
transpose-parametrized-2-32-32-sampling
transpose-parametrized-2-32-32-sys_run

Note: pytest applies decorators bottom-up, so the bottom @pytest.mark.parametrize varies first (outermost loop). Place mode on top so it varies last in the name.

Standardized test names

During collection, _standardize_test_name() in conftest.py builds a short name used for -k filtering and CTest identity.

• The test_ / test- prefix is stripped from the method name.
• By default, the Test prefix is stripped from the class name, then the class segment and method segment are joined.

Shape: Prefer names that read as <word>-<word>-.... Standardized names use lowercase hyphenated form: underscores become hyphens, repeated hyphens are collapsed, and the combined string is lowercased.

Long CamelCase classes: A class like TestRocprofilerSystemsRun would produce a long, hard-to-read segment. Put @pytest.mark.class_name("rocprofiler-systems-run") on the class to supply an explicit hyphenated prefix. Example:

@pytest.mark.class_name("rocprofiler-systems-run")
class TestRocprofilerSystemsRun(RocprofsysTest):
    def test_help(self):
        ...

That yields a stable name such as rocprofiler-systems-run-help.

@pytest.mark.depends_on(...) refers to the standardized name; use the same naming rules when declaring dependencies.

ROCpd Validation

ROCpd requires an environment fixture for injection. Mark the test with @pytest.mark.rocpd("<env_fixture_name>"). Do not set ROCPROFSYS_USE_ROCPD=ON explicitly. It is injected automatically when conditions are met.

@pytest.fixture
def my_env() -> dict[str, str]:
    return {}  # Do NOT specify ROCPROFSYS_USE_ROCPD=ON

class TestExample(RocprofsysTest):
    @pytest.mark.rocpd("my_env")
    def test(self, my_env):
        result = self.run_test("sampling", "target", env=my_env)
        self.assert_rocpd(result)  # Skipped automatically if ROCpd is unavailable

Custom Commands

To run an arbitrary command, use the baseline runner with the command option. You must provide absolute paths and handle missing targets yourself:

class TestExample(RocprofsysTest):
    def test(self, rocprof_config):
        script_path = rocprof_config.rocprofsys_tests_dir / "example-script.sh"
        if not script_path.exists():
            pytest.skip("example-script.sh not found")

        try:
            target = rocprof_config.get_target_executable("example-binary")
        except FileNotFoundError:
            pytest.skip("example-binary not found")

        result = self.run_test(
            "baseline", target, command=[script_path, target]
        )
        self.assert_regex(result)

If your custom command uses a rocprof-sys binary (e.g., rocprof-sys-instrument), add the corresponding marker (e.g., @pytest.mark.runtime_instrument).

Timeouts

Tests have a default timeout of 300 seconds (including subtests). To override, use @pytest.mark.timeout(<seconds>). This timeout is passed to CTest.

Serialization

If a test requires significant resources, mark it with @pytest.mark.serialize to prevent it from running in parallel.

Test Dependencies

If a test depends on the output of another test, use depends_on and preserve together.

Use the standardized test name of the producer in both @pytest.mark.depends_on(...) and any path under test_output_base (output directories are named after request.node.name, which is set during collection). Those names follow the rules in Standardized test names above—derive them from class + method (and parametrization) the same way, or copy the name from a generated CTestTestfile.cmake / ctest -N listing when in doubt.

• @pytest.mark.preserve("file1", "file2", ...) — Prevents files from being deleted after the test completes, even when ROCPROFSYS_KEEP_TEST_OUTPUT=0.
• @pytest.mark.depends_on("producer-generate", ...) — Declares a CTest dependency on one or more tests. Each argument must exactly match the producer’s standardized name (implementation: _standardize_test_name() in conftest.py).

class TestProducer(RocprofsysTest):
    @pytest.mark.preserve("coverage.json")
    def test_generate(self, ...):
        ...

class TestConsumer(RocprofsysTest):
    @pytest.mark.depends_on("producer-generate")
    def test_consume(self, test_output_base):
        file_path = test_output_base / "producer-generate" / "coverage.json"
        ...

Adding Markers

Every marker must be registered in pytest_configure() before use. Check the non_functional_markers and generic_functional_markers lists first; if the marker is not there, add it.

There are two types of markers:

• Informational markers — Used to categorize tests for CTest label filtering (e.g., -L "avail"). They don't affect test execution. Add these to the non_functional_markers list.

• Functional markers — Affect test behavior, typically used to skip tests based on system capabilities. Their logic is implemented in pytest_collection_modifyitems. Add these to the generic_functional_markers list (or register them individually with config.addinivalue_line if they need a custom description).

When adding a functional marker:

1. Register it in pytest_configure().
2. Add skip logic in pytest_collection_modifyitems. The convention is to define a checker function named <keyword>_unavailable_reason(...) that returns either None (when the requirement is met) or a str explaining why the test must be skipped (e.g. gpu_unavailable_reason, mpi_unavailable_reason). Then plug that function into the _skip(...) call for your marker.
3. If the marker depends on a system capability not already tracked by SystemCapabilities, add it to capabilities.py.

CTest label behavior: By default, all markers are included as CTest labels (e.g., ctest -L "rocm" filters by the @pytest.mark.rocm marker). To change how a marker appears in the generated CTest definitions, add it to one of these sets in _generate_ctest_definitions():

• no_report_markers — Marker is not added as a CTest label (e.g., timeout, serialize).
• no_report_args_markers — Marker name is added as a label, but its arguments are hidden (e.g., rocpd).
• only_report_args_markers — Only the marker's arguments are added as labels, not the marker name itself (e.g., mpi_implementation).

Template

# Copyright (c) Advanced Micro Devices, Inc.
# SPDX-License-Identifier:  MIT

"""<Description of what this test module covers>."""

from __future__ import annotations
import pytest
from conftest import RocprofsysTest

pytestmark = [
    # Module level markers
]


# =============================================================================
# Fixtures
# =============================================================================

@pytest.fixture
def <test>_env() -> dict[str, str]:
    # Environment for your test

# =============================================================================
# <Feature> tests
# =============================================================================

# Any class level markers here
class Test<NAME>(RocprofsysTest):

    # Any test level markers here
    def test_<NAME>(self, <test>_env):
        result = self.run_test(
            # mode,
            # target,
            # env =<test>_env,
            # ...
        )

        # Subtests
        self.assert_regex(result)
        # ...