TheRock/rocm-systems/projects/rccl/tools/scripts/test_runner/README.md

RCCL Test Runner

A Python-based test runner focused on RCCL unit and functional tests with hierarchical configuration support and integrated code coverage reporting. Extensible to support performance benchmarks, MPI tests, and custom test scripts.

Overview

This test runner provides a maintainable, extensible alternative to shell-based test execution. It uses JSON configuration files with hierarchical inheritance, and integrates with LLVM code coverage tools.

Key Features

• Multiple Test Types: Support for GTest, performance tests, and custom executables
• Hierarchical Configuration: Use "extends" directive to inherit and merge configurations
• Environment Variable Management: Global, configuration, suite, and test-specific environment variables
• Path Variable Expansion: Use environment variables in paths with nested default value expansion
• Custom Library Support: Use pre-built RCCL libraries from custom locations via environment variables
• Configurable Build System: Customize CMake options, environment variables, and parallel jobs via config
• MPI Support: Full support for multi-rank and multi-node tests
• Flexible Test Filtering: Both --test-name and --suite-name accept gtest-style glob patterns (P2P* = prefix, *SHM* = contains, *:-NET* = exclude; colon = OR; case-sensitive); filter by individual test name or by suite name
• Build Integration: Automated RCCL building with CMake
• Code Coverage: Integrated LLVM coverage report generation (HTML and text)
• Clean Output: Automatic filtering of MPI verbose messages (enable with --verbose)
• Verbose Logging: Detailed output for debugging and troubleshooting

Quick Start

Basic Usage

# Run with specific configuration
python test_runner.py --config my_tests.json

# Run with verbose output
python test_runner.py --config my_tests.json --verbose

# Run a specific test by exact name (searches all enabled suites for a match)
python test_runner.py --config my_tests.json --test-name SHM_ComprehensiveWorkflow

# Run all tests whose name starts with 'P2P' (glob prefix match)
python test_runner.py --config my_tests.json --test-name "P2P_*"

# Run multiple tests (colon-separated globs)
python test_runner.py --config my_tests.json --test-name "P2P_*:SHM_Basic"

# Run all suites whose name starts with 'P2P' (glob prefix match)
python test_runner.py --config my_tests.json --suite-name "P2P*"

# Run two specific suites by exact name (colon-separated)
python test_runner.py --config my_tests.json --suite-name "P2P Tests - 2-Rank:P2P Tests - 4-Rank"

Generate Coverage Report

# Build, run tests, and generate coverage report
python test_runner.py --config test_config_sample.json --coverage-report --verbose

# Use existing build and generate coverage
python test_runner.py --config test_config_sample.json --no-build --coverage-report

Use Custom RCCL Library

# Use pre-built RCCL library from custom location
export RCCL_LIB_PATH=/path/to/custom/rccl/build
python test_runner.py --config test_config_sample.json

# Or use RCCL_BUILD_DIR (alternative name)
export RCCL_BUILD_DIR=/path/to/custom/rccl/build
python test_runner.py --config test_config_sample.json

# When set, build step is automatically skipped
# --no-build is not needed

Rerun Failed Tests with Debug Environment

# Run tests and automatically rerun any failures with additional debug environment variables
python test_runner.py --config test_config_sample.json --rerun-failed

# Combine with verbose mode for detailed output
python test_runner.py --config test_config_sample.json --rerun-failed --verbose

# Stop immediately if a rerun also fails (fail-fast mode)
python test_runner.py --config test_config_sample.json --rerun-failed --stop-on-rerun-failure

Skip MPI Tests

# Skip MPI entirely: removes --enable-mpi-tests from build flags,
# skips MPI installation check, and skips tests with num_ranks > 1
python test_runner.py --config mi300x_mellanox_ib.json --skip-mpi-check

# Useful for single-rank unit tests that don't require MPI
python test_runner.py --config unit_tests_only.json --skip-mpi-check

Environment Variables

The test runner supports the following environment variables to customize behavior:

Library and Build Configuration

Variable	Description	Example
RCCL_LIB_PATH	Path to pre-built RCCL library directory (contains librccl.so and test/ subdirectory). When set, the build step is automatically skipped.	/path/to/rccl/build
RCCL_BUILD_DIR	Alternative name for RCCL_LIB_PATH. Either variable can be used.	/path/to/rccl/build
RCCL_TEST_MPI_HOSTFILE	Path to MPI hostfile for multi-node tests.	~/.mpi_hostfile
RCCL_MPI_LOG_ALL_RANKS	Set to 1 to capture stdout/stderr from every MPI rank into rccl_test_rank_<N>.log in the working directory (rank 0 is tee'd to console + file; ranks 1-N go to file only). Useful for debugging failures on non-zero ranks.	RCCL_MPI_LOG_ALL_RANKS=1
RCCL_TEST_GPUS_PER_NODE	Override the detected number of GPUs per node (used for "auto" sizing and GPU-count skipping). See GPU Count Detection.	4

Configuration Path Variables

These can be overridden via environment variables or specified in the JSON config:

Variable	Description	Default
WORKDIR	RCCL source and build directory	Current rccl repository root
ROCM_PATH	ROCm installation path	/opt/rocm
MPI_PATH	MPI installation path	System default or config-specific

Priority Order

When determining which RCCL library to use, the test runner follows this priority:

1. RCCL_LIB_PATH or RCCL_BUILD_DIR environment variable (highest priority)
   • Skips build automatically
   • Must contain librccl.so and test/ subdirectory
2. --no-build flag with local build
   • Uses local build/debug/ or build/release/ directory (based on install_flags)
   • Requires prior build
3. Default build process (lowest priority)
   • Builds RCCL via install.sh into build/debug/ or build/release/
   • Uses install_flags and cmake_options from JSON config

Example Usage:

# Priority 1: Use custom library (build skipped automatically)
export RCCL_LIB_PATH=/path/to/prebuilt/rccl/build
python test_runner.py --config my_tests.json

# Priority 2: Use existing local build (no new build)
python test_runner.py --config my_tests.json --no-build

# Priority 3: Fresh build (default)
python test_runner.py --config my_tests.json

Configuration File Format

Basic Structure

{
  "system_configurations": {
    "name": "system-name",
    "description": "System description"
  },
  "paths": {
    "workdir": "/path/to/rccl",
    "rocm_path": "/opt/rocm",
    "mpi_path": "/path/to/mpi"
  },
  "env_variables": {
    "GLOBAL_VAR": "value"
  },
  "test_configurations": {
    "config_name": {
      "env_variables": {...},
      "tests": [...]
    }
  },
  "test_suites": [
    {
      "name": "Test Suite Name",
      "config": "config_name",
      "enabled": true
    }
  ]
}

Environment Variable Expansion in Paths

The paths section supports environment variable expansion, allowing you to avoid hardcoding paths and make configurations portable across different systems.

Supported Syntax

{
  "paths": {
    "workdir": "${HOME}/code/rccl",
    "rocm_path": "$ROCM_PATH",
    "mpi_path": "${MPI_PATH:-/opt/mpi}"
  }
}

Syntax Options:

• ${VAR} - Expands to the value of VAR, left as-is if undefined
• $VAR - Expands to the value of VAR, left as-is if undefined
• ${VAR:-default} - Expands to the value of VAR, or default if undefined (bash-style default)

Examples

{
  "paths": {
    "workdir": "${WORKDIR:-${HOME}/code/rti/scripts/rccl}",
    "rocm_path": "${ROCM_PATH:-/opt/rocm}",
    "mpi_path": "${MPI_PATH:-${HOME}/softwares/ompi}"
  }
}

Usage:

# Use environment variables
export WORKDIR=/custom/path/to/rccl
export ROCM_PATH=/opt/rocm-6.0
export MPI_PATH=/usr/local/mpi

python test_runner.py --config test_config_sample.json

# Or use defaults (no environment variables set)
python test_runner.py --config test_config_sample.json

Benefits:

• Portability: Share configurations across different systems
• Flexibility: Override paths without modifying config files
• CI/CD: Easy integration with build systems and pipelines
• Multi-user: Same config works for different user environments

Test Types Supported

The test runner uses the is_gtest boolean flag to distinguish between test types:

• is_gtest: true (default) - GTest-based unit tests using --gtest_filter syntax
• is_gtest: false - Non-GTest tests (performance benchmarks, custom scripts, etc.)

This simplified approach supports all test categories while reducing configuration complexity.

GTest Tests (is_gtest: true)

Used for unit tests with GTest framework. The test_filter field uses GTest filter syntax.

{
  "name": "AllReduce_InPlace",
  "description": "Test AllReduce collective operation with in-place buffers",
  "is_gtest": true,
  "binary": "rccl-UnitTests",
  "test_filter": "AllReduce.InPlace",
  "num_ranks": 1,
  "num_nodes": 1,
  "timeout": 60
}

Command generated:

./rccl-UnitTests --gtest_filter=AllReduce.InPlace

Performance Tests (is_gtest: false)

Used for performance benchmarks. Arguments are passed directly without GTest syntax.

{
  "name": "Perf_Bandwidth",
  "description": "Bandwidth benchmark for AllReduce",
  "is_gtest": false,
  "binary": "all_reduce_perf",
  "command_args": "-b 8 -e 128M -f 2",
  "num_ranks": 2,
  "num_nodes": 1,
  "timeout": 300
}

Command generated:

mpirun -np 2 ./all_reduce_perf -b 8 -e 128M -f 2

Custom Scripts (is_gtest: false)

Used for custom validation scripts or any non-GTest executables.

{
  "name": "Custom_Validation",
  "description": "Custom GPU validation script",
  "is_gtest": false,
  "binary": "validate_gpus.sh",
  "command_args": "--full-check --verbose",
  "num_ranks": 1,
  "num_nodes": 1,
  "timeout": 120
}

Command generated:

./validate_gpus.sh --full-check --verbose

Key Differences:

Feature	is_gtest: true	is_gtest: false
Test framework	GTest (Google Test)	Any executable
Filter syntax	--gtest_filter=<pattern>	Plain arguments
test_filter field	GTest pattern (e.g., Suite.Test*)	Passed as plain argument
command_args field	Appended after filter	Primary argument method
Typical use cases	Unit tests, functional tests	Performance tests, custom scripts

Test Definition Fields

Field	Required	Type	Description
name	Yes	string	Unique test identifier
description	Recommended	string	Human-readable test description
is_gtest	Optional	boolean	Whether test uses GTest framework (default: true). Set to false for perf or custom tests
binary	Yes	string	Test binary name (relative to build/test/)
test_filter	Optional	string	Test filter (GTest filter syntax for gtest, plain argument for non-gtest)
command_args	Optional	string	Additional command-line arguments
num_ranks	Optional	integer or "auto"	Number of MPI ranks (default: 1). "auto" = detected GPUs/node × num_nodes (use all GPUs). See GPU Count Detection.
num_nodes	Optional	integer	Number of nodes (default: 1)
num_gpus	Optional	integer or "auto"	GPUs per node - controls rank distribution. "auto" (also the default when omitted) = detected GPUs/node.
timeout	Optional	integer	Timeout in seconds (0 = unlimited)
env_variables	Optional	object	Test-specific environment variables
rerun_env_variables	Optional	object	Additional environment variables for failed test reruns (merged with env_variables)
mpi_args	Optional	string or list	Extra mpirun/MCA arguments appended to the base MPI arguments (MPI tests only). See Custom MPI Arguments.

Configuration Inheritance

Use the "extends" directive to inherit from parent configurations:

{
  "test_configurations": {
    "base": {
      "env_variables": {
        "NCCL_DEBUG": "INFO"
      }
    },
    "shm_tests": {
      "extends": "base",
      "env_variables": {
        "NCCL_SHM_DISABLE": "0"
      },
      "tests": [...]
    },
    "advanced_shm": {
      "extends": ["base", "shm_tests"],
      "env_variables": {
        "NCCL_SHM_USE_CUDA_MEMCPY": "1"
      }
    }
  }
}

Hierarchical Defaults

To reduce repetition, you can specify default values at multiple levels with a clear override hierarchy:

Priority Order (highest to lowest):

1. Individual test - highest priority, overrides everything
2. Test suite level - overrides configuration defaults
3. Configuration level - base defaults for all tests in that config
4. Built-in defaults - system fallback values

Supported default fields: is_gtest, binary, num_ranks, num_nodes, num_gpus, timeout

Example with Three-Level Hierarchy

{
  "test_configurations": {
    "p2p_tests": {
      "is_gtest": true,
      "binary": "rccl-UnitTestsMPI",
      "num_ranks": 2,
      "num_nodes": 1,
      "num_gpus": 2,
      "timeout": 120,
      "env_variables": {
        "NCCL_P2P_DISABLE": "0"
      },
      "tests": [
        {
          "name": "P2P_Basic",
          "description": "Basic P2P test",
          "test_filter": "P2pMPITest.Basic"
          // Uses config defaults: is_gtest=true, binary, num_ranks=2, num_nodes=1, num_gpus=2, timeout=120
        },
        {
          "name": "P2P_LongRunning",
          "description": "Long-running P2P test",
          "test_filter": "P2pMPITest.LongRunning",
          "timeout": 300
          // Overrides timeout=300, inherits other config defaults
        }
      ]
    }
  },
  "test_suites": [
    {
      "name": "P2P_Basic_Suite",
      "config": "p2p_tests",
      "num_ranks": 4,
      "num_gpus": 4,
      "timeout": 180
      // Suite-level: overrides config's num_ranks, num_gpus, and timeout
      // Tests in this suite will use: num_ranks=4, num_gpus=4, timeout=180
    },
    {
      "name": "P2P_Stress_Suite",
      "config": "p2p_tests",
      "num_nodes": 2,
      "num_ranks": 4,
      "num_gpus": 2,
      "timeout": 600
      // Suite-level: overrides config's num_nodes, num_ranks, num_gpus, and timeout
      // Tests in this suite will use: num_nodes=2, num_ranks=4, num_gpus=2, timeout=600
    }
  ]
}

Benefits:

• Less Repetition: Define common values once
• Easier Maintenance: Update defaults in one place
• Flexible Overrides: Tests can still customize any field
• Cleaner Config: Shorter, more readable test definitions

Command-Line Options

Required:
  -c, --config CONFIG           Test configuration file (JSON format)

Optional:
  -v, --verbose             Enable verbose output (shows build paths, commands, etc.)
  -o, --output DIR          Output directory for logs and reports
  --test-name GLOB[:GLOB…]  Run only tests matching any glob pattern; ':' = OR, '*' = wildcard, '-' prefix = exclude, case-sensitive (e.g. 'P2P_*' = all P2P tests; '*:-SHM*' = all except SHM)
  --suite-name GLOB[:GLOB…] Run only suites matching any glob pattern; ':' = OR, '*' = wildcard, '-' prefix = exclude, case-sensitive (e.g. 'P2P*' = all P2P suites; '*:-NET*' = all except NET)
  --no-build                Skip build step and use existing build
  --skip-tests              Skip test execution (useful with --coverage-report)
  --coverage-report         Generate code coverage report (HTML + text)
  --build-dir PATH          Custom build directory path (default: <workdir>/build/debug or build/release)
  --rerun-failed            Rerun failed tests with additional environment variables
  --skip-mpi-check          Skip MPI: removes --enable-mpi-tests from build, skips MPI check, skips tests with num_ranks > 1
  --stop-on-rerun-failure   Stop testing immediately if a rerun also fails (requires --rerun-failed)
  --system NAME             Select system-specific MPI args profile from config (e.g. 'ainic', 'thor2')
  --mpi-args "ARGS"         Extra mpirun arguments appended after the config's base/suite/test mpi_args (MPI tests only). Also reads RCCL_TEST_MPI_ARGS env var.
  --mpich                   Use MPICH syntax (-env) instead of OpenMPI (-x) for passing env vars to mpirun
  --overwrite               Overwrite previous workspace directories
  --report-suffix SUFFIX    Suffix for report directory (default: blank)
  --emit-results            Emit structured results (JSON/JSONL + tarball) for the dashboard
  --results-dir DIR         Directory for emitted results + tarballs (default: <workspace>/results)
  --run-label LABEL         Optional label stored with the emitted run (e.g. 'nightly', a PR number)
  --tag TAG                 Tag to attach to the emitted run for dashboard filtering (repeatable)
  --tags A,B,C              Comma-separated tags to attach to the emitted run (merged with --tag)
  --db-push                 Also push results to PostgreSQL (DSN from RCCL_RESULTS_DSN); implies --emit-results
  --db-timeout SECONDS      PostgreSQL connect + statement timeout for --db-push (default: 10)
  -h, --help                Show help message and exit

Environment variables:
  RCCL_TEST_MPI_ARGS        Extra mpirun arguments appended after --mpi-args (same effect as --mpi-args)

Code Coverage Reports

The test runner integrates with LLVM tools to generate comprehensive code coverage reports.

Generating Coverage

# Build and test with coverage (recommended)
python test_runner.py --config test_config_sample.json --coverage-report --verbose

# Generate report from existing profraw files
python test_runner.py --config test_config_sample.json --no-build --skip-tests --coverage-report

Coverage Output

When --coverage-report is specified, the runner generates:

1. HTML Report: Visual coverage report in reports/ directory

   • View with: firefox reports/index.html
   • Shows line-by-line coverage with syntax highlighting

2. Text Report: Function-level coverage summary

   • Location: reports/function_coverage_report.txt
   • Includes per-function and per-file statistics

Coverage Implementation Details

• Uses LLVM instrumentation (-fprofile-instr-generate -fcoverage-mapping)
• Collects .profraw files during test execution
• Merges profiles with llvm-profdata
• Generates reports with llvm-cov show and llvm-cov report
• Filters out irrelevant files (test/, gtest, external dependencies)

Result Emission & Dashboard

The runner can emit structured, machine-readable results, either as local files (picked up by the dashboard's periodic sweep) or pushed directly to PostgreSQL.

Emitting results

# Local files only (durable; feeds the dashboard sweep):
python test_runner.py -c configs/rccl_perf_tests.json --emit-results --results-dir /path/to/results

# Local files + best-effort direct DB push:
export RCCL_RESULTS_DSN='postgresql://<user>:<pass>@<host>:5432/<db>'
python test_runner.py -c configs/rccl_perf_tests.json --db-push

• --db-push implies --emit-results. If the DB push fails or times out, the run still succeeds and the local tarball is retained for the sweep.
• The DSN is read only from RCCL_RESULTS_DSN; it is never hardcoded or written into the emitted files.
• Enabling emission also turns on per-test log capture so busbw/algbw can be parsed from rccl-tests output. Default behaviour (no capture) is unchanged when emission is off.

What is emitted

Per invocation, under --results-dir (default <workspace>/results):

• run.json - run manifest: RCCL SHA, host/telemetry metadata, config, env, summary, tags.
• tests.jsonl - one line per test (status PASSED/FAILED/SKIPPED/TIMEOUT, exec mode, dtype, duration).
• perf.jsonl - one line per (size, place) perf row (latency, algbw, busbw).
• coverage.json - llvm-cov totals (only when --coverage-report produced a report).
• <run_id>.tar.gz and latest.tar.gz - self-contained snapshots the sweep pulls.

Coverage is emitted only where the host has an instrumented build plus llvm-profdata/llvm-cov; perf and per-test results do not require them.

Tagging runs

Attach tags at emit time for later filtering in the dashboard:

python test_runner.py -c <config> --emit-results --tag nightly --tag mi300x
# or: --tags nightly,mi300x

Tags set here are the run's initial tags. Once a run is ingested, its tags are mutable only by dashboard admins.

See db/README.md for the database schema and the full emission and scrape/sweep contract.

Examples

Run All Enabled Test Suites

python test_runner.py --config test_config_sample.json --verbose

Run Specific Tests or Suites

Both --test-name and --suite-name accept gtest-style glob patterns:

Syntax	Meaning	Example
Name	Exact match	--test-name P2P_AllTests
Name*	Prefix match	--test-name "P2P_*"
*Name*	Contains match	--test-name "*AllReduce*"
A:B	A or B (OR)	--test-name "P2P_*:SHM_Basic"
-Name	Exclude	--test-name "*:-SHM*"

Matching is against the "name" field (tests) or suite name (suites), and is case-sensitive.

# Single test by exact name
python test_runner.py --config test_config_sample.json --test-name P2P_AllTests

# All tests whose name starts with 'P2P_'
python test_runner.py --config test_config_sample.json --test-name "P2P_*"

# All tests except SHM tests
python test_runner.py --config test_config_sample.json --test-name "*:-SHM*"

# All suites whose name starts with 'P2P' (glob prefix match)
python test_runner.py --config test_config_sample.json --suite-name "P2P*"

# Two specific suites matched by exact name (colon-separated)
python test_runner.py --config test_config_sample.json --suite-name "P2P Tests - 2-Rank:P2P Tests - 4-Rank"

Skip Build (Use Existing)

python test_runner.py --config test_config_sample.json --no-build

Build and Generate Coverage

# Full workflow: build, test, coverage
python test_runner.py --config adhoc_test_config.json --coverage-report --verbose

Generate Coverage from Existing Build

# Skip build, use existing profraw files
python test_runner.py --config adhoc_test_config.json --no-build --skip-tests --coverage-report

Custom Output Directory

python test_runner.py --config test_config_sample.json -o /path/to/output --verbose

Run with Custom Build Directory

python test_runner.py --config test_config_sample.json --build-dir /tmp/my_rccl_build --coverage-report

Rerun Failed Tests with Debug Environment

# Automatically rerun failed tests with enhanced debug environment variables
python test_runner.py --config test_config_sample.json --rerun-failed --verbose

# Combine with coverage report
python test_runner.py --config test_config_sample.json --rerun-failed --coverage-report

# Fail-fast mode: stop if rerun also fails
python test_runner.py --config test_config_sample.json --rerun-failed --stop-on-rerun-failure

Environment Variable Merging

Environment variables are merged hierarchically (later values override earlier):

1. Global env_variables (top-level in config)
2. Configuration env_variables (test configuration level)
3. Test Suite env_variables (suite level)
4. Test-specific env_variables (individual test level)

Example:

{
  "env_variables": {
    "NCCL_DEBUG": "INFO"
  },
  "test_configurations": {
    "shm_tests": {
      "env_variables": {
        "NCCL_SHM_DISABLE": "0"
      },
      "tests": [
        {
          "name": "SHM_Test",
          "env_variables": {
            "NCCL_DEBUG": "TRACE"
          }
        }
      ]
    }
  }
}

Result: NCCL_DEBUG=TRACE, NCCL_SHM_DISABLE=0

Test Execution

Single-Node Tests

• All ranks run on a single node
• Multiple ranks map to different GPUs
• Examples: SHM tests, P2P tests, unit tests

{
  "name": "SHM_Test",
  "num_ranks": 2,
  "num_nodes": 1
}

Multi-Node Tests

• Ranks distributed across multiple nodes via MPI
• Requires SLURM allocation or hostfile configuration
• Use num_gpus to control ranks per node (default: 8)
• Examples: NET transport tests, InfiniBand tests

{
  "name": "NET_Test_4Nodes_2GPUs",
  "num_ranks": 8,
  "num_nodes": 4,
  "num_gpus": 2
}

num_gpus Field:

• Controls how many MPI ranks are placed on each node
• Overrides hostfile slots specification
• For multi-node tests, uses --map-by ppr:{num_gpus}:node
• Default value: 8 (matches typical 8-GPU nodes)

Example: 2 nodes, 1 GPU per node

{
  "name": "NET_Test_2Nodes_1GPU",
  "num_ranks": 2,
  "num_nodes": 2,
  "num_gpus": 1
}

Command: mpirun -np 2 --hostfile file --map-by ppr:1:node ...

Setting Up Multi-Node Tests

Option 1: MPI Hostfile

export RCCL_TEST_MPI_HOSTFILE=/path/to/hostfile
python test_runner.py --config net_ib_test_config.json

Option 2: Default Hostfile Create ~/.mpi_hostfile with node names (one per line):

node01 slots=8
node02 slots=8

GPU Count Detection and Auto-Sizing

To make configurations portable across nodes with different GPU counts (e.g. 4-GPU vs 8-GPU nodes), the runner can detect how many GPUs a node has and size/skip tests accordingly.

Detection

The GPUs-per-node count is detected once, using this priority:

1. RCCL_TEST_GPUS_PER_NODE environment variable (explicit override)
2. Visible-device masks: HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, or CUDA_VISIBLE_DEVICES (counts the listed IDs)
3. rocminfo GPU-agent count

If none of these yield a value, detection returns "unknown" and the runner falls back to defaults (treating "auto" as 8) and disables GPU-count-based skipping.

"auto" values

Both num_gpus and num_ranks accept the literal string "auto":

• num_gpus: "auto" → resolves to the detected GPUs per node. This is also the default when num_gpus is omitted.
• num_ranks: "auto" → resolves to num_gpus × num_nodes (i.e. use all GPUs across all nodes).

Example — a single-node test that always uses every GPU on the node:

{
  "implicit_launch_order": {
    "is_gtest": true,
    "binary": "rccl-UnitTestsMPI",
    "num_ranks": "auto",
    "num_nodes": 1,
    "tests": [ { "name": "MultiCommunicatorChain", "test_filter": "..." } ]
  }
}

On an 8-GPU node this runs with 8 ranks; on a 4-GPU node, 4 ranks — no config change required.

Automatic skipping on insufficient GPUs

When the GPU count is known, a test is automatically SKIPPED (not failed) if the node does not have enough GPUs:

• Single-node tests: skipped when num_ranks > detected GPUs/node.
• Multi-node tests: skipped when num_gpus (ranks per node) > detected GPUs/node.

This lets fixed-size tests (e.g. an 8-rank test whose gtest filters require exactly 8 ranks) live in one shared config and simply self-skip on smaller nodes, instead of failing. Use RCCL_TEST_GPUS_PER_NODE to force a specific value (or to re-enable skipping in environments where detection fails).

Custom MPI Arguments

For MPI tests (num_ranks > 1), the runner builds the mpirun command line as:

mpirun -np <ranks> <host args> <map-by args> <mpi_args> ./<binary> ...

The <mpi_args> portion is resolved from the mpi_args field, which can be specified at multiple levels. Every mpi_args value may be either a string or a list of strings (lists are joined with spaces), so both of these are equivalent:

"mpi_args": "--bind-to none --oversubscribe"
"mpi_args": ["--bind-to none", "--oversubscribe"]

Resolution and Merge Order

There are two categories of mpi_args:

1. Base arguments — replace the built-in defaults. The base is chosen with this priority:
   1. Top-level mpi_args dict entry matching the active --system
   2. Top-level mpi_args string/list (applies to all systems)
   3. Built-in defaults (--mca btl ^vader,openib --mca pml ucx --bind-to none for Open MPI, -bind-to none for MPICH)
2. Additive arguments — appended on top of the base, regardless of --system:
   • Test-suite-level mpi_args
   • Individual test-level mpi_args
   • The --mpi-args CLI flag (applies to every MPI test in the run)
   • The RCCL_TEST_MPI_ARGS environment variable (same effect as --mpi-args)

Final command arguments are:

<base args>  <suite mpi_args>  <test mpi_args>  <--mpi-args>  <RCCL_TEST_MPI_ARGS>

The --mpi-args flag and RCCL_TEST_MPI_ARGS env var let you inject extra mpirun/MCA arguments without editing the config (both accept a single string; the CLI flag is appended before the env var). Example:

# via CLI flag
./run_tests.py -c config.json --mpi-args "--mca btl_tcp_if_include eth0 --oversubscribe"

# via environment variable
RCCL_TEST_MPI_ARGS="--oversubscribe" ./run_tests.py -c config.json

Note: Because base arguments replace the defaults, a top-level mpi_args should include any default flags you still want (e.g. --bind-to none). Suite- and test-level mpi_args are purely additive and do not remove anything.

Per-System Arguments (top-level dict)

Use a dict keyed by system name and select it with --system:

{
  "mpi_args": {
    "mellanox": "--mca oob_tcp_if_include eth1 --mca btl ^vader,openib --bind-to none",
    "thor2":    "--oversubscribe --mca plm_rsh_agent srun --mca btl ^openib --bind-to none"
  }
}

python test_runner.py --config net_ib_transport.json --system mellanox

If --system is not given (or has no matching key), the built-in defaults are used as the base.

Global Arguments (top-level string/list)

Apply the same base arguments to every MPI test:

{
  "mpi_args": ["--mca pml ucx", "--bind-to none", "--oversubscribe"]
}

Suite- and Test-Level Arguments (additive)

These are appended to the base and work regardless of --system:

{
  "test_suites": [
    {
      "name": "Oversubscribed_Suite",
      "config": "perf_tests",
      "mpi_args": "--oversubscribe"
    }
  ],
  "test_configurations": {
    "perf_tests": {
      "tests": [
        {
          "name": "AllReduce_Perf",
          "is_gtest": false,
          "binary": "all_reduce_perf",
          "command_args": "-b 8 -e 128M -f 2",
          "num_ranks": 2,
          "mpi_args": "--mca btl_tcp_if_include eth0"
        }
      ]
    }
  }
}

For the example above (no --system, no top-level mpi_args), the AllReduce test runs with:

mpirun -np 2 <host args> --mca btl ^vader,openib --mca pml ucx --bind-to none --oversubscribe --mca btl_tcp_if_include eth0 ./all_reduce_perf ...

(base defaults + suite --oversubscribe + test --mca btl_tcp_if_include eth0)

Inheritance

When using the extends directive, list-form mpi_args are appended and de-duplicated across parent/child configurations (see Configuration Inheritance).

Advanced Features

Rerun Failed Tests with Enhanced Debugging

When tests fail, it's often helpful to rerun them with additional debug output. The --rerun-failed flag automatically reruns any failed or timed-out tests with enhanced environment variables specified in the configuration.

Configuration

Add rerun_env_variables at the configuration level or individual test level:

{
  "test_configurations": {
    "unit_tests": {
      "env_variables": {
        "NCCL_DEBUG": ""
      },
      "rerun_env_variables": {
        "NCCL_DEBUG": "INFO"
      },
      "tests": [
        {
          "name": "MyTest",
          "test_filter": "MyTest.*"
        }
      ]
    }
  }
}

Usage

# Run tests and automatically rerun failures with debug environment
python test_runner.py --config my_tests.json --rerun-failed

# Behavior:
# 1. Test runs and fails
# 2. Test is IMMEDIATELY rerun with rerun_env_variables
# 3. Continue to next test (regardless of rerun result)
# No need to wait for all tests to complete - reruns happen inline

Environment Variable Merging

The rerun environment variables are merged with the original test environment:

1. Original test environment variables
2. Plus rerun_env_variables (overrides original if same key)

Example:

{
  "env_variables": {
    "NCCL_DEBUG": "WARN",
    "NCCL_LAUNCH_MODE": "GROUP"
  },
  "rerun_env_variables": {
    "NCCL_DEBUG": "INFO",
    "NCCL_DEBUG_SUBSYS": "ALL"
  }
}

Rerun environment:

• NCCL_DEBUG=INFO (overridden)
• NCCL_LAUNCH_MODE=GROUP (inherited)
• NCCL_DEBUG_SUBSYS=ALL (added)

Hierarchical Rerun Variables

Like other configuration fields, rerun_env_variables supports hierarchical inheritance:

{
  "test_configurations": {
    "default": {
      "rerun_env_variables": {
        "NCCL_DEBUG": "INFO"
      }
    },
    "shm_tests": {
      "extends": "default",
      "rerun_env_variables": {
        "NCCL_DEBUG_SUBSYS": "SHM"
      }
      // Inherits NCCL_DEBUG=INFO from default
      // Adds NCCL_DEBUG_SUBSYS=SHM
    }
  }
}

Test-Level Overrides

Individual tests can override rerun environment variables:

{
  "test_configurations": {
    "unit_tests": {
      "rerun_env_variables": {
        "NCCL_DEBUG": "INFO"
      },
      "tests": [
        {
          "name": "StandardTest"
          // Uses NCCL_DEBUG=INFO for reruns
        },
        {
          "name": "VerboseTest",
          "rerun_env_variables": {
            "NCCL_DEBUG": "TRACE",
            "NCCL_DEBUG_SUBSYS": "ALL"
          }
          // Uses TRACE and ALL for reruns instead
        }
      ]
    }
  }
}

Output Format

When a test fails and --rerun-failed is set, it is immediately rerun:

Running test: MyTest
  Result: FAILED (2.345 seconds)

================================================================================
RERUNNING FAILED TEST IMMEDIATELY
================================================================================

Rerunning test: MyTest
  Original result: FAILED
  Additional env variables:
    NCCL_DEBUG=INFO

Running test: MyTest (rerun)
  Result: PASSED (2.456 seconds)
  Rerun exit code: 0
================================================================================

# Next test continues...

At the end, both original and rerun results are summarized:

Detailed Results:
--------------------------------------------------------------------------------
Test Suite                               Test Name            Result     Duration
--------------------------------------------------------------------------------
Unit Tests                               MyTest               FAILED     2.345 seconds
--------------------------------------------------------------------------------

Rerun Results (with additional environment variables):
--------------------------------------------------------------------------------
Test Name                                                    Result     Duration
--------------------------------------------------------------------------------
MyTest                                                       PASSED     2.456 seconds
--------------------------------------------------------------------------------

Execution Flow

When --rerun-failed is set:

1. Test runs → If it passes, continue to next test
2. Test fails → Immediately rerun with rerun_env_variables
3. Rerun completes → Continue to next test (regardless of rerun pass/fail)
4. All tests complete → Summary shows both original and rerun results

The test suite continues executing even if reruns fail, ensuring all tests get a chance to run.

Fail-Fast Mode

Add --stop-on-rerun-failure to stop immediately if a rerun fails:

1. Test runs → If it passes, continue to next test
2. Test fails → Immediately rerun with rerun_env_variables
3. Rerun passes → Continue to next test
4. Rerun fails → STOP - No more tests executed

This is useful when:

• You want to fail fast and investigate the first persistent failure
• Running in CI/CD and want to save time if a test fails even after rerun
• Debugging a specific issue and don't need to run all tests

Example:

# Stop immediately if any test fails after rerun
python test_runner.py --config test_config.json --rerun-failed --stop-on-rerun-failure

Use Cases

• Debugging flaky tests: Add verbose logging only for failed tests
• Network issues: Enable network subsystem debugging for NET transport failures
• Memory issues: Add memory debugging flags for allocation failures
• CI/CD pipelines: Automatically gather more information on failures without verbose output for all tests
• Immediate feedback: See rerun results right after a failure instead of waiting for all tests

Build Configuration (New!)

Customize the RCCL build process through the build_configuration section in your JSON config file. The test runner invokes install.sh directly, so build options map to install.sh flags.

Basic Structure

{
  "build_configuration": {
    "install_flags": ["--debug", "-c", "-t", "-l", "--log-trace", "--enable-mpi-tests", "--no_clean"],
    "cmake_options": "-DCMAKE_EXPORT_COMPILE_COMMANDS=ON",
    "env_variables": {
      "HIPCC_COMPILE_FLAGS_APPEND": "-g -O1"
    },
    "parallel_jobs": 64
  }
}

Examples

Fast Development Build (No Coverage):

{
  "build_configuration": {
    "install_flags": ["-t", "-l", "--no_clean"],
    "parallel_jobs": 128
  }
}

Debug Build with Coverage:

{
  "build_configuration": {
    "install_flags": ["--debug", "-c", "-t", "-l", "--log-trace", "--no_clean"],
    "cmake_options": "-DCMAKE_CXX_FLAGS=-Wl,--build-id=sha1"
  }
}

Release Build:

{
  "build_configuration": {
    "install_flags": ["-t", "-l", "--no_clean"]
  }
}

Test Specific Functions Only:

{
  "build_configuration": {
    "install_flags": ["--debug", "-t", "--no_clean"],
    "cmake_options": "-DONLY_FUNCS=Broadcast|Reduce"
  }
}

All Options:

• install_flags - List of install.sh command-line flags (e.g. --debug, -t, -c, -l)
• cmake_options - Additional CMake options string passed via install.sh --cmake-options (e.g. -DFOO=BAR)
• env_variables - Build environment variables
• parallel_jobs - Number of parallel build threads (passed via -j)

Enhanced Environment Variable Expansion

Environment variables in the paths section now support nested expansion in default values:

{
  "paths": {
    "workdir": "${WORKDIR:-$HOME/code/rti/scripts/rccl}",
    "rocm_path": "${ROCM_PATH:-/opt/rocm}",
    "mpi_path": "${MPI_PATH:-$HOME/softwares/ompi}"
  }
}

Key Feature: If WORKDIR is not set, the default $HOME/code/rti/scripts/rccl will expand $HOME automatically!

Flexible Binary Paths

Specify test binary locations in multiple ways for maximum flexibility:

1. Default (Relative to build_dir/test/)

{
  "binary": "all_reduce_perf"
}

Result: <workdir>/build/debug/test/all_reduce_perf

2. Absolute Path

{
  "binary": "/opt/custom_rccl_build/test/all_reduce_perf"
}

Result: Uses the absolute path directly

3. Environment Variable in Binary Name

{
  "binary": "${MY_RCCL_TESTS}/all_reduce_perf"
}

Result: Expands $MY_RCCL_TESTS environment variable

4. Home Directory Expansion

{
  "binary": "~/my_builds/rccl/test/all_reduce_perf"
}

Result: Expands ~ to home directory

5. Using test_binary_dir in Paths

{
  "paths": {
    "test_binary_dir": "${RCCL_TEST_BIN_DIR}"
  },
  "test_configurations": {
    "my_tests": {
      "binary": "all_reduce_perf"
    }
  }
}

Result: ${RCCL_TEST_BIN_DIR}/all_reduce_perf

6. Using test_binary_dir in Test Config

{
  "test_configurations": {
    "my_tests": {
      "tests": [
        {
          "name": "CustomBinary",
          "test_binary_dir": "/opt/rccl/tests",
          "binary": "all_reduce_perf"
        }
      ]
    }
  }
}

Result: /opt/rccl/tests/all_reduce_perf

Resolution Priority Order

1. Absolute path in binary - Highest priority
2. Environment variable expansion (if results in absolute path)
3. test_binary_dir in test config + binary
4. test_binary_dir in paths + binary
5. Default: build_dir/test/ + binary - Lowest priority

Use Cases

• CI/CD with pre-built binaries: Use absolute paths or RCCL_TEST_BIN_DIR
• Multiple RCCL versions: Different test_binary_dir per configuration
• Custom build locations: Environment variables for flexibility
• Standard builds: Use default (no configuration needed)

Verbose Mode

Use --verbose to see the resolved binary path:

python test_runner.py --config test.json --verbose

Output includes:

Binary:  all_reduce_perf
Binary path: /home/user/code/rti/scripts/rccl/build/debug/test/all_reduce_perf

Configuration Best Practices

Reduce Repetition: Move common values to configuration level

{
  "test_configurations": {
    "p2p_tests": {
      "timeout": 120,
      "env_variables": {
        "NCCL_P2P_USE_CUDA_MEMCPY": "1",
        "NCCL_LEGACY_CUDA_REGISTER": "1"
      },
      "tests": [
        {
          "name": "Test1"
          // Inherits timeout and env vars from config level
        },
        {
          "name": "Test2",
          "timeout": 300
          // Overrides timeout, inherits env vars
        }
      ]
    }
  }
}

Benefits:

• ✅ Single source of truth for common settings
• ✅ Easier maintenance
• ✅ Tests can still override when needed
• ✅ Cleaner, more readable configurations

Development and Testing

Validate Configuration

# Test JSON syntax
python3 -m json.tool test_config_sample.json

# Test configuration loading
python3 -c "from lib.test_config import TestConfigProcessor; \
            p = TestConfigProcessor('test_config_sample.json'); \
            print('Configuration valid!')"

# Dry run (validate without executing)
python test_runner.py --config test_config_sample.json --skip-tests --verbose

Adding New Tests

1. Add test definition to appropriate configuration in JSON file
2. Specify is_gtest, description, and required fields
3. Test with dry run first: --skip-tests --verbose
4. Run actual test: --test-name YourTest --verbose (exact name) or --test-name "YourTest*" --verbose (glob) or --suite-name "My Suite" --verbose

Test Type Handling

The test runner uses a boolean is_gtest flag to distinguish between test types:

• is_gtest: true (default): Uses GTest framework with --gtest_filter=<filter> syntax
• is_gtest: false: Runs binary with plain arguments (for performance tests, custom scripts, etc.)

This simplified approach eliminates the need for multiple test type conditionals while supporting all test categories (gtest, perf, custom).

Troubleshooting

"Configuration file not found"

• Check the path to your JSON config file
• Use absolute paths or ensure you're in the correct directory
• Verify file permissions

"MPI path not found"

• Update paths.mpi_path in your configuration
• Ensure MPI is installed: which mpirun
• Check MPI_PATH environment variable

"Test binary not found"

• Build first: remove --no-build flag
• Check binary name in build/test/ directory
• Verify CMAKE built successfully

Multi-node tests hang

• Ensure SLURM allocation or hostfile is configured
• Check network connectivity: ping other_node
• Verify MPI can reach nodes: mpirun -np 2 hostname
• Check firewall settings

CMake configuration fails

• Check ROCm path: ls $ROCM_PATH
• Verify compiler: $ROCM_PATH/bin/amdclang++ --version
• Check MPI path: ls $MPI_PATH/bin/mpirun

Coverage report fails

• Ensure LLVM tools are available: which llvm-profdata llvm-cov
• Check for .profraw files in build directory
• Verify coverage build flags were set correctly
• Run with --verbose to see detailed error messages

"LLVM_PROFILE_FILE not being used"

• Ensure --coverage-report flag is specified
• Check that tests are actually executing (not skipped)
• Verify environment variables with --verbose

---

Appendix: Environment Variables Reference

This section provides a quick reference for all environment variables supported by the test runner.

Library and Build Location

Variable	Description	Example
RCCL_LIB_PATH	Path to pre-built RCCL library directory. Automatically skips build.	export RCCL_LIB_PATH=/path/to/rccl/build
RCCL_BUILD_DIR	Alternative name for RCCL_LIB_PATH.	export RCCL_BUILD_DIR=/home/user/rccl_builds/debug

Requirements: Directory must contain librccl.so and test/ subdirectory.

Configuration Paths

These override the paths specified in the JSON configuration file:

Variable	Description	Example
WORKDIR	RCCL source and build directory	export WORKDIR=/home/user/code/rccl
ROCM_PATH	ROCm installation path	export ROCM_PATH=/opt/rocm-6.0
MPI_PATH	MPI installation path	export MPI_PATH=/usr/local/openmpi

Test Execution

Variable	Description	Example
RCCL_TEST_MPI_HOSTFILE	Path to MPI hostfile for multi-node tests	export RCCL_TEST_MPI_HOSTFILE=~/.mpi_hostfile
RCCL_MPI_LOG_ALL_RANKS	Set to 1 to redirect each MPI rank's stdout/stderr to rccl_test_rank_<N>.log (rank 0 is also tee'd to the console). Useful when debugging failures on non-zero ranks or when tests read per-rank log files for assertions.	export RCCL_MPI_LOG_ALL_RANKS=1
RCCL_TEST_GPUS_PER_NODE	Override the detected GPUs-per-node count used for "auto" rank/GPU sizing and for skipping tests that need more GPUs than the node has. See GPU Count Detection.	export RCCL_TEST_GPUS_PER_NODE=4

Note: RCCL_TEST_MPI_HOSTFILE falls back to ~/.mpi_hostfile if not set. For SLURM environments, hostfile is auto-generated from SLURM_NODELIST.

Test-Specific Variables

These can be set globally or specified in the JSON configuration per test:

Variable	Description	Example
NCCL_DEBUG	NCCL debug level (VERSION, WARN, INFO, TRACE)	export NCCL_DEBUG=INFO
NCCL_DEBUG_SUBSYS	NCCL debug subsystems to enable	export NCCL_DEBUG_SUBSYS=INIT,COLL,NET
HSA_NO_SCRATCH_RECLAIM	Disable HIP scratch memory reclaim	export HSA_NO_SCRATCH_RECLAIM=1
NCCL_LAUNCH_MODE	NCCL launch mode (GROUP, PARALLEL)	export NCCL_LAUNCH_MODE=GROUP

Coverage and Profiling

Variable	Description	Example
LLVM_PROFILE_FILE	LLVM coverage profile output pattern	export LLVM_PROFILE_FILE=rccl_%p_%m.profraw

Note: Automatically set by test runner to prevent collisions. Manual override not recommended.

Complete Example

#!/bin/bash
# Configure paths
export WORKDIR=/home/user/code/rccl
export ROCM_PATH=/opt/rocm-6.0
export MPI_PATH=/usr/local/openmpi

# Use pre-built library
export RCCL_LIB_PATH=/home/user/rccl_builds/instrumented

# Configure MPI
export RCCL_TEST_MPI_HOSTFILE=~/.mpi_hostfile

# Enable debug output
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL,NET

# Run tests
python test_runner.py --config my_tests.json --verbose

Variable Priority

When the same configuration can be specified in multiple places, the priority is:

1. Environment variables (highest priority)
2. Test-specific configuration (in JSON)
3. Test suite configuration (in JSON)
4. Test configuration defaults (in JSON)
5. Built-in defaults (lowest priority)

Example: If ROCM_PATH is set as an environment variable, it overrides the rocm_path value in the JSON configuration file.