# RCCL Test Runner

A Python-based test runner focused on RCCL unit and functional tests with hierarchical configuration support and integrated code coverage reporting. Extensible to support performance benchmarks, MPI tests, and custom test scripts.

## Overview

This test runner provides a maintainable, extensible alternative to shell-based test execution. It uses JSON configuration files with hierarchical inheritance, and integrates with LLVM code coverage tools.

## Key Features

- **Multiple Test Types**: Support for GTest, performance tests, and custom executables
- **Hierarchical Configuration**: Use `"extends"` directive to inherit and merge configurations
- **Environment Variable Management**: Global, configuration, suite, and test-specific environment variables
- **Path Variable Expansion**: Use environment variables in paths with nested default value expansion
- **Custom Library Support**: Use pre-built RCCL libraries from custom locations via environment variables
- **Configurable Build System**: Customize CMake options, environment variables, and parallel jobs via config
- **MPI Support**: Full support for multi-rank and multi-node tests
- **Flexible Test Filtering**: Both `--test-name` and `--suite-name` accept gtest-style glob patterns (`P2P*` = prefix, `*SHM*` = contains, `*:-NET*` = exclude; colon = OR; case-sensitive); filter by individual test name or by suite name
- **Build Integration**: Automated RCCL building with CMake
- **Code Coverage**: Integrated LLVM coverage report generation (HTML and text)
- **Clean Output**: Automatic filtering of MPI verbose messages (enable with --verbose)
- **Verbose Logging**: Detailed output for debugging and troubleshooting

## Quick Start

### Basic Usage

```bash
# Run with specific configuration
python test_runner.py --config my_tests.json

# Run with verbose output
python test_runner.py --config my_tests.json --verbose

# Run a specific test by exact name (searches all enabled suites for a match)
python test_runner.py --config my_tests.json --test-name SHM_ComprehensiveWorkflow

# Run all tests whose name starts with 'P2P' (glob prefix match)
python test_runner.py --config my_tests.json --test-name "P2P_*"

# Run multiple tests (colon-separated globs)
python test_runner.py --config my_tests.json --test-name "P2P_*:SHM_Basic"

# Run all suites whose name starts with 'P2P' (glob prefix match)
python test_runner.py --config my_tests.json --suite-name "P2P*"

# Run two specific suites by exact name (colon-separated)
python test_runner.py --config my_tests.json --suite-name "P2P Tests - 2-Rank:P2P Tests - 4-Rank"
```

### Generate Coverage Report

```bash
# Build, run tests, and generate coverage report
python test_runner.py --config test_config_sample.json --coverage-report --verbose

# Use existing build and generate coverage
python test_runner.py --config test_config_sample.json --no-build --coverage-report
```

### Use Custom RCCL Library

```bash
# Use pre-built RCCL library from custom location
export RCCL_LIB_PATH=/path/to/custom/rccl/build
python test_runner.py --config test_config_sample.json

# Or use RCCL_BUILD_DIR (alternative name)
export RCCL_BUILD_DIR=/path/to/custom/rccl/build
python test_runner.py --config test_config_sample.json

# When set, build step is automatically skipped
# --no-build is not needed
```

### Rerun Failed Tests with Debug Environment

```bash
# Run tests and automatically rerun any failures with additional debug environment variables
python test_runner.py --config test_config_sample.json --rerun-failed

# Combine with verbose mode for detailed output
python test_runner.py --config test_config_sample.json --rerun-failed --verbose

# Stop immediately if a rerun also fails (fail-fast mode)
python test_runner.py --config test_config_sample.json --rerun-failed --stop-on-rerun-failure
```

### Skip MPI Tests

```bash
# Skip MPI entirely: removes --enable-mpi-tests from build flags,
# skips MPI installation check, and skips tests with num_ranks > 1
python test_runner.py --config mi300x_mellanox_ib.json --skip-mpi-check

# Useful for single-rank unit tests that don't require MPI
python test_runner.py --config unit_tests_only.json --skip-mpi-check
```

## Environment Variables

The test runner supports the following environment variables to customize behavior:

### Library and Build Configuration

| Variable | Description | Example |
|----------|-------------|---------|
| `RCCL_LIB_PATH` | Path to pre-built RCCL library directory (contains `librccl.so` and `test/` subdirectory). When set, the build step is automatically skipped. | `/path/to/rccl/build` |
| `RCCL_BUILD_DIR` | Alternative name for `RCCL_LIB_PATH`. Either variable can be used. | `/path/to/rccl/build` |
| `RCCL_TEST_MPI_HOSTFILE` | Path to MPI hostfile for multi-node tests. | `~/.mpi_hostfile` |
| `RCCL_MPI_LOG_ALL_RANKS` | Set to `1` to capture stdout/stderr from every MPI rank into `rccl_test_rank_<N>.log` in the working directory (rank 0 is tee'd to console + file; ranks 1-N go to file only). Useful for debugging failures on non-zero ranks. | `RCCL_MPI_LOG_ALL_RANKS=1` |
| `RCCL_TEST_GPUS_PER_NODE` | Override the detected number of GPUs per node (used for `"auto"` sizing and GPU-count skipping). See [GPU Count Detection](#gpu-count-detection-and-auto-sizing). | `4` |

### Configuration Path Variables

These can be overridden via environment variables or specified in the JSON config:

| Variable | Description | Default |
|----------|-------------|---------|
| `WORKDIR` | RCCL source and build directory | Current rccl repository root |
| `ROCM_PATH` | ROCm installation path | `/opt/rocm` |
| `MPI_PATH` | MPI installation path | System default or config-specific |

### Priority Order

When determining which RCCL library to use, the test runner follows this priority:

1. **`RCCL_LIB_PATH` or `RCCL_BUILD_DIR` environment variable** (highest priority)
   - Skips build automatically
   - Must contain `librccl.so` and `test/` subdirectory
2. **`--no-build` flag with local build**
   - Uses local `build/debug/` or `build/release/` directory (based on `install_flags`)
   - Requires prior build
3. **Default build process** (lowest priority)
   - Builds RCCL via `install.sh` into `build/debug/` or `build/release/`
   - Uses `install_flags` and `cmake_options` from JSON config

**Example Usage:**

```bash
# Priority 1: Use custom library (build skipped automatically)
export RCCL_LIB_PATH=/path/to/prebuilt/rccl/build
python test_runner.py --config my_tests.json

# Priority 2: Use existing local build (no new build)
python test_runner.py --config my_tests.json --no-build

# Priority 3: Fresh build (default)
python test_runner.py --config my_tests.json
```

## Configuration File Format

### Basic Structure

```json
{
  "system_configurations": {
    "name": "system-name",
    "description": "System description"
  },
  "paths": {
    "workdir": "/path/to/rccl",
    "rocm_path": "/opt/rocm",
    "mpi_path": "/path/to/mpi"
  },
  "env_variables": {
    "GLOBAL_VAR": "value"
  },
  "test_configurations": {
    "config_name": {
      "env_variables": {...},
      "tests": [...]
    }
  },
  "test_suites": [
    {
      "name": "Test Suite Name",
      "config": "config_name",
      "enabled": true
    }
  ]
}
```

### Environment Variable Expansion in Paths

The `paths` section supports environment variable expansion, allowing you to avoid hardcoding paths and make configurations portable across different systems.

#### Supported Syntax

```json
{
  "paths": {
    "workdir": "${HOME}/code/rccl",
    "rocm_path": "$ROCM_PATH",
    "mpi_path": "${MPI_PATH:-/opt/mpi}"
  }
}
```

**Syntax Options:**
- `${VAR}` - Expands to the value of `VAR`, left as-is if undefined
- `$VAR` - Expands to the value of `VAR`, left as-is if undefined
- `${VAR:-default}` - Expands to the value of `VAR`, or `default` if undefined (bash-style default)

#### Examples

```json
{
  "paths": {
    "workdir": "${WORKDIR:-${HOME}/code/rti/scripts/rccl}",
    "rocm_path": "${ROCM_PATH:-/opt/rocm}",
    "mpi_path": "${MPI_PATH:-${HOME}/softwares/ompi}"
  }
}
```

**Usage:**
```bash
# Use environment variables
export WORKDIR=/custom/path/to/rccl
export ROCM_PATH=/opt/rocm-6.0
export MPI_PATH=/usr/local/mpi

python test_runner.py --config test_config_sample.json

# Or use defaults (no environment variables set)
python test_runner.py --config test_config_sample.json
```

**Benefits:**
- **Portability**: Share configurations across different systems
- **Flexibility**: Override paths without modifying config files
- **CI/CD**: Easy integration with build systems and pipelines
- **Multi-user**: Same config works for different user environments

### Test Types Supported

The test runner uses the `is_gtest` boolean flag to distinguish between test types:

- **`is_gtest: true`** (default) - GTest-based unit tests using `--gtest_filter` syntax
- **`is_gtest: false`** - Non-GTest tests (performance benchmarks, custom scripts, etc.)

This simplified approach supports all test categories while reducing configuration complexity.

#### GTest Tests (`is_gtest: true`)

Used for unit tests with GTest framework. The `test_filter` field uses GTest filter syntax.

```json
{
  "name": "AllReduce_InPlace",
  "description": "Test AllReduce collective operation with in-place buffers",
  "is_gtest": true,
  "binary": "rccl-UnitTests",
  "test_filter": "AllReduce.InPlace",
  "num_ranks": 1,
  "num_nodes": 1,
  "timeout": 60
}
```

**Command generated:**
```bash
./rccl-UnitTests --gtest_filter=AllReduce.InPlace
```

#### Performance Tests (`is_gtest: false`)

Used for performance benchmarks. Arguments are passed directly without GTest syntax.

```json
{
  "name": "Perf_Bandwidth",
  "description": "Bandwidth benchmark for AllReduce",
  "is_gtest": false,
  "binary": "all_reduce_perf",
  "command_args": "-b 8 -e 128M -f 2",
  "num_ranks": 2,
  "num_nodes": 1,
  "timeout": 300
}
```

**Command generated:**
```bash
mpirun -np 2 ./all_reduce_perf -b 8 -e 128M -f 2
```

#### Custom Scripts (`is_gtest: false`)

Used for custom validation scripts or any non-GTest executables.

```json
{
  "name": "Custom_Validation",
  "description": "Custom GPU validation script",
  "is_gtest": false,
  "binary": "validate_gpus.sh",
  "command_args": "--full-check --verbose",
  "num_ranks": 1,
  "num_nodes": 1,
  "timeout": 120
}
```

**Command generated:**
```bash
./validate_gpus.sh --full-check --verbose
```

**Key Differences:**

| Feature | `is_gtest: true` | `is_gtest: false` |
|---------|------------------|-------------------|
| Test framework | GTest (Google Test) | Any executable |
| Filter syntax | `--gtest_filter=<pattern>` | Plain arguments |
| `test_filter` field | GTest pattern (e.g., `Suite.Test*`) | Passed as plain argument |
| `command_args` field | Appended after filter | Primary argument method |
| Typical use cases | Unit tests, functional tests | Performance tests, custom scripts |

### Test Definition Fields

| Field | Required | Type | Description |
|-------|----------|------|-------------|
| `name` | Yes | string | Unique test identifier |
| `description` | Recommended | string | Human-readable test description |
| `is_gtest` | Optional | boolean | Whether test uses GTest framework (default: true). Set to false for perf or custom tests |
| `binary` | Yes | string | Test binary name (relative to build/test/) |
| `test_filter` | Optional | string | Test filter (GTest filter syntax for gtest, plain argument for non-gtest) |
| `command_args` | Optional | string | Additional command-line arguments |
| `num_ranks` | Optional | integer or `"auto"` | Number of MPI ranks (default: 1). `"auto"` = detected GPUs/node × `num_nodes` (use all GPUs). See [GPU Count Detection](#gpu-count-detection-and-auto-sizing). |
| `num_nodes` | Optional | integer | Number of nodes (default: 1) |
| `num_gpus` | Optional | integer or `"auto"` | GPUs per node - controls rank distribution. `"auto"` (also the default when omitted) = detected GPUs/node. |
| `timeout` | Optional | integer | Timeout in seconds (0 = unlimited) |
| `env_variables` | Optional | object | Test-specific environment variables |
| `rerun_env_variables` | Optional | object | Additional environment variables for failed test reruns (merged with env_variables) |
| `mpi_args` | Optional | string or list | Extra `mpirun`/MCA arguments appended to the base MPI arguments (MPI tests only). See [Custom MPI Arguments](#custom-mpi-arguments). |

### Configuration Inheritance

Use the `"extends"` directive to inherit from parent configurations:

```json
{
  "test_configurations": {
    "base": {
      "env_variables": {
        "NCCL_DEBUG": "INFO"
      }
    },
    "shm_tests": {
      "extends": "base",
      "env_variables": {
        "NCCL_SHM_DISABLE": "0"
      },
      "tests": [...]
    },
    "advanced_shm": {
      "extends": ["base", "shm_tests"],
      "env_variables": {
        "NCCL_SHM_USE_CUDA_MEMCPY": "1"
      }
    }
  }
}
```

### Hierarchical Defaults

To reduce repetition, you can specify default values at multiple levels with a clear override hierarchy:

**Priority Order (highest to lowest):**
1. **Individual test** - highest priority, overrides everything
2. **Test suite level** - overrides configuration defaults
3. **Configuration level** - base defaults for all tests in that config
4. **Built-in defaults** - system fallback values

**Supported default fields:** `is_gtest`, `binary`, `num_ranks`, `num_nodes`, `num_gpus`, `timeout`

#### Example with Three-Level Hierarchy

```json
{
  "test_configurations": {
    "p2p_tests": {
      "is_gtest": true,
      "binary": "rccl-UnitTestsMPI",
      "num_ranks": 2,
      "num_nodes": 1,
      "num_gpus": 2,
      "timeout": 120,
      "env_variables": {
        "NCCL_P2P_DISABLE": "0"
      },
      "tests": [
        {
          "name": "P2P_Basic",
          "description": "Basic P2P test",
          "test_filter": "P2pMPITest.Basic"
          // Uses config defaults: is_gtest=true, binary, num_ranks=2, num_nodes=1, num_gpus=2, timeout=120
        },
        {
          "name": "P2P_LongRunning",
          "description": "Long-running P2P test",
          "test_filter": "P2pMPITest.LongRunning",
          "timeout": 300
          // Overrides timeout=300, inherits other config defaults
        }
      ]
    }
  },
  "test_suites": [
    {
      "name": "P2P_Basic_Suite",
      "config": "p2p_tests",
      "num_ranks": 4,
      "num_gpus": 4,
      "timeout": 180
      // Suite-level: overrides config's num_ranks, num_gpus, and timeout
      // Tests in this suite will use: num_ranks=4, num_gpus=4, timeout=180
    },
    {
      "name": "P2P_Stress_Suite",
      "config": "p2p_tests",
      "num_nodes": 2,
      "num_ranks": 4,
      "num_gpus": 2,
      "timeout": 600
      // Suite-level: overrides config's num_nodes, num_ranks, num_gpus, and timeout
      // Tests in this suite will use: num_nodes=2, num_ranks=4, num_gpus=2, timeout=600
    }
  ]
}
```

**Benefits:**
- **Less Repetition**: Define common values once
- **Easier Maintenance**: Update defaults in one place
- **Flexible Overrides**: Tests can still customize any field
- **Cleaner Config**: Shorter, more readable test definitions

## Command-Line Options

```
Required:
  -c, --config CONFIG           Test configuration file (JSON format)

Optional:
  -v, --verbose             Enable verbose output (shows build paths, commands, etc.)
  -o, --output DIR          Output directory for logs and reports
  --test-name GLOB[:GLOB…]  Run only tests matching any glob pattern; ':' = OR, '*' = wildcard, '-' prefix = exclude, case-sensitive (e.g. 'P2P_*' = all P2P tests; '*:-SHM*' = all except SHM)
  --suite-name GLOB[:GLOB…] Run only suites matching any glob pattern; ':' = OR, '*' = wildcard, '-' prefix = exclude, case-sensitive (e.g. 'P2P*' = all P2P suites; '*:-NET*' = all except NET)
  --no-build                Skip build step and use existing build
  --skip-tests              Skip test execution (useful with --coverage-report)
  --coverage-report         Generate code coverage report (HTML + text)
  --build-dir PATH          Custom build directory path (default: <workdir>/build/debug or build/release)
  --rerun-failed            Rerun failed tests with additional environment variables
  --skip-mpi-check          Skip MPI: removes --enable-mpi-tests from build, skips MPI check, skips tests with num_ranks > 1
  --stop-on-rerun-failure   Stop testing immediately if a rerun also fails (requires --rerun-failed)
  --system NAME             Select system-specific MPI args profile from config (e.g. 'ainic', 'thor2')
  --mpi-args "ARGS"         Extra mpirun arguments appended after the config's base/suite/test mpi_args (MPI tests only). Also reads RCCL_TEST_MPI_ARGS env var.
  --mpich                   Use MPICH syntax (-env) instead of OpenMPI (-x) for passing env vars to mpirun
  --overwrite               Overwrite previous workspace directories
  --report-suffix SUFFIX    Suffix for report directory (default: blank)
  --emit-results            Emit structured results (JSON/JSONL + tarball) for the dashboard
  --results-dir DIR         Directory for emitted results + tarballs (default: <workspace>/results)
  --run-label LABEL         Optional label stored with the emitted run (e.g. 'nightly', a PR number)
  --tag TAG                 Tag to attach to the emitted run for dashboard filtering (repeatable)
  --tags A,B,C              Comma-separated tags to attach to the emitted run (merged with --tag)
  --db-push                 Also push results to PostgreSQL (DSN from RCCL_RESULTS_DSN); implies --emit-results
  --db-timeout SECONDS      PostgreSQL connect + statement timeout for --db-push (default: 10)
  -h, --help                Show help message and exit

Environment variables:
  RCCL_TEST_MPI_ARGS        Extra mpirun arguments appended after --mpi-args (same effect as --mpi-args)
```

## Code Coverage Reports

The test runner integrates with LLVM tools to generate comprehensive code coverage reports.

### Generating Coverage

```bash
# Build and test with coverage (recommended)
python test_runner.py --config test_config_sample.json --coverage-report --verbose

# Generate report from existing profraw files
python test_runner.py --config test_config_sample.json --no-build --skip-tests --coverage-report
```

### Coverage Output

When `--coverage-report` is specified, the runner generates:

1. **HTML Report**: Visual coverage report in `reports/` directory
   - View with: `firefox reports/index.html`
   - Shows line-by-line coverage with syntax highlighting

2. **Text Report**: Function-level coverage summary
   - Location: `reports/function_coverage_report.txt`
   - Includes per-function and per-file statistics

### Coverage Implementation Details

- Uses LLVM instrumentation (`-fprofile-instr-generate -fcoverage-mapping`)
- Collects `.profraw` files during test execution
- Merges profiles with `llvm-profdata`
- Generates reports with `llvm-cov show` and `llvm-cov report`
- Filters out irrelevant files (test/, gtest, external dependencies)

## Result Emission & Dashboard

The runner can emit structured, machine-readable results, either as local files
(picked up by the dashboard's periodic sweep) or pushed directly to PostgreSQL.

### Emitting results

```bash
# Local files only (durable; feeds the dashboard sweep):
python test_runner.py -c configs/rccl_perf_tests.json --emit-results --results-dir /path/to/results

# Local files + best-effort direct DB push:
export RCCL_RESULTS_DSN='postgresql://<user>:<pass>@<host>:5432/<db>'
python test_runner.py -c configs/rccl_perf_tests.json --db-push
```

- `--db-push` implies `--emit-results`. If the DB push fails or times out, the
  run still succeeds and the local tarball is retained for the sweep.
- The DSN is read only from `RCCL_RESULTS_DSN`; it is never hardcoded or written
  into the emitted files.
- Enabling emission also turns on per-test log capture so `busbw`/`algbw` can be
  parsed from rccl-tests output. Default behaviour (no capture) is unchanged when
  emission is off.

### What is emitted

Per invocation, under `--results-dir` (default `<workspace>/results`):

- `run.json` - run manifest: RCCL SHA, host/telemetry metadata, config, env, summary, tags.
- `tests.jsonl` - one line per test (status PASSED/FAILED/SKIPPED/TIMEOUT, exec mode, dtype, duration).
- `perf.jsonl` - one line per (size, place) perf row (latency, algbw, busbw).
- `coverage.json` - llvm-cov totals (only when `--coverage-report` produced a report).
- `<run_id>.tar.gz` and `latest.tar.gz` - self-contained snapshots the sweep pulls.

Coverage is emitted only where the host has an instrumented build plus
`llvm-profdata`/`llvm-cov`; perf and per-test results do not require them.

### Tagging runs

Attach tags at emit time for later filtering in the dashboard:

```bash
python test_runner.py -c <config> --emit-results --tag nightly --tag mi300x
# or: --tags nightly,mi300x
```

Tags set here are the run's initial tags. Once a run is ingested, its tags are
mutable only by dashboard admins.

See [`db/README.md`](db/README.md) for the database schema and the full
emission and scrape/sweep contract.

## Examples

### Run All Enabled Test Suites

```bash
python test_runner.py --config test_config_sample.json --verbose
```

### Run Specific Tests or Suites

Both `--test-name` and `--suite-name` accept gtest-style glob patterns:

| Syntax | Meaning | Example |
|--------|---------|---------|
| `Name` | Exact match | `--test-name P2P_AllTests` |
| `Name*` | Prefix match | `--test-name "P2P_*"` |
| `*Name*` | Contains match | `--test-name "*AllReduce*"` |
| `A:B` | A or B (OR) | `--test-name "P2P_*:SHM_Basic"` |
| `-Name` | Exclude | `--test-name "*:-SHM*"` |

Matching is against the `"name"` field (tests) or suite name (suites), and is case-sensitive.

```bash
# Single test by exact name
python test_runner.py --config test_config_sample.json --test-name P2P_AllTests

# All tests whose name starts with 'P2P_'
python test_runner.py --config test_config_sample.json --test-name "P2P_*"

# All tests except SHM tests
python test_runner.py --config test_config_sample.json --test-name "*:-SHM*"

# All suites whose name starts with 'P2P' (glob prefix match)
python test_runner.py --config test_config_sample.json --suite-name "P2P*"

# Two specific suites matched by exact name (colon-separated)
python test_runner.py --config test_config_sample.json --suite-name "P2P Tests - 2-Rank:P2P Tests - 4-Rank"
```

### Skip Build (Use Existing)

```bash
python test_runner.py --config test_config_sample.json --no-build
```

### Build and Generate Coverage

```bash
# Full workflow: build, test, coverage
python test_runner.py --config adhoc_test_config.json --coverage-report --verbose
```

### Generate Coverage from Existing Build

```bash
# Skip build, use existing profraw files
python test_runner.py --config adhoc_test_config.json --no-build --skip-tests --coverage-report
```

### Custom Output Directory

```bash
python test_runner.py --config test_config_sample.json -o /path/to/output --verbose
```

### Run with Custom Build Directory

```bash
python test_runner.py --config test_config_sample.json --build-dir /tmp/my_rccl_build --coverage-report
```

### Rerun Failed Tests with Debug Environment

```bash
# Automatically rerun failed tests with enhanced debug environment variables
python test_runner.py --config test_config_sample.json --rerun-failed --verbose

# Combine with coverage report
python test_runner.py --config test_config_sample.json --rerun-failed --coverage-report

# Fail-fast mode: stop if rerun also fails
python test_runner.py --config test_config_sample.json --rerun-failed --stop-on-rerun-failure
```

## Environment Variable Merging

Environment variables are merged hierarchically (later values override earlier):

1. **Global** `env_variables` (top-level in config)
2. **Configuration** `env_variables` (test configuration level)
3. **Test Suite** `env_variables` (suite level)
4. **Test-specific** `env_variables` (individual test level)

Example:
```json
{
  "env_variables": {
    "NCCL_DEBUG": "INFO"
  },
  "test_configurations": {
    "shm_tests": {
      "env_variables": {
        "NCCL_SHM_DISABLE": "0"
      },
      "tests": [
        {
          "name": "SHM_Test",
          "env_variables": {
            "NCCL_DEBUG": "TRACE"
          }
        }
      ]
    }
  }
}
```

Result: `NCCL_DEBUG=TRACE`, `NCCL_SHM_DISABLE=0`

## Test Execution

### Single-Node Tests

- All ranks run on a single node
- Multiple ranks map to different GPUs
- Examples: SHM tests, P2P tests, unit tests

```json
{
  "name": "SHM_Test",
  "num_ranks": 2,
  "num_nodes": 1
}
```

### Multi-Node Tests

- Ranks distributed across multiple nodes via MPI
- Requires SLURM allocation or hostfile configuration
- Use `num_gpus` to control ranks per node (default: 8)
- Examples: NET transport tests, InfiniBand tests

```json
{
  "name": "NET_Test_4Nodes_2GPUs",
  "num_ranks": 8,
  "num_nodes": 4,
  "num_gpus": 2
}
```

**`num_gpus` Field:**
- Controls how many MPI ranks are placed on each node
- Overrides hostfile `slots` specification
- For multi-node tests, uses `--map-by ppr:{num_gpus}:node`
- Default value: 8 (matches typical 8-GPU nodes)

**Example: 2 nodes, 1 GPU per node**
```json
{
  "name": "NET_Test_2Nodes_1GPU",
  "num_ranks": 2,
  "num_nodes": 2,
  "num_gpus": 1
}
```
Command: `mpirun -np 2 --hostfile file --map-by ppr:1:node ...`

### Setting Up Multi-Node Tests

**Option 1: MPI Hostfile**
```bash
export RCCL_TEST_MPI_HOSTFILE=/path/to/hostfile
python test_runner.py --config net_ib_test_config.json
```

**Option 2: Default Hostfile**
Create `~/.mpi_hostfile` with node names (one per line):
```
node01 slots=8
node02 slots=8
```

## GPU Count Detection and Auto-Sizing

To make configurations portable across nodes with different GPU counts (e.g. 4-GPU vs 8-GPU nodes), the runner can detect how many GPUs a node has and size/skip tests accordingly.

### Detection

The GPUs-per-node count is detected once, using this priority:

1. `RCCL_TEST_GPUS_PER_NODE` environment variable (explicit override)
2. Visible-device masks: `HIP_VISIBLE_DEVICES`, `ROCR_VISIBLE_DEVICES`, or `CUDA_VISIBLE_DEVICES` (counts the listed IDs)
3. `rocminfo` GPU-agent count

If none of these yield a value, detection returns "unknown" and the runner falls back to defaults (treating `"auto"` as 8) and disables GPU-count-based skipping.

### `"auto"` values

Both `num_gpus` and `num_ranks` accept the literal string `"auto"`:

- `num_gpus: "auto"` → resolves to the detected GPUs per node. **This is also the default when `num_gpus` is omitted.**
- `num_ranks: "auto"` → resolves to `num_gpus × num_nodes` (i.e. use *all* GPUs across all nodes).

Example — a single-node test that always uses every GPU on the node:

```json
{
  "implicit_launch_order": {
    "is_gtest": true,
    "binary": "rccl-UnitTestsMPI",
    "num_ranks": "auto",
    "num_nodes": 1,
    "tests": [ { "name": "MultiCommunicatorChain", "test_filter": "..." } ]
  }
}
```

On an 8-GPU node this runs with 8 ranks; on a 4-GPU node, 4 ranks — no config change required.

### Automatic skipping on insufficient GPUs

When the GPU count is known, a test is automatically **SKIPPED** (not failed) if the node does not have enough GPUs:

- **Single-node** tests: skipped when `num_ranks` > detected GPUs/node.
- **Multi-node** tests: skipped when `num_gpus` (ranks per node) > detected GPUs/node.

This lets fixed-size tests (e.g. an 8-rank test whose gtest filters require exactly 8 ranks) live in one shared config and simply self-skip on smaller nodes, instead of failing. Use `RCCL_TEST_GPUS_PER_NODE` to force a specific value (or to re-enable skipping in environments where detection fails).

## Custom MPI Arguments

For MPI tests (`num_ranks > 1`), the runner builds the `mpirun` command line as:

```
mpirun -np <ranks> <host args> <map-by args> <mpi_args> ./<binary> ...
```

The `<mpi_args>` portion is resolved from the `mpi_args` field, which can be
specified at multiple levels. **Every `mpi_args` value may be either a string or
a list of strings** (lists are joined with spaces), so both of these are
equivalent:

```json
"mpi_args": "--bind-to none --oversubscribe"
"mpi_args": ["--bind-to none", "--oversubscribe"]
```

### Resolution and Merge Order

There are two categories of `mpi_args`:

1. **Base arguments** — *replace* the built-in defaults. The base is chosen with
   this priority:
   1. Top-level `mpi_args` dict entry matching the active `--system`
   2. Top-level `mpi_args` string/list (applies to all systems)
   3. Built-in defaults (`--mca btl ^vader,openib --mca pml ucx --bind-to none`
      for Open MPI, `-bind-to none` for MPICH)
2. **Additive arguments** — *appended* on top of the base, regardless of
   `--system`:
   - Test-suite-level `mpi_args`
   - Individual test-level `mpi_args`
   - The `--mpi-args` CLI flag (applies to every MPI test in the run)
   - The `RCCL_TEST_MPI_ARGS` environment variable (same effect as `--mpi-args`)

Final command arguments are:

```
<base args>  <suite mpi_args>  <test mpi_args>  <--mpi-args>  <RCCL_TEST_MPI_ARGS>
```

The `--mpi-args` flag and `RCCL_TEST_MPI_ARGS` env var let you inject extra
`mpirun`/MCA arguments without editing the config (both accept a single string;
the CLI flag is appended before the env var). Example:

```bash
# via CLI flag
./run_tests.py -c config.json --mpi-args "--mca btl_tcp_if_include eth0 --oversubscribe"

# via environment variable
RCCL_TEST_MPI_ARGS="--oversubscribe" ./run_tests.py -c config.json
```

> **Note:** Because base arguments *replace* the defaults, a top-level
> `mpi_args` should include any default flags you still want (e.g.
> `--bind-to none`). Suite- and test-level `mpi_args` are purely additive and do
> not remove anything.

### Per-System Arguments (top-level dict)

Use a dict keyed by system name and select it with `--system`:

```json
{
  "mpi_args": {
    "mellanox": "--mca oob_tcp_if_include eth1 --mca btl ^vader,openib --bind-to none",
    "thor2":    "--oversubscribe --mca plm_rsh_agent srun --mca btl ^openib --bind-to none"
  }
}
```

```bash
python test_runner.py --config net_ib_transport.json --system mellanox
```

If `--system` is not given (or has no matching key), the built-in defaults are
used as the base.

### Global Arguments (top-level string/list)

Apply the same base arguments to every MPI test:

```json
{
  "mpi_args": ["--mca pml ucx", "--bind-to none", "--oversubscribe"]
}
```

### Suite- and Test-Level Arguments (additive)

These are appended to the base and work regardless of `--system`:

```json
{
  "test_suites": [
    {
      "name": "Oversubscribed_Suite",
      "config": "perf_tests",
      "mpi_args": "--oversubscribe"
    }
  ],
  "test_configurations": {
    "perf_tests": {
      "tests": [
        {
          "name": "AllReduce_Perf",
          "is_gtest": false,
          "binary": "all_reduce_perf",
          "command_args": "-b 8 -e 128M -f 2",
          "num_ranks": 2,
          "mpi_args": "--mca btl_tcp_if_include eth0"
        }
      ]
    }
  }
}
```

For the example above (no `--system`, no top-level `mpi_args`), the AllReduce
test runs with:

```
mpirun -np 2 <host args> --mca btl ^vader,openib --mca pml ucx --bind-to none --oversubscribe --mca btl_tcp_if_include eth0 ./all_reduce_perf ...
```

(base defaults + suite `--oversubscribe` + test `--mca btl_tcp_if_include eth0`)

### Inheritance

When using the `extends` directive, list-form `mpi_args` are appended and
de-duplicated across parent/child configurations (see
[Configuration Inheritance](#configuration-inheritance)).

## Advanced Features

### Rerun Failed Tests with Enhanced Debugging

When tests fail, it's often helpful to rerun them with additional debug output. The `--rerun-failed` flag automatically reruns any failed or timed-out tests with enhanced environment variables specified in the configuration.

#### Configuration

Add `rerun_env_variables` at the configuration level or individual test level:

```json
{
  "test_configurations": {
    "unit_tests": {
      "env_variables": {
        "NCCL_DEBUG": ""
      },
      "rerun_env_variables": {
        "NCCL_DEBUG": "INFO"
      },
      "tests": [
        {
          "name": "MyTest",
          "test_filter": "MyTest.*"
        }
      ]
    }
  }
}
```

#### Usage

```bash
# Run tests and automatically rerun failures with debug environment
python test_runner.py --config my_tests.json --rerun-failed

# Behavior:
# 1. Test runs and fails
# 2. Test is IMMEDIATELY rerun with rerun_env_variables
# 3. Continue to next test (regardless of rerun result)
# No need to wait for all tests to complete - reruns happen inline
```

#### Environment Variable Merging

The rerun environment variables are **merged** with the original test environment:

1. Original test environment variables
2. Plus `rerun_env_variables` (overrides original if same key)

**Example:**

```json
{
  "env_variables": {
    "NCCL_DEBUG": "WARN",
    "NCCL_LAUNCH_MODE": "GROUP"
  },
  "rerun_env_variables": {
    "NCCL_DEBUG": "INFO",
    "NCCL_DEBUG_SUBSYS": "ALL"
  }
}
```

**Rerun environment:**
- `NCCL_DEBUG=INFO` (overridden)
- `NCCL_LAUNCH_MODE=GROUP` (inherited)
- `NCCL_DEBUG_SUBSYS=ALL` (added)

#### Hierarchical Rerun Variables

Like other configuration fields, `rerun_env_variables` supports hierarchical inheritance:

```json
{
  "test_configurations": {
    "default": {
      "rerun_env_variables": {
        "NCCL_DEBUG": "INFO"
      }
    },
    "shm_tests": {
      "extends": "default",
      "rerun_env_variables": {
        "NCCL_DEBUG_SUBSYS": "SHM"
      }
      // Inherits NCCL_DEBUG=INFO from default
      // Adds NCCL_DEBUG_SUBSYS=SHM
    }
  }
}
```

#### Test-Level Overrides

Individual tests can override rerun environment variables:

```json
{
  "test_configurations": {
    "unit_tests": {
      "rerun_env_variables": {
        "NCCL_DEBUG": "INFO"
      },
      "tests": [
        {
          "name": "StandardTest"
          // Uses NCCL_DEBUG=INFO for reruns
        },
        {
          "name": "VerboseTest",
          "rerun_env_variables": {
            "NCCL_DEBUG": "TRACE",
            "NCCL_DEBUG_SUBSYS": "ALL"
          }
          // Uses TRACE and ALL for reruns instead
        }
      ]
    }
  }
}
```

#### Output Format

When a test fails and `--rerun-failed` is set, it is immediately rerun:

```
Running test: MyTest
  Result: FAILED (2.345 seconds)

================================================================================
RERUNNING FAILED TEST IMMEDIATELY
================================================================================

Rerunning test: MyTest
  Original result: FAILED
  Additional env variables:
    NCCL_DEBUG=INFO

Running test: MyTest (rerun)
  Result: PASSED (2.456 seconds)
  Rerun exit code: 0
================================================================================

# Next test continues...
```

At the end, both original and rerun results are summarized:

```
Detailed Results:
--------------------------------------------------------------------------------
Test Suite                               Test Name            Result     Duration
--------------------------------------------------------------------------------
Unit Tests                               MyTest               FAILED     2.345 seconds
--------------------------------------------------------------------------------

Rerun Results (with additional environment variables):
--------------------------------------------------------------------------------
Test Name                                                    Result     Duration
--------------------------------------------------------------------------------
MyTest                                                       PASSED     2.456 seconds
--------------------------------------------------------------------------------
```

#### Execution Flow

When `--rerun-failed` is set:

1. **Test runs** → If it passes, continue to next test
2. **Test fails** → Immediately rerun with `rerun_env_variables`
3. **Rerun completes** → Continue to next test (regardless of rerun pass/fail)
4. **All tests complete** → Summary shows both original and rerun results

The test suite continues executing even if reruns fail, ensuring all tests get a chance to run.

#### Fail-Fast Mode

Add `--stop-on-rerun-failure` to stop immediately if a rerun fails:

1. **Test runs** → If it passes, continue to next test
2. **Test fails** → Immediately rerun with `rerun_env_variables`
3. **Rerun passes** → Continue to next test
4. **Rerun fails** → **STOP** - No more tests executed

This is useful when:
- You want to fail fast and investigate the first persistent failure
- Running in CI/CD and want to save time if a test fails even after rerun
- Debugging a specific issue and don't need to run all tests

Example:
```bash
# Stop immediately if any test fails after rerun
python test_runner.py --config test_config.json --rerun-failed --stop-on-rerun-failure
```

#### Use Cases

- **Debugging flaky tests:** Add verbose logging only for failed tests
- **Network issues:** Enable network subsystem debugging for NET transport failures
- **Memory issues:** Add memory debugging flags for allocation failures
- **CI/CD pipelines:** Automatically gather more information on failures without verbose output for all tests
- **Immediate feedback:** See rerun results right after a failure instead of waiting for all tests

### Build Configuration (New!)

Customize the RCCL build process through the `build_configuration` section in your JSON config file.
The test runner invokes `install.sh` directly, so build options map to `install.sh` flags.

#### Basic Structure

```json
{
  "build_configuration": {
    "install_flags": ["--debug", "-c", "-t", "-l", "--log-trace", "--enable-mpi-tests", "--no_clean"],
    "cmake_options": "-DCMAKE_EXPORT_COMPILE_COMMANDS=ON",
    "env_variables": {
      "HIPCC_COMPILE_FLAGS_APPEND": "-g -O1"
    },
    "parallel_jobs": 64
  }
}
```

#### Examples

**Fast Development Build (No Coverage):**
```json
{
  "build_configuration": {
    "install_flags": ["-t", "-l", "--no_clean"],
    "parallel_jobs": 128
  }
}
```

**Debug Build with Coverage:**
```json
{
  "build_configuration": {
    "install_flags": ["--debug", "-c", "-t", "-l", "--log-trace", "--no_clean"],
    "cmake_options": "-DCMAKE_CXX_FLAGS=-Wl,--build-id=sha1"
  }
}
```

**Release Build:**
```json
{
  "build_configuration": {
    "install_flags": ["-t", "-l", "--no_clean"]
  }
}
```

**Test Specific Functions Only:**
```json
{
  "build_configuration": {
    "install_flags": ["--debug", "-t", "--no_clean"],
    "cmake_options": "-DONLY_FUNCS=Broadcast|Reduce"
  }
}
```

**All Options:**
- `install_flags` - List of `install.sh` command-line flags (e.g. `--debug`, `-t`, `-c`, `-l`)
- `cmake_options` - Additional CMake options string passed via `install.sh --cmake-options` (e.g. `-DFOO=BAR`)
- `env_variables` - Build environment variables
- `parallel_jobs` - Number of parallel build threads (passed via `-j`)

### Enhanced Environment Variable Expansion

Environment variables in the `paths` section now support **nested expansion** in default values:

```json
{
  "paths": {
    "workdir": "${WORKDIR:-$HOME/code/rti/scripts/rccl}",
    "rocm_path": "${ROCM_PATH:-/opt/rocm}",
    "mpi_path": "${MPI_PATH:-$HOME/softwares/ompi}"
  }
}
```

**Key Feature:** If `WORKDIR` is not set, the default `$HOME/code/rti/scripts/rccl` will expand `$HOME` automatically!

### Flexible Binary Paths

Specify test binary locations in multiple ways for maximum flexibility:

#### 1. Default (Relative to build_dir/test/)

```json
{
  "binary": "all_reduce_perf"
}
```
Result: `<workdir>/build/debug/test/all_reduce_perf`

#### 2. Absolute Path

```json
{
  "binary": "/opt/custom_rccl_build/test/all_reduce_perf"
}
```
Result: Uses the absolute path directly

#### 3. Environment Variable in Binary Name

```json
{
  "binary": "${MY_RCCL_TESTS}/all_reduce_perf"
}
```
Result: Expands `$MY_RCCL_TESTS` environment variable

#### 4. Home Directory Expansion

```json
{
  "binary": "~/my_builds/rccl/test/all_reduce_perf"
}
```
Result: Expands `~` to home directory

#### 5. Using test_binary_dir in Paths

```json
{
  "paths": {
    "test_binary_dir": "${RCCL_TEST_BIN_DIR}"
  },
  "test_configurations": {
    "my_tests": {
      "binary": "all_reduce_perf"
    }
  }
}
```
Result: `${RCCL_TEST_BIN_DIR}/all_reduce_perf`

#### 6. Using test_binary_dir in Test Config

```json
{
  "test_configurations": {
    "my_tests": {
      "tests": [
        {
          "name": "CustomBinary",
          "test_binary_dir": "/opt/rccl/tests",
          "binary": "all_reduce_perf"
        }
      ]
    }
  }
}
```
Result: `/opt/rccl/tests/all_reduce_perf`

#### Resolution Priority Order

1. **Absolute path in binary** - Highest priority
2. **Environment variable expansion** (if results in absolute path)
3. **test_binary_dir in test config** + binary
4. **test_binary_dir in paths** + binary
5. **Default:** `build_dir/test/` + binary - Lowest priority

#### Use Cases

- **CI/CD with pre-built binaries:** Use absolute paths or `RCCL_TEST_BIN_DIR`
- **Multiple RCCL versions:** Different `test_binary_dir` per configuration
- **Custom build locations:** Environment variables for flexibility
- **Standard builds:** Use default (no configuration needed)

#### Verbose Mode

Use `--verbose` to see the resolved binary path:
```bash
python test_runner.py --config test.json --verbose
```

Output includes:
```
Binary:  all_reduce_perf
Binary path: /home/user/code/rti/scripts/rccl/build/debug/test/all_reduce_perf
```

### Configuration Best Practices

**Reduce Repetition:** Move common values to configuration level

```json
{
  "test_configurations": {
    "p2p_tests": {
      "timeout": 120,
      "env_variables": {
        "NCCL_P2P_USE_CUDA_MEMCPY": "1",
        "NCCL_LEGACY_CUDA_REGISTER": "1"
      },
      "tests": [
        {
          "name": "Test1"
          // Inherits timeout and env vars from config level
        },
        {
          "name": "Test2",
          "timeout": 300
          // Overrides timeout, inherits env vars
        }
      ]
    }
  }
}
```

**Benefits:**
- ✅ Single source of truth for common settings
- ✅ Easier maintenance
- ✅ Tests can still override when needed
- ✅ Cleaner, more readable configurations

## Development and Testing

### Validate Configuration

```bash
# Test JSON syntax
python3 -m json.tool test_config_sample.json

# Test configuration loading
python3 -c "from lib.test_config import TestConfigProcessor; \
            p = TestConfigProcessor('test_config_sample.json'); \
            print('Configuration valid!')"

# Dry run (validate without executing)
python test_runner.py --config test_config_sample.json --skip-tests --verbose
```

### Adding New Tests

1. Add test definition to appropriate configuration in JSON file
2. Specify `is_gtest`, `description`, and required fields
3. Test with dry run first: `--skip-tests --verbose`
4. Run actual test: `--test-name YourTest --verbose` (exact name) or `--test-name "YourTest*" --verbose` (glob) or `--suite-name "My Suite" --verbose`

### Test Type Handling

The test runner uses a boolean `is_gtest` flag to distinguish between test types:

- **`is_gtest: true`** (default): Uses GTest framework with `--gtest_filter=<filter>` syntax
- **`is_gtest: false`**: Runs binary with plain arguments (for performance tests, custom scripts, etc.)

This simplified approach eliminates the need for multiple test type conditionals while supporting all test categories (gtest, perf, custom).

## Troubleshooting

### "Configuration file not found"
- Check the path to your JSON config file
- Use absolute paths or ensure you're in the correct directory
- Verify file permissions

### "MPI path not found"
- Update `paths.mpi_path` in your configuration
- Ensure MPI is installed: `which mpirun`
- Check MPI_PATH environment variable

### "Test binary not found"
- Build first: remove `--no-build` flag
- Check binary name in `build/test/` directory
- Verify CMAKE built successfully

### Multi-node tests hang
- Ensure SLURM allocation or hostfile is configured
- Check network connectivity: `ping other_node`
- Verify MPI can reach nodes: `mpirun -np 2 hostname`
- Check firewall settings

### CMake configuration fails
- Check ROCm path: `ls $ROCM_PATH`
- Verify compiler: `$ROCM_PATH/bin/amdclang++ --version`
- Check MPI path: `ls $MPI_PATH/bin/mpirun`

### Coverage report fails
- Ensure LLVM tools are available: `which llvm-profdata llvm-cov`
- Check for `.profraw` files in build directory
- Verify coverage build flags were set correctly
- Run with `--verbose` to see detailed error messages

### "LLVM_PROFILE_FILE not being used"
- Ensure `--coverage-report` flag is specified
- Check that tests are actually executing (not skipped)
- Verify environment variables with `--verbose`

---

## Appendix: Environment Variables Reference

This section provides a quick reference for all environment variables supported by the test runner.

### Library and Build Location

| Variable | Description | Example |
|----------|-------------|---------|
| `RCCL_LIB_PATH` | Path to pre-built RCCL library directory. Automatically skips build. | `export RCCL_LIB_PATH=/path/to/rccl/build` |
| `RCCL_BUILD_DIR` | Alternative name for `RCCL_LIB_PATH`. | `export RCCL_BUILD_DIR=/home/user/rccl_builds/debug` |

**Requirements**: Directory must contain `librccl.so` and `test/` subdirectory.

### Configuration Paths

These override the paths specified in the JSON configuration file:

| Variable | Description | Example |
|----------|-------------|---------|
| `WORKDIR` | RCCL source and build directory | `export WORKDIR=/home/user/code/rccl` |
| `ROCM_PATH` | ROCm installation path | `export ROCM_PATH=/opt/rocm-6.0` |
| `MPI_PATH` | MPI installation path | `export MPI_PATH=/usr/local/openmpi` |

### Test Execution

| Variable | Description | Example |
|----------|-------------|---------|
| `RCCL_TEST_MPI_HOSTFILE` | Path to MPI hostfile for multi-node tests | `export RCCL_TEST_MPI_HOSTFILE=~/.mpi_hostfile` |
| `RCCL_MPI_LOG_ALL_RANKS` | Set to `1` to redirect each MPI rank's stdout/stderr to `rccl_test_rank_<N>.log` (rank 0 is also tee'd to the console). Useful when debugging failures on non-zero ranks or when tests read per-rank log files for assertions. | `export RCCL_MPI_LOG_ALL_RANKS=1` |
| `RCCL_TEST_GPUS_PER_NODE` | Override the detected GPUs-per-node count used for `"auto"` rank/GPU sizing and for skipping tests that need more GPUs than the node has. See [GPU Count Detection](#gpu-count-detection-and-auto-sizing). | `export RCCL_TEST_GPUS_PER_NODE=4` |

**Note**: `RCCL_TEST_MPI_HOSTFILE` falls back to `~/.mpi_hostfile` if not set. For SLURM environments, hostfile is auto-generated from `SLURM_NODELIST`.

### Test-Specific Variables

These can be set globally or specified in the JSON configuration per test:

| Variable | Description | Example |
|----------|-------------|---------|
| `NCCL_DEBUG` | NCCL debug level (VERSION, WARN, INFO, TRACE) | `export NCCL_DEBUG=INFO` |
| `NCCL_DEBUG_SUBSYS` | NCCL debug subsystems to enable | `export NCCL_DEBUG_SUBSYS=INIT,COLL,NET` |
| `HSA_NO_SCRATCH_RECLAIM` | Disable HIP scratch memory reclaim | `export HSA_NO_SCRATCH_RECLAIM=1` |
| `NCCL_LAUNCH_MODE` | NCCL launch mode (GROUP, PARALLEL) | `export NCCL_LAUNCH_MODE=GROUP` |

### Coverage and Profiling

| Variable | Description | Example |
|----------|-------------|---------|
| `LLVM_PROFILE_FILE` | LLVM coverage profile output pattern | `export LLVM_PROFILE_FILE=rccl_%p_%m.profraw` |

**Note**: Automatically set by test runner to prevent collisions. Manual override not recommended.

### Complete Example

```bash
#!/bin/bash
# Configure paths
export WORKDIR=/home/user/code/rccl
export ROCM_PATH=/opt/rocm-6.0
export MPI_PATH=/usr/local/openmpi

# Use pre-built library
export RCCL_LIB_PATH=/home/user/rccl_builds/instrumented

# Configure MPI
export RCCL_TEST_MPI_HOSTFILE=~/.mpi_hostfile

# Enable debug output
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL,NET

# Run tests
python test_runner.py --config my_tests.json --verbose
```

### Variable Priority

When the same configuration can be specified in multiple places, the priority is:

1. **Environment variables** (highest priority)
2. **Test-specific configuration** (in JSON)
3. **Test suite configuration** (in JSON)
4. **Test configuration defaults** (in JSON)
5. **Built-in defaults** (lowest priority)

**Example**: If `ROCM_PATH` is set as an environment variable, it overrides the `rocm_path` value in the JSON configuration file.