TheRock/rocm-systems/projects/rccl/test/ext-plugins/README.md

# RCCL Plugin Tests

## Description

This directory contains automated tests for RCCL (ROCm Communication Collectives Library) plugins:

1. **CSV Tuner Plugin**: Validates the functionality of the CSV-based tuning plugin across different collective operations (AllReduce, Broadcast, Reduce, AllGather, and ReduceScatter) and various configuration scenarios.

2. **Profiler Plugin**: Validates the profiler plugin that captures detailed runtime events for collective and P2P operations, including Group, Collective, P2P, ProxyOp, ProxyStep, and GPU kernel events.

3. **Inspector Plugin**: Validates the inspector plugin that produces structured JSONL output for collective operations. Tests cover schema correctness, per-rank output files, timing source validation, and verbose event trace fields across single-node and multi-node configurations.

The tests are written in Python using the pytest framework, making it easy to run, maintain, and extend the test coverage.

## Directory Structure

```
ext-plugins/
├── README.md                    # This file - documentation for the test suite
├── requirements.txt             # Python dependencies required for testing
├── pytest.ini                   # Pytest configuration and test markers
├── .gitignore                   # Git ignore rules for Python/pytest artifacts
├── venv/                        # Python virtual environment (created after setup)
├── logs/                        # Test execution logs and output files
├── profiler_dumps/              # Profiler plugin output (JSON trace files)
├── inspector_dumps/             # Inspector plugin output (JSONL log files)
├── assets/                      # Test configuration files and assets
│   └── csv_confs/               # CSV configuration files for testing
│       ├── incorrect_values_config.conf
│       ├── multinode_config.conf
│       ├── no_matching_config.conf
│       ├── singlenode_config.conf
│       ├── unsupported_algo_proto_config.conf
│       ├── valid_config_with_wildcards.conf
│       └── valid_config_without_wildcards.conf
└── tests/                       # Test suite directory
    ├── conftest.py              # Pytest fixtures and shared test configuration
    ├── ext-tuner/               # CSV Tuner Plugin tests (plugin sources at plugins/tuner/)
    │   ├── test_allreduce.py
    │   ├── test_broadcast.py
    │   ├── test_reduce.py
    │   ├── test_allgather.py
    │   └── test_reducescatter.py
    ├── ext-profiler/            # Profiler Plugin tests (plugin sources at plugins/profiler/)
    │   ├── test_allreduce.py
    │   ├── test_broadcast.py
    │   ├── test_reduce.py
    │   ├── test_allgather.py
    │   ├── test_alltoall.py
    │   ├── test_reducescatter.py
    │   └── test_sendrecv.py
    └── ext-inspector/           # Inspector Plugin specific tests
        ├── conftest.py          # Inspector-specific fixtures and validators
        ├── test_allreduce.py
        ├── test_allgather.py
        ├── test_broadcast.py
        ├── test_reduce.py
        └── test_reducescatter.py
```

## Installation & Setup

### Prerequisites

- Python 3.6 or higher
- RCCL library installed
- ROCm environment configured
- **Important**: Set the following environment variables to match your environment:

```bash
export RCCL_INSTALL_DIR=/path/to/rccl          # RCCL source root (contains plugins/profiler/, plugins/tuner/)
export OMPI_INSTALL_DIR=/path/to/ompi/install   # OpenMPI installation directory
export RCCL_TESTS_DIR=/path/to/rccl-tests       # rccl-tests build directory
```

These are read by `tests/conftest.py` at runtime. If not set (or if plugin libraries are missing), plugin-specific tests are skipped with a clear reason in pytest output.

### Building the RCCL Plugins

Before running the tests, you need to build the RCCL plugin libraries.

#### Building the CSV Tuner Plugin

The CSV tuner plugin is located in the `plugins/tuner/example` directory.

**Step 1: Navigate to the plugin directory**

```bash
cd rccl/plugins/tuner/example
```

**Step 2: Build the plugin**

```bash
make
```

This will compile the plugin and create `libnccl-tuner-example.so` in the same directory.

#### Building the Profiler Plugin

The profiler plugin is located in the `plugins/profiler/example` directory.

**Step 1: Navigate to the plugin directory**

```bash
cd rccl/plugins/profiler/example
```

**Step 2: Build the plugin**

```bash
make
```

This will compile the plugin and create `librccl-profiler-example.so` in the same directory. The profiler plugin captures detailed runtime events including:
- **Group events**: High-level operation grouping
- **Collective events**: AllReduce, Broadcast, Reduce, ReduceScatter operations
- **P2P events**: Send, Recv, AllGather, AllToAll operations
- **ProxyOp events**: Network proxy operations (ScheduleSend, ScheduleRecv, ProgressSend, ProgressRecv)
- **ProxyStep events**: Detailed network steps (SendWait, RecvWait, GPU waits)
- **ProxyCtrl events**: Proxy thread control (Append, Sleep)
- **GPU events**: Kernel channel execution

#### Building the Inspector Plugin

The inspector plugin is located in the `plugins/profiler/inspector` directory.

**Step 1: Navigate to the plugin directory**

```bash
cd rccl/plugins/profiler/inspector
```

**Step 2: Build the plugin**

```bash
make
```

This will compile the plugin and create `librccl-profiler-inspector.so` in the same directory. The inspector plugin produces structured JSONL output for collective operations, including:
- **Header**: Rank ID, total ranks, and node count
- **Metadata**: Output format version, recording mechanism, hostname, PID, and dump timestamp
- **Collective performance**: Operation type, sequence number, message size, execution time, timing source, algorithm bandwidth, and bus bandwidth
- **Verbose event trace** (optional): Per-kernel channel start/stop sequence numbers and timestamps

### Step 1: Create Virtual Environment

Create a Python virtual environment to isolate the test dependencies:

```bash
python3 -m venv venv
```

### Step 2: Activate Virtual Environment

Activate the virtual environment using the appropriate command for your shell:

**On Linux/Mac (bash/zsh):**
```bash
source venv/bin/activate
```

Once activated, you should see `(venv)` at the beginning of your command prompt.

### Step 3: Install Dependencies

Install the required Python packages:

```bash
pip install -r requirements.txt
```

## Running Tests

### Run All Tests

To run the entire test suite:

```bash
pytest --cache-clear
```

### Run Tests with Verbose Output

For more detailed test output:

```bash
pytest -v --cache-clear
```

### Run Tests by Marker

Tests are organized using pytest markers. You can run specific groups of tests:

**Run plugin-specific tests:**
```bash
pytest -m ext_tuner --cache-clear       # CSV Tuner Plugin tests only
pytest -m ext_profiler --cache-clear    # Profiler Plugin tests only
pytest -m ext_inspector --cache-clear   # Inspector Plugin tests only
```

**Run tests for specific collective operations (across all plugins):**
```bash
pytest -m allreduce --cache-clear      # AllReduce tests
pytest -m broadcast --cache-clear      # Broadcast tests
pytest -m reduce --cache-clear         # Reduce tests
pytest -m allgather --cache-clear      # AllGather tests
pytest -m reducescatter --cache-clear  # ReduceScatter tests
pytest -m alltoall --cache-clear       # AllToAll tests (profiler only)
pytest -m sendrecv --cache-clear       # SendRecv tests (profiler only)
```

**Combine markers to run specific tests:**
```bash
pytest -m "ext_profiler and allreduce" --cache-clear    # Profiler AllReduce tests only
pytest -m "ext_tuner and broadcast" --cache-clear       # Tuner Broadcast tests only
pytest -m "ext_inspector and allreduce" --cache-clear   # Inspector AllReduce tests only
```

### Run Tests with Log Output

To see live log output during test execution:

```bash
pytest -v -s --cache-clear
```

### Generate Test Report

To generate a detailed test report:

```bash
pytest --verbose --tb=short
```

## Additional Notes

- **Deactivating Virtual Environment**: When you're done testing, deactivate the virtual environment by running:
  ```bash
  deactivate
  ```

- **Log Files**: Test execution logs are stored in the `logs/` directory for later review and debugging.

- **Profiler Output**: Profiler plugin tests generate JSON trace files in the `profiler_dumps/` directory. These files contain detailed event traces that can be analyzed for debugging or performance analysis. The directory is automatically cleaned before each test session by the pytest fixture.

- **Inspector Output**: Inspector plugin tests generate JSONL log files in the `inspector_dumps/` directory, with one file per rank per test. Each line is a self-contained JSON record containing header, metadata, and collective performance data. The directory is automatically cleaned before each test session.