TheRock/rocm-systems/projects/rccl/test/common/ProcessIsolatedTestFramework.md

Process Isolated Test Runner

A lightweight C++ testing framework for running Google Test cases in isolated processes with clean environment settings.

Table of Contents

• Overview
• Why Use Process Isolation?
• Quick Start
• Core Concepts
• Parallel Execution
• API Reference
• Examples
• Best Practices
• Troubleshooting

---

Overview

ProcessIsolatedTestRunner is a framework that executes tests in separate processes using fork()+execv(). This ensures complete isolation between tests, particularly useful when testing code with static variables, one-time initialization, or environment-dependent behavior.

Key Features:

• ✅ Process-based test isolation via fork()+execv() (each test re-execs the binary fresh)
• ✅ Per-test environment variable management
• ✅ Configurable timeouts
• ✅ Sequential or parallel execution with bounded concurrency
• ✅ GPU-aware parallel scheduling (non-overlapping GPU assignment across concurrent tests)
• ✅ Thread-safe test registration
• ✅ Detailed test result reporting in registration order

Location: test/common/ProcessIsolatedTestRunner.hpp

---

Why use Process Isolation?

Problem: Static Variable Pollution

Consider this RCCL code with static variables:

void rcclSetP2pNetChunkSize(struct ncclComm* comm, int& chunkSize) {
  static int p2pNetChunkSize = RCCL_VALUE_UNSET;  // ← Static variable!

  if (p2pNetChunkSize == RCCL_VALUE_UNSET) {
    const char* inputStr = getenv("NCCL_P2P_NET_CHUNKSIZE");
    if (inputStr) {
      // Parse the environment variable value
      p2pNetChunkSize = parseValue(inputStr);  // e.g., "12345" → 12345
    } else {
      // No env var set, calculate value based on architecture...
      p2pNetChunkSize = calculateValue();
    }
  }
  chunkSize = p2pNetChunkSize;
}

How the static variable gets set:

1. First time called: p2pNetChunkSize == RCCL_VALUE_UNSET is true
2. Code reads environment variable with getenv("NCCL_P2P_NET_CHUNKSIZE")
3. If env var exists → parse its value (e.g., "12345" string) and assign to static variable
4. If env var doesn't exist → calculate default value and assign to static variable
5. Static variable is now set and persists for the lifetime of the process

Without Process Isolation:

TEST(MyTest, FirstTest) {
  setenv("NCCL_P2P_NET_CHUNKSIZE", "12345", 1);
  rcclSetP2pNetChunkSize(comm, chunkSize);
  // ✓ getenv() returns "12345"
  // ✓ Static variable p2pNetChunkSize gets set to 12345
  // ✓ chunkSize is now 12345
}

TEST(MyTest, SecondTest) {
  unsetenv("NCCL_P2P_NET_CHUNKSIZE");
  rcclSetP2pNetChunkSize(comm, chunkSize);
  // ❌ getenv() returns nullptr (env var cleared)
  // ❌ BUT: p2pNetChunkSize != RCCL_VALUE_UNSET (still 12345 from FirstTest!)
  // ❌ Code skips the if-block, never reads env var or recalculates
  // ❌ chunkSize is STILL 12345 from previous test!
  // This test will fail or produce incorrect results
}

The Problem: Static variables are initialized once per process and persist across multiple tests. Even if you change or clear environment variables, the static variable retains its old value.

With Process Isolation:

// Each test runs in a separate process
// Static variables are reset for each test
// ✅ Tests are truly independent

Common Use Cases

1. Testing environment variable behavior - When code reads env vars into static variables
2. Testing architecture-specific logic - Different GPU architectures with cached state
3. Testing initialization code - One-time initialization patterns
4. Testing configuration changes - When config is cached statically

---

Quick Start

Basic Example (Using Macros)

The simplest way to use ProcessIsolatedTestRunner is with the macros:

#include "common/ProcessIsolatedTestRunner.hpp"

TEST(Rcclwrap, MyIsolatedTest) {
  // Single test with environment variables - all in one call!
  RUN_ISOLATED_TEST_WITH_ENV("TestWithCleanEnvironment",
    []() {
      // This runs in a separate process
      const char* value = getenv("MY_VARIABLE");
      EXPECT_STREQ(value, "test_value");
      EXPECT_TRUE(someFunction());
    },
    {{"MY_VARIABLE", "test_value"}}
  );
}

TEST(Rcclwrap, MyIsolatedTests) {
  // Multiple tests with different configurations
  RUN_ISOLATED_TESTS(
    ProcessIsolatedTestRunner::TestConfig("Test1", []() {
      EXPECT_TRUE(checkCondition1());
    }),
    ProcessIsolatedTestRunner::TestConfig("Test2", []() {
      EXPECT_TRUE(checkCondition2());
    }).withEnvironment({{"VAR", "value"}}),
    ProcessIsolatedTestRunner::TestConfig("Test3", []() {
      EXPECT_TRUE(checkCondition3());
    }).withTimeout(std::chrono::seconds(60))
  );
}

Manual API (For Advanced Use Cases)

You can also use the API directly for more control:

#include "common/ProcessIsolatedTestRunner.hpp"

TEST(Rcclwrap, MyIsolatedTests) {
  // Register a test with environment variables
  ProcessIsolatedTestRunner::registerTest(
      ProcessIsolatedTestRunner::TestConfig(
          "TestWithCleanEnvironment",
          []() {
            // This runs in a separate process
            const char* value = getenv("MY_VARIABLE");
            EXPECT_STREQ(value, "test_value");

            // Your test logic here
            EXPECT_TRUE(someFunction());
          })
          .withEnvironment({{"MY_VARIABLE", "test_value"}})
  );

  // Execute all registered tests
  bool allTestsPassed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(allTestsPassed);
}

---

Core Concepts

1. Test Configuration (TestConfig)

Defines how a test should be executed:

TestConfig config(
    "TestName",           // Test name (for reporting)
    []() { /* logic */ }  // Test function (lambda or function pointer)
);

// Optional configurations
config.withEnvironment({{"VAR1", "value1"}, {"VAR2", "value2"}})
      .withTimeout(std::chrono::seconds(60))
      .withCleanEnvironment(false);  // Inherit parent environment

2. Test Registration

Tests must be registered before execution:

// Method 1: Full configuration
ProcessIsolatedTestRunner::registerTest(config);

// Method 2: Simple (name + logic only)
ProcessIsolatedTestRunner::registerTest("SimplTest", []() {
  EXPECT_TRUE(true);
});

// Method 3: With environment
ProcessIsolatedTestRunner::registerTest(
    "EnvTest",
    []() { /* logic */ },
    {{"ENV_VAR", "value"}}
);

3. Test Execution

⚠️ IMPORTANT: Tests do NOT run automatically after registration. You MUST explicitly call executeAllTests() to run them.

Execute all registered tests:

// Default options (continue on failure, no verbose logging)
bool passed = ProcessIsolatedTestRunner::executeAllTests();

// Custom options
ProcessIsolatedTestRunner::ExecutionOptions options;
options.stopOnFirstFailure = true;   // Stop after first failure
options.verboseLogging = true;       // Print detailed logs

bool passed = ProcessIsolatedTestRunner::executeAllTests(options);

Common Mistake:

// ❌ BAD: Tests registered but never executed!
TEST(MyTest, IsolatedTests) {
  ProcessIsolatedTestRunner::registerTest("Test1", []() { /* ... */ });
  ProcessIsolatedTestRunner::registerTest("Test2", []() { /* ... */ });
  // Missing executeAllTests() - tests will NOT run!
}

// ✅ GOOD: Tests registered and executed
TEST(MyTest, IsolatedTests) {
  ProcessIsolatedTestRunner::registerTest("Test1", []() { /* ... */ });
  ProcessIsolatedTestRunner::registerTest("Test2", []() { /* ... */ });
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);
}

4. Test Results

Each test produces a TestResult:

struct TestResult {
  std::string testName;               // Name of the test
  bool passed;                        // Whether the test passed
  bool skipped;                       // Whether the test was skipped
  int exitCode;                       // Process exit code
  pid_t processId;                    // Process ID that ran the test
  std::chrono::milliseconds duration; // Execution duration
  std::string errorMessage;           // Error message if failed
  std::unordered_map<std::string, std::string> environment;  // Env used
};

---

Parallel Execution

By default, tests run one at a time (maxParallelJobs = 1). Setting maxParallelJobs > 1 launches up to that many child processes simultaneously, reducing total wall-clock time for large test suites.

Concurrency control

ProcessIsolatedTestRunner::ExecutionOptions opts;
opts.maxParallelJobs = 4;   // up to 4 tests run at the same time
                            // kAutoParallelism = GPU pool size (or hardware_concurrency() if no pool)
                            // 1 = sequential (default)

RUN_ISOLATED_TESTS_WITH_OPTIONS(opts,
    ProcessIsolatedTestRunner::TestConfig("Test1", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("Test2", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("Test3", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("Test4", []() { ... })
);

Output ordering: Results are always printed and reported in registration order, regardless of which child process finishes first.

stopOnFirstFailure: When set, the runner stops launching new tests once a failure is detected. Tests that are already running are allowed to finish normally — they are not killed early.

GPU-aware scheduling

When tests run in parallel, multiple child processes would otherwise all try to use the same GPU (e.g., device 0), conflicting with each other. The runner solves this by maintaining a GPU slot pool and assigning each child process a non-overlapping subset of physical device indices, injected as HIP_VISIBLE_DEVICES before execv().

How the pool is built

ExecutionOptions::gpuPool holds the physical device indices to distribute. If left empty (the default), the runner auto-detects the pool in this priority order:

1. Parse HIP_VISIBLE_DEVICES from the environment (comma-separated indices)
2. Count GPU nodes in /sys/class/kfd/kfd/topology/nodes with a non-zero gpu_id → pool [0, 1, ..., N-1]

If neither source yields a non-empty pool, executeAllTests() records a non-fatal GTest failure and returns false immediately.

// Override auto-detection to use only GPUs 0 and 1:
opts.gpuPool = {0, 1};

How slots are assigned

Each TestConfig declares how many GPU slots it needs via withNumGpus():

numGpus value	Meaning
0 (default)	CPU-only test — no GPU slot acquired, runs freely without HIP_VISIBLE_DEVICES restriction
1	one dedicated GPU from the pool
N	exactly N GPU slots — test blocks until N slots are free
N > pool size	clamped to the full pool — test runs exclusively (all slots held) to prevent contention with siblings

Before launching each child, the runner atomically waits until both a concurrency slot (active < maxParallelJobs) and enough free GPU slots are available. The assigned indices are written into HIP_VISIBLE_DEVICES inside the fork child so the re-exec'd process sees only its GPUs. When the child exits the slots are released and the next waiting test can proceed.

Example: parallel tests with per-test GPU declaration

ProcessIsolatedTestRunner::ExecutionOptions opts;
opts.maxParallelJobs = 4;
// gpuPool auto-detected (e.g. [0,1,2,3] on a 4-GPU node)

RUN_ISOLATED_TESTS_WITH_OPTIONS(opts,
    // Each of these gets one GPU; up to 4 run simultaneously
    ProcessIsolatedTestRunner::TestConfig("SingleGpuTest_A", []() {
        // HIP_VISIBLE_DEVICES is set to e.g. "0" — test sees only that GPU
        HIPCALL(hipSetDevice(0));  // device 0 here = whichever physical GPU was assigned
        /* ... */
    }),
    ProcessIsolatedTestRunner::TestConfig("SingleGpuTest_B", []() {
        HIPCALL(hipSetDevice(0));
        /* ... */
    }),

    // This test needs 2 GPUs — it blocks until 2 slots are free
    ProcessIsolatedTestRunner::TestConfig("TwoGpuTest", []() {
        // HIP_VISIBLE_DEVICES = e.g. "2,3"
        // hipGetDeviceCount() returns 2; hipSetDevice(0/1) map to physical 2/3
        /* ... */
    }).withNumGpus(2),

    // This test requires all 4 GPUs — blocks until every slot is free,
    // so no other GPU test can run concurrently (exclusive execution).
    ProcessIsolatedTestRunner::TestConfig("ExclusiveTest", []() {
        /* ... */
    }).withNumGpus(4)   // request entire 4-GPU pool → runs exclusively
);

What the child process sees

Inside the re-exec'd child, HIP_VISIBLE_DEVICES is set to the assigned indices (e.g. "2,3"). HIP then remaps those to logical device IDs 0 and 1. So a test that calls hipSetDevice(0) always gets the first device from its assigned subset — it never touches a GPU that belongs to another concurrent test.

System GPUs:  [0]  [1]  [2]  [3]
              └─ Test A ─┘  └─ Test B ─┘   ← running simultaneously
  HIP_VISIBLE_DEVICES="0,1"  "2,3"
  hipSetDevice(0) → phys 0   hipSetDevice(0) → phys 2

When GPU slot management is disabled

GPU slot management is skipped (no HIP_VISIBLE_DEVICES override) in two cases:

• Sequential mode (maxParallelJobs = 1): no concurrent tests, no conflict possible.
• No GPUs detected: auto-detection found zero devices.

In both cases tests see the full device list as usual.

Interaction with withEnvironment({{"HIP_VISIBLE_DEVICES", ...}})

In parallel mode, the runner injects HIP_VISIBLE_DEVICES after applying the test's own environment variables. Any HIP_VISIBLE_DEVICES set via withEnvironment() or setVariable() is overridden by the slot manager's assignment; a warning is logged to stderr when this happens. Use withNumGpus(N) to declare device requirements rather than hard-coding device indices.

---

API Reference

Macros (Recommended)

These macros provide the simplest way to use ProcessIsolatedTestRunner with minimal boilerplate.

RUN_ISOLATED_TEST(test_name, test_body)

Register and execute a single isolated test.

RUN_ISOLATED_TEST("MySimpleTest", []() {
  EXPECT_TRUE(someFunction());
});

RUN_ISOLATED_TEST_WITH_ENV(test_name, test_body, ...)

Register and execute a single isolated test with environment variables.

Uses variadic macros (... and __VA_ARGS__) to automatically handle commas in initializer lists without requiring extra parentheses.

RUN_ISOLATED_TEST_WITH_ENV("MyEnvTest",
  []() {
    const char* value = getenv("MY_VAR");
    EXPECT_STREQ(value, "expected_value");
  },
  {{"MY_VAR", "expected_value"}}
);

// Multiple environment variables work naturally:
RUN_ISOLATED_TEST_WITH_ENV("MultiEnvTest",
  []() { /* test code */ },
  {{"VAR1", "val1"}, {"VAR2", "val2"}, {"VAR3", "val3"}}  // Commas handled automatically
);

Note: The macro uses __VA_ARGS__ internally, which automatically handles commas in the environment variable initializer list. Users don't need to worry about preprocessor comma issues.

RUN_ISOLATED_TESTS(...)

Register and execute multiple isolated tests with various configurations.

RUN_ISOLATED_TESTS(
  ProcessIsolatedTestRunner::TestConfig("Test1", []() { ... }),
  ProcessIsolatedTestRunner::TestConfig("Test2", []() { ... })
    .withEnvironment({{"VAR", "value"}}),
  ProcessIsolatedTestRunner::TestConfig("Test3", []() { ... })
    .withTimeout(std::chrono::seconds(60))
);

RUN_ISOLATED_TESTS_WITH_OPTIONS(options, ...)

Register and execute multiple isolated tests with custom execution options.

ProcessIsolatedTestRunner::ExecutionOptions opts;
opts.stopOnFirstFailure = true;
opts.verboseLogging = true;

RUN_ISOLATED_TESTS_WITH_OPTIONS(opts,
  ProcessIsolatedTestRunner::TestConfig("Test1", []() { ... }),
  ProcessIsolatedTestRunner::TestConfig("Test2", []() { ... })
);

Main Methods (For Manual Use)

registerTest()

Register a test for later execution. Registration fails with a diagnostic if:

• The name contains a null byte (would be silently truncated by setenv/getenv).
• The name is a duplicate of an already-registered test (the re-exec child matches on the first occurrence, so the second would never run).

// Variant 1: Full configuration
static void registerTest(const TestConfig& config);

// Variant 2: Simple registration
static void registerTest(
    const std::string& name,
    std::function<void()> testLogic
);

// Variant 3: With environment
static void registerTest(
    const std::string& name,
    std::function<void()> testLogic,
    const std::unordered_map<std::string, std::string>& env
);

executeAllTests()

Execute all registered tests.

static bool executeAllTests(
    const ExecutionOptions& options = ExecutionOptions()
);

Returns: true if all tests passed, false if any failed.

Execution mode is controlled by ExecutionOptions::maxParallelJobs:

• 1 (default) — sequential, one test at a time
• N > 1 — up to N tests run simultaneously with GPU-aware scheduling
• ExecutionOptions::kAutoParallelism (0) — parallelism = GPU pool size, or std::thread::hardware_concurrency() when no GPU pool is available

Note: This method automatically clears all test registrations and results after execution, ensuring a clean state for the next test suite. Users do not need to call clear() manually.

getTestResults()

Retrieve detailed results from the last execution.

static std::vector<TestResult> getTestResults();

clear()

Clear all registered tests and results.

static void clear();

Note: Calling this method manually is typically not necessary, as executeAllTests() automatically clears registrations after execution. This method is primarily useful for advanced use cases or when tests are registered but not executed.

⚠️ Automatic Warning: If clear() is called when tests have been registered but not fully executed, it will automatically print a warning to stderr:

⚠️  WARNING: ProcessIsolatedTestRunner::clear() called with 2 unexecuted test(s)!
   Registered: 2 test(s)
   Executed:   0 test(s)
   Did you forget to call executeAllTests()?

getTestCount()

Get the number of currently registered tests (before execution).

static size_t getTestCount();

Use case: Verify that tests were actually registered and executed.

TEST(MyTest, VerifyExecution) {
  ProcessIsolatedTestRunner::clear();

  // Register tests
  ProcessIsolatedTestRunner::registerTest("Test1", []() { /* ... */ });
  ProcessIsolatedTestRunner::registerTest("Test2", []() { /* ... */ });

  // Check registration count
  size_t registeredCount = ProcessIsolatedTestRunner::getTestCount();
  EXPECT_EQ(registeredCount, 2) << "Expected 2 tests to be registered";

  // Execute
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);

  // Verify execution count
  auto results = ProcessIsolatedTestRunner::getTestResults();
  EXPECT_EQ(results.size(), registeredCount)
      << "Registered " << registeredCount << " tests but only "
      << results.size() << " executed";
}

ExecutionOptions Fields

Field	Type	Default	Description
stopOnFirstFailure	bool	false	Stop launching new tests after the first failure (in-flight tests finish)
verboseLogging	bool	true	Print per-test status lines
maxParallelJobs	size_t	1	Max concurrent child processes. kAutoParallelism (0) = GPU pool size (or hardware_concurrency if no pool), 1 = sequential
gpuPool	vector<int>	{}	Physical GPU indices to distribute. Empty = auto-detect

TestConfig Methods

withEnvironment()

Set environment variables for the test.

TestConfig& withEnvironment(
    const std::unordered_map<std::string, std::string>& env
);

Note: Variables are set in the child process only.

withTimeout()

Set a timeout for test execution.

TestConfig& withTimeout(std::chrono::seconds timeoutSeconds);

Default: 30 seconds

withCleanEnvironment()

Control whether to inherit parent process environment.

TestConfig& withCleanEnvironment(bool inherit = true);

Default: true (inherits parent environment)

withNumGpus()

Declare how many GPU slots this test needs during parallel execution.

TestConfig& withNumGpus(size_t n);

Value	Meaning
0 (default)	CPU-only — no GPU slot acquired; HIP_VISIBLE_DEVICES is not overridden
1	one dedicated GPU from the pool
N	exactly N GPU slots — test blocks until N slots are free
N > pool size	clamped to pool size — test runs exclusively (holds all slots)

The runner injects HIP_VISIBLE_DEVICES into the child process with the assigned device indices. Has no effect in sequential mode (maxParallelJobs = 1).

ProcessIsolatedTestRunner::TestConfig("MultiGpuTest", []() {
    // Sees exactly 2 GPUs via HIP_VISIBLE_DEVICES
    int count;
    hipGetDeviceCount(&count);
    EXPECT_EQ(count, 2);
}).withNumGpus(2)

---

Examples

Note: The examples below use helper functions from RcclWrapTests.cpp:

// Helper to create a mock NCCL communicator with specified architecture and ranks
static void CreateMockComm(ncclComm_t &mockComm,
                           struct ncclTopoSystem &mockTopo,
                           struct ncclTopoNode &mockGpuNode,
                           const char *arch,
                           int nRanks);

// Helper to cleanup a mock communicator
static void CleanupMockComm(ncclComm_t &mockComm);

Example 1: Testing Environment Variable Behavior

TEST(Rcclwrap, EnvironmentVariableTests) {
  // Test 1: With environment variable set
  ProcessIsolatedTestRunner::registerTest(
      ProcessIsolatedTestRunner::TestConfig(
          "WithEnvVarSet",
          []() {
            ncclComm_t mockComm = nullptr;
            struct ncclTopoSystem mockTopo;
            struct ncclTopoNode mockGpuNode;
            CreateMockComm(mockComm, mockTopo, mockGpuNode, "gfx942", 128);

            int chunkSize = RCCL_VALUE_UNSET;
            rcclSetP2pNetChunkSize(mockComm, chunkSize);

            // Should use default architecture-based value
            EXPECT_EQ(chunkSize, 1 << 19);

            CleanupMockComm(mockComm);
          })
          .withEnvironment({{"NCCL_P2P_NET_CHUNKSIZE", "999999"}})
  );

  // Test 2: Without environment variable (clean state)
  ProcessIsolatedTestRunner::registerTest(
      ProcessIsolatedTestRunner::TestConfig(
          "WithoutEnvVar",
          []() {
            // Verify environment is clean
            const char* value = getenv("NCCL_P2P_NET_CHUNKSIZE");
            EXPECT_EQ(value, nullptr);

            // Test default behavior
            ncclComm_t mockComm = nullptr;
            struct ncclTopoSystem mockTopo;
            struct ncclTopoNode mockGpuNode;
            CreateMockComm(mockComm, mockTopo, mockGpuNode, "gfx942", 32);

            int chunkSize = RCCL_VALUE_UNSET;
            rcclSetP2pNetChunkSize(mockComm, chunkSize);
            EXPECT_EQ(chunkSize, 1 << 17);  // Default for < 64 ranks

            CleanupMockComm(mockComm);
          })
  );

  // Execute both tests in isolated processes
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);
}

Example 2: Testing Multiple Architectures

TEST(Rcclwrap, ArchitectureTests) {
  struct TestCase {
    std::string name;
    std::string arch;
    int ranks;
    int expectedChunkSize;
  };

  std::vector<TestCase> testCases = {
    {"GFX942_SmallRanks", "gfx942", 32, 1 << 17},
    {"GFX942_LargeRanks", "gfx942", 128, 1 << 19},
    {"GFX950_SmallRanks", "gfx950", 8, 1 << 17},
    {"GFX950_MediumRanks", "gfx950", 24, 1 << 18},
    {"GFX950_LargeRanks", "gfx950", 64, 1 << 19},
  };

  for (const auto& tc : testCases) {
    ProcessIsolatedTestRunner::registerTest(
        ProcessIsolatedTestRunner::TestConfig(
            tc.name,
            [tc]() {
              ncclComm_t mockComm = nullptr;
              struct ncclTopoSystem mockTopo;
              struct ncclTopoNode mockGpuNode;
              CreateMockComm(mockComm, mockTopo, mockGpuNode, tc.arch.c_str(), tc.ranks);

              int chunkSize = RCCL_VALUE_UNSET;
              rcclSetP2pNetChunkSize(mockComm, chunkSize);

              EXPECT_EQ(chunkSize, tc.expectedChunkSize)
                  << "Failed for " << tc.arch << " with " << tc.ranks << " ranks";

              CleanupMockComm(mockComm);
            })
    );
  }

  ProcessIsolatedTestRunner::ExecutionOptions options;
  options.verboseLogging = true;
  options.stopOnFirstFailure = false;  // Run all tests even if one fails

  bool passed = ProcessIsolatedTestRunner::executeAllTests(options);
  EXPECT_TRUE(passed);
}

Example 3: Testing with Timeouts

TEST(Rcclwrap, TimeoutHandling) {
  // Test that completes quickly
  ProcessIsolatedTestRunner::registerTest(
      ProcessIsolatedTestRunner::TestConfig(
          "FastTest",
          []() {
            EXPECT_TRUE(true);
          })
          .withTimeout(std::chrono::seconds(5))
  );

  // Test with longer timeout for complex operations
  ProcessIsolatedTestRunner::registerTest(
      ProcessIsolatedTestRunner::TestConfig(
          "SlowTest",
          []() {
            // Simulate slow operation
            std::this_thread::sleep_for(std::chrono::seconds(2));
            EXPECT_TRUE(true);
          })
          .withTimeout(std::chrono::seconds(10))
  );

  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);
}

Example 4: Stop on First Failure

TEST(Rcclwrap, CriticalTests) {
  // Register multiple critical tests
  ProcessIsolatedTestRunner::registerTest(
      "CriticalTest1", []() { EXPECT_TRUE(checkCriticalCondition1()); });

  ProcessIsolatedTestRunner::registerTest(
      "CriticalTest2", []() { EXPECT_TRUE(checkCriticalCondition2()); });

  ProcessIsolatedTestRunner::registerTest(
      "CriticalTest3", []() { EXPECT_TRUE(checkCriticalCondition3()); });

  // Stop on first failure - don't waste time if critical tests fail
  ProcessIsolatedTestRunner::ExecutionOptions options;
  options.stopOnFirstFailure = true;

  bool passed = ProcessIsolatedTestRunner::executeAllTests(options);
  EXPECT_TRUE(passed) << "Critical test suite failed";
}

---

Best Practices

1. Use Macros for Simple Cases

// ✅ GOOD: Simple and clean using macros
TEST(MyTest, SimpleIsolatedTest) {
  RUN_ISOLATED_TEST("CheckSomething", []() {
    EXPECT_TRUE(checkSomething());
  });
}

// ❌ MORE VERBOSE: Manual registration (still valid for complex cases)
TEST(MyTest, SimpleIsolatedTest) {
  ProcessIsolatedTestRunner::registerTest("CheckSomething", []() {
    EXPECT_TRUE(checkSomething());
  });
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);
}

2. Always Execute Registered Tests (When Using Manual API)

TEST(MyTest, IsolatedTests) {
  // Register tests
  ProcessIsolatedTestRunner::registerTest(/* ... */);

  // ✅ IMPORTANT: Don't forget to execute!
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);
}

When Using Manual API (Optional Verification):

You can verify that tests were registered and executed:

TEST(MyTest, IsolatedTests) {
  // Register tests
  ProcessIsolatedTestRunner::registerTest("Test1", []() { /* ... */ });
  ProcessIsolatedTestRunner::registerTest("Test2", []() { /* ... */ });

  // Get count of registered tests
  size_t registeredCount = ProcessIsolatedTestRunner::getTestCount();
  EXPECT_EQ(registeredCount, 2) << "Expected 2 tests to be registered";

  // Execute all tests (automatically clears after execution)
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);

  // Optional: Verify execution count matches registration count
  auto results = ProcessIsolatedTestRunner::getTestResults();
  EXPECT_EQ(results.size(), registeredCount)
      << "Registered " << registeredCount << " but executed " << results.size();
}

3. Use Descriptive Test Names

// ❌ BAD: Vague name
RUN_ISOLATED_TEST("Test1", []() { /* ... */ });

// ✅ GOOD: Descriptive name
RUN_ISOLATED_TEST("GFX942_LargeRanks_P2PChunkSize_ExpectHighValue",
  []() { /* ... */ }
);

4. Group Related Tests

TEST(Rcclwrap, AllP2PChunkSizeTests) {
  // Using macros to group related tests
  RUN_ISOLATED_TESTS(
    ProcessIsolatedTestRunner::TestConfig("GFX942_Test1", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("GFX942_Test2", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("GFX950_Test1", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("GFX950_Test2", []() { ... })
  );
}

5. Use Options for Better Control

// For debugging: verbose + stop on failure
ProcessIsolatedTestRunner::ExecutionOptions debugOptions;
debugOptions.stopOnFirstFailure = true;
debugOptions.verboseLogging = true;

RUN_ISOLATED_TESTS_WITH_OPTIONS(debugOptions,
  ProcessIsolatedTestRunner::TestConfig("Test1", []() { ... }),
  ProcessIsolatedTestRunner::TestConfig("Test2", []() { ... })
);

// For CI: run all tests, collect all failures
ProcessIsolatedTestRunner::ExecutionOptions ciOptions;
ciOptions.stopOnFirstFailure = false;
ciOptions.verboseLogging = false;

RUN_ISOLATED_TESTS_WITH_OPTIONS(ciOptions,
  ProcessIsolatedTestRunner::TestConfig("Test1", []() { ... }),
  ProcessIsolatedTestRunner::TestConfig("Test2", []() { ... })
);

6. Set Appropriate Timeouts

// ✅ GOOD: Different timeouts for different test types
RUN_ISOLATED_TESTS(
  ProcessIsolatedTestRunner::TestConfig("QuickTest", []() { ... })
    .withTimeout(std::chrono::seconds(5)),
  ProcessIsolatedTestRunner::TestConfig("NormalTest", []() { ... })
    .withTimeout(std::chrono::seconds(30)),
  ProcessIsolatedTestRunner::TestConfig("SlowTest", []() { ... })
    .withTimeout(std::chrono::seconds(120))
);

// ❌ BAD: Same long timeout for everything
RUN_ISOLATED_TESTS(
  ProcessIsolatedTestRunner::TestConfig("Test1", []() { ... })
    .withTimeout(std::chrono::seconds(300)),
  ProcessIsolatedTestRunner::TestConfig("Test2", []() { ... })
    .withTimeout(std::chrono::seconds(300))
);

7. Clean Up Resources in Tests

RUN_ISOLATED_TEST("ResourceTest", []() {
  ncclComm_t comm = nullptr;
  struct ncclTopoSystem topo;
  struct ncclTopoNode gpuNode;
  CreateMockComm(comm, topo, gpuNode, "gfx942", 32);

  try {
    // Your test logic
    EXPECT_TRUE(someTest(comm));

    // ✅ GOOD: Clean up in all paths
    CleanupMockComm(comm);
  } catch (...) {
    CleanupMockComm(comm);
    throw;
  }
});

8. Use RAII for GPU Resource Management

When tests allocate GPU memory, use RAII wrappers to ensure cleanup:

// ✅ GOOD: RAII ensures cleanup even on failure
struct GPUBuffer {
  void* ptr = nullptr;
  size_t size;

  GPUBuffer(size_t s) : size(s) {
    hipError_t err = hipMalloc(&ptr, size);
    ASSERT_EQ(err, hipSuccess);
  }

  ~GPUBuffer() {
    if (ptr) {
      hipFree(ptr);
      ptr = nullptr;
    }
  }

  // Prevent copying
  GPUBuffer(const GPUBuffer&) = delete;
  GPUBuffer& operator=(const GPUBuffer&) = delete;
};

RUN_ISOLATED_TEST("GPUTest", []() {
  GPUBuffer buffer(1024);  // Automatically cleaned up
  // ... test logic ...
  // No manual cleanup needed - destructor handles it
});

// ❌ BAD: Manual cleanup can be forgotten
RUN_ISOLATED_TEST("GPUTest", []() {
  void* buffer;
  hipMalloc(&buffer, 1024);
  // ... test logic ...
  // If test fails before this line, buffer leaks!
  hipFree(buffer);
});

9. Avoid GPU Initialization in Test Fixtures

When using process isolation, avoid initializing GPU resources in test fixture SetUp() methods:

// ❌ BAD: GPU initialization in fixture (runs in parent process)
class GPUTests : public ::testing::Test {
protected:
  void SetUp() override {
    hipMalloc(&gpuBuffer, 1024);  // Parent process — unsafe; HIP state
                                  // must not be inherited across fork()+execv()
  }
  void* gpuBuffer;
};

// ✅ GOOD: GPU initialization inside isolated test
class GPUTests : public ::testing::Test {
  // Empty fixture or only CPU resources in SetUp()
};

TEST_F(GPUTests, MyTest) {
  RUN_ISOLATED_TEST("GPUOperation", []() {
    void* gpuBuffer;
    hipMalloc(&gpuBuffer, 1024);  // Child process only - safe!
    // ... test logic ...
    hipFree(gpuBuffer);
  });
}

// ✅ EVEN BETTER: Use RAII + helper structure
struct GPUTestEnvironment {
  void* buffer;
  void setup() { hipMalloc(&buffer, 1024); }
  void cleanup() { if (buffer) hipFree(buffer); }
  ~GPUTestEnvironment() { cleanup(); }
};

TEST_F(GPUTests, MyTest) {
  RUN_ISOLATED_TEST("GPUOperation", []() {
    GPUTestEnvironment env;
    env.setup();
    // ... test logic ...
    env.cleanup();  // Explicit + destructor cleanup
  });
}

---

Troubleshooting

Test Hangs / Times Out

Symptom: Test never completes, eventually times out.

Solutions:

1. Increase timeout: .withTimeout(std::chrono::seconds(120))
2. Check for deadlocks in test logic
3. Enable verbose logging to see where it hangs:

   options.verboseLogging = true;
   

Environment Variables Not Being Set

Symptom: getenv() returns nullptr in test.

Solutions:

1. Verify environment variable name is correct
2. Check that you're calling withEnvironment():

   config.withEnvironment({{"VAR_NAME", "value"}})
   

3. Verify the test is actually executing (check test name)

Tests Pass Individually but Fail Together

Symptom: Individual tests pass, but fail when run in a suite.

Cause: This is the exact problem that ProcessIsolatedTestRunner solves!

Solution: Already solved - each test runs in isolated process. If you're still seeing this, check:

1. Are you using executeAllTests() correctly?
2. Are there shared external resources (files, network, etc.)?

Fork Failures

Symptom: Error messages about fork() failing.

Solutions:

1. Check system resource limits: ulimit -u (max processes)
2. Reduce number of tests or run in smaller batches
3. Check for resource leaks in parent process

Test Results Not Available

Symptom: getTestResults() returns empty vector.

Solution:

// Call executeAllTests() first
ProcessIsolatedTestRunner::executeAllTests();

// Then get results
auto results = ProcessIsolatedTestRunner::getTestResults();

Tests Registered but Never Executed

Symptom: Tests pass but you suspect they didn't actually run.

Cause: Forgot to call executeAllTests() after registration.

Detection:

TEST(MyTest, IsolatedTests) {
  // Register tests
  ProcessIsolatedTestRunner::registerTest("Test1", []() { EXPECT_TRUE(true); });
  ProcessIsolatedTestRunner::registerTest("Test2", []() { EXPECT_TRUE(true); });

  // ❌ FORGOT TO CALL executeAllTests()!

  // Later, when the test ends, registered tests are lost
}

Solution:

TEST(MyTest, IsolatedTests) {
  // Register tests
  ProcessIsolatedTestRunner::registerTest("Test1", []() { EXPECT_TRUE(true); });
  ProcessIsolatedTestRunner::registerTest("Test2", []() { EXPECT_TRUE(true); });

  // ✅ ALWAYS execute registered tests
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);

  // ✅ Optionally verify execution count
  auto results = ProcessIsolatedTestRunner::getTestResults();
  EXPECT_EQ(results.size(), 2) << "Expected 2 tests to execute";
}

Prevention: Always verify that getTestResults().size() matches your expected number of tests:

// After execution
auto results = ProcessIsolatedTestRunner::getTestResults();
EXPECT_EQ(results.size(), expectedTestCount)
    << "Test count mismatch - some tests may not have executed";

---

Implementation Details

How It Works

1. Registration Phase:

   • Tests are registered into a static vector
   • Each test gets a TestConfig with name, logic, and environment

2. Execution Phase (exec-only isolation):

   • Parent detects the GPU pool once (from HIP_VISIBLE_DEVICES or KFD sysfs); fails with a non-fatal GTest error and returns false if the pool is empty
   • For each test the parent:
     1. Acquires a concurrency slot (active < maxParallelJobs) and GPU slots (numGpus free indices) — atomically under one condition variable
     2. fork() creates a child
     3. Inside the fork child: environment variables are applied, then HIP_VISIBLE_DEVICES is set to the assigned GPU indices, then execv("/proc/self/exe") re-executes the binary with --gtest_filter pointing at the parent test and a sentinel env var naming the specific isolated test to run
     4. The re-exec'd process finds the sentinel, runs the matching lambda, flushes coverage, and calls _exit()
     5. Parent thread drains stdout/stderr pipes and calls waitpid() for its child
     6. Concurrency and GPU slots are released; the next waiting test can start
   • In parallel mode, each test runs in a background std::async thread; the parent maintains a bounded sliding window of maxParallelJobs active tasks

3. Result Collection:

   • Exit codes are captured from child processes
   • Timing information is recorded
   • All results are stored and reported in registration order

4. Automatic Cleanup:

   • After execution completes, executeAllTests() automatically clears all test registrations and results
   • This ensures a clean state for the next test suite without manual intervention

Why fork()+execv() instead of plain fork()

Plain fork() after HIP has been initialized is unsafe — the GPU runtime state (device contexts, memory handles, threads) is copied into the child but is not valid there. execv() replaces the child's entire address space with a fresh binary image, so the child starts with no inherited runtime state. This also ensures the LLVM profile runtime initializes with the correct child PID. Before execv() the fork child sets LLVM_PROFILE_FILE=rccl_tests_%p_%m.profraw (with overwrite=0) so each re-exec'd process writes a unique file. A LLVM_PROFILE_FILE already in the environment — from CI or via withEnvironment() — takes precedence.

The re-exec'd child skips GPU enumeration calls in EnvVars (e.g., hipGetDeviceCount, getArchInfo) because they are irrelevant there and concurrent hipGetDeviceCount forks cause KFD file-descriptor contention.

Exit Codes

enum RcclTestCode {
  RCCL_TEST_SUCCESS = 0,           // Test passed
  RCCL_TEST_FAILURE = 1,           // Test failed (assertion)
  RCCL_TEST_UNKNOWN_EXCEPTION = 2, // Uncaught exception
  RCCL_TEST_TIMEOUT = 3,           // Test timed out
  RCCL_TEST_SKIPPED = 4            // Test was skipped
};

Thread Safety

The framework uses mutexes for thread-safe operations:

• Test registration (write)
• Result recording (write)
• Result retrieval (read)

---

Limitations

1. Process Overhead: Each test forks and re-execs the binary (higher overhead than plain fork())
2. Linux/Unix Only: Uses fork()+execv() — not available on Windows
3. No Shared State: Tests cannot share data between processes
4. GPU pool is fixed at launch: The GPU pool is detected once before the first test runs; hot-plug changes during a run are not reflected

---

FAQ

Q: When should I use ProcessIsolatedTestRunner vs regular Google Test?

A: Use ProcessIsolatedTestRunner when:

• Testing code with static variables
• Testing environment variable behavior
• Testing one-time initialization
• Need guaranteed clean state between tests

Use regular Google Test when:

• Tests are truly independent
• No static state concerns
• Need parallel execution
• Testing simple units

Q: Can I use this with MPI tests?

A: Not directly. Process Isolated test runner is for single-process tests. For MPI tests, use MPI Test Runner instead. Process Isolated test runner is currently hooked into rccl-UnitTestsFixtures binary and MPI test runner is hooked into rccl-UnitTestsMPI binary. These are two independent implementation.

Q: How do I debug a test that's running in an isolated process?

A:

1. Enable verbose logging
2. Add print statements in your test lambda
3. Temporarily run the test logic outside the framework
4. Use GDB

Q: Can I run tests in parallel?

A: Yes. Set ExecutionOptions::maxParallelJobs to a value greater than 1. The runner spawns up to that many child processes simultaneously and automatically assigns non-overlapping GPU subsets to each so concurrent tests do not interfere on the GPU. Results are always reported in registration order.

ProcessIsolatedTestRunner::ExecutionOptions opts;
opts.maxParallelJobs = 4;  // 4 concurrent tests

RUN_ISOLATED_TESTS_WITH_OPTIONS(opts,
    ProcessIsolatedTestRunner::TestConfig("A", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("B", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("C", []() { ... }),
    ProcessIsolatedTestRunner::TestConfig("D", []() { ... })
);

See the Parallel Execution section for full details on GPU slot management.

Q: Does this work with CTest/CMake?

A: Yes! The tests are still Google Test cases, so they work with standard test runners.

Q: Should I use the macros or the manual API?

A: Use the macros (RUN_ISOLATED_TEST, RUN_ISOLATED_TESTS, etc.) for most cases - they're simpler and less error-prone. Use the manual API (registerTest() + executeAllTests()) only when you need more control over the registration/execution flow, such as:

• Dynamically generating test configurations at runtime
• Sharing test registration logic across multiple TEST blocks
• Advanced control flow scenarios

Q: Do tests run automatically after registration, or do I need to call executeAllTests()?

A: You MUST call executeAllTests() explicitly. Tests do NOT run automatically. If you forget to call it, your tests will be silently ignored. Always follow this pattern:

TEST(MyTest, IsolatedTests) {
  ProcessIsolatedTestRunner::registerTest("MyTest", []() { /* ... */ });

  // ✅ REQUIRED: Execute the tests
  bool passed = ProcessIsolatedTestRunner::executeAllTests();
  EXPECT_TRUE(passed);
}

Q: How can I detect if I forgot to execute registered tests?

A: After executeAllTests(), verify that getTestResults().size() matches your expected test count:

// Register N tests
ProcessIsolatedTestRunner::registerTest("Test1", []() { /* ... */ });
ProcessIsolatedTestRunner::registerTest("Test2", []() { /* ... */ });

// Execute
bool passed = ProcessIsolatedTestRunner::executeAllTests();

// Verify count
auto results = ProcessIsolatedTestRunner::getTestResults();
EXPECT_EQ(results.size(), 2) << "Expected 2 tests to run";

Q: Do I need to call clear() manually?

A: No. The clear() method is only useful for advanced use cases where you need to clear tests that were registered but never executed. If you manually call clear() when tests were registered but not executed, it will warn you:

⚠️  WARNING: ProcessIsolatedTestRunner::clear() called with 2 unexecuted test(s)!
   Registered: 2 test(s)
   Executed:   0 test(s)
   Did you forget to call executeAllTests()?

---

See Also

• ProcessIsolatedTestRunner.hpp - Full API documentation
• ProcessIsolatedTestRunner.cpp - Implementation details
• RcclWrapTests.cpp - Usage examples