TheRock/rocm-systems/projects/rocm-core/rdhc

ROCm Deployment Health Check (RDHC)

Overview

RDHC is a comprehensive health check tool for ROCm deployments. It validates GPU presence, driver status, kernel parameters, library dependencies, and tests installed ROCm components.

Features

• Cross-Platform Support: Works on Ubuntu, RHEL, and SLES distributions
• Comprehensive Testing: GPU validation, driver checks, library dependencies, kernel parameters, and component-specific tests
• Dynamic Component Detection: Automatically identifies installed ROCm components
• Flexible Reporting: Pretty table output and JSON export options
• Configurable Verbosity: Support for verbose, normal, and silent modes
• Optional check control: Skip cluster-only or environment-limited tests; relax kernel parameter strictness (see [Optional CLI behavior])

Test Categories

Default Tests (Quick Mode)

1. GPU Presence - Detects AMD GPUs in the system
2. AMDGPU Driver - Validates driver installation and initialization
3. Kernel Parameters - Checks ROCm-related kernel settings
4. rocminfo - Validates ROCm information utility
5. rocm_agent_enumerator - Checks GPU agent enumeration
6. amd-smi - Tests AMD System Management Interface
7. Library Dependencies - Validates ROCm library dependencies
8. Environment Variables - Checks ROCm-related environment settings
9. Multinode Cluster Readiness - Validates network and MPI configuration
10. Atomic Operations - Checks if atomic operations are enabled on GPUs

Tests 9 and 10 can be skipped when not relevant (single-node systems, containers without full PCI/RDMA). Kernel parameter checks (3) can optionally be reported as warnings only instead of failures.

Component Tests (--all mode)

Tests installed ROCm components by compiling and executing example programs:

• HIP (hipcc, hip-runtime-amd)
• Math Libraries (hipBLAS, hipFFT, rocBLAS, rocFFT, etc.)
• Primitives (hipCUB, rocPRIM, rocThrust)
• Solvers (hipSOLVER, rocSOLVER, rocSPARSE)
• Deep Learning (MIOpen)
• Applications (from rocm-examples repository)

Output

The tool provides three types of output:

1. Console Output - Real-time test progress and results
2. Summary Tables - Formatted tables showing:
   • General system information
   • GPU device information
   • Firmware version information
   • Test results with status and details
3. JSON Export - Detailed results in JSON format for further analysis

Install dependency pip packages

sudo pip3 install -r requirements.txt

Usage

./rdhc.py -h
usage: ./rdhc.py [options]

ROCm Deployment Health Check Tool

optional arguments:
  -h, --help            show this help message and exit
  --quick               Run quick tests only (default)
  --all                 Default tests + Compile and executes simple program for each component.
  -v, --verbose         Enable verbose output
  -s, --silent          Silent mode (errors only)
  -j FILE, --json FILE  Export results to JSON file
  -d DIR, --dir DIR     Directory path for temporary files (must not exist; created with mode 0700)
  --cleanup             Remove auto-generated temp directory after tests complete
  --rocm-install-prefix DIR  ROCm installation prefix. If set, this path is used; otherwise `ROCM_PATH` env or `/opt/rocm` is used.
  --skip-multinode-readiness   Skip multinode cluster readiness test (e.g. single-node or no RDMA stack)
  --skip-atomic-operations     Skip PCIe atomic operations test (e.g. containers or limited lspci)
  --skip-optional-cluster-checks  Same as `--skip-multinode-readiness` and `--skip-atomic-operations` together
  --kernel-params-warnings-only   Treat kernel parameter check failures as warnings instead of errors

Usage examples

# Run quick test (default tests only)
./rdhc.py

# Run all tests including compile and execute the rocm-example program for each component
./rdhc.py --all

# Run all tests with verbose output
./rdhc.py --all -v

# Enable verbose output
./rdhc.py -v

# Run in silent mode (only errors shown)
./rdhc.py -s

# Export results to a specific JSON file
./rdhc.py --all --json rdhc-results.json

# Specify a directory for temp files (directory must not exist)
./rdhc.py --all -d /home/user/rdhc-run1/

# Auto-generate temp directory and clean it up after tests
./rdhc.py --all --cleanup

# Custom install prefix
./rdhc.py --rocm-install-prefix /usr/local/rocm

# Custom prefix and run all tests
./rdhc.py --rocm-install-prefix /usr/local/rocm --all -v

# Skip multinode + atomic checks (single-node or container-friendly)
./rdhc.py --skip-optional-cluster-checks

# Or skip only one of them
./rdhc.py --skip-multinode-readiness
./rdhc.py --skip-atomic-operations

# Kernel cmdline/sysctl checks: mismatches count as warnings, not FAIL
./rdhc.py --kernel-params-warnings-only

# Combine with other options
./rdhc.py --all --skip-optional-cluster-checks --kernel-params-warnings-only -v

Note on privileges: Most checks run without elevated privileges. Some checks (e.g., lspci -vvv for atomic operations) may require root access. Run as root when necessary, or use --skip-atomic-operations if running unprivileged.

Optional CLI behavior

These options improve usability when some checks are not applicable or cannot run reliably:

Option	Purpose
--skip-multinode-readiness	Skips the Multinode Cluster Readiness test (MPI, NIC drivers, RDMA modules). Use on single-node hosts or systems without an InfiniBand/RDMA stack.
--skip-atomic-operations	Skips the Atomic Operations test (lspci-based). Use in containers or hosts where PCIe capability details are not visible without full access.
--skip-optional-cluster-checks	Equivalent to passing both --skip-multinode-readiness and --skip-atomic-operations.
--kernel-params-warnings-only	For the Kernel Parameters test, entries that would normally count as errors are treated as warnings so the overall check does not FAIL solely on kernel/cmdline tuning differences.

Skipped tests are recorded with status NOT TESTED and a short reason in the report/JSON.

RDHC Environment VARIABLES

RDHC tool will use the following ENV variables and act accordingly if they are set.

# ROCm installation path can be set by the below ENV variable. Default is "/opt/rocm/"
export ROCM_PATH="/opt/rocm"

# For library dependency validation, the lib search depth can be set by the below ENV.
# Default is full depth. It checks for all the lib files in ROCM_PATH/lib/ folder recursively.
export LIBDIR_MAX_DEPTH=""

# if you want to check the libs only from the ROCM_PATH/lib/ folder set the depth as 1.
export LIBDIR_MAX_DEPTH=1

Troubleshooting

Python Package Installation Issues (Ubuntu 24.04)

If sudo pip3 install fails with an "externally-managed-environment" error (common in Ubuntu 24.04), use a Python virtual environment instead:

# Create a virtual environment (one-time setup)
python3 -m venv ~/rdhc-venv

# Activate the virtual environment
source ~/rdhc-venv/bin/activate

# Install required packages
pip3 install -r requirements.txt

# Run the tool (virtual environment activated)
./rdhc.py
./rdhc.py --all

Temporary Directory Behavior

When using --all mode, RDHC clones and builds rocm-examples:

• Without -d flag: A secure temporary directory is auto-generated in /tmp (e.g., /tmp/rdhc-abc123/) with mode 0700. By default, this directory persists after the run. Use --cleanup to remove it automatically.
• With -d flag: The specified directory is created with mode 0700. The directory must not already exist (for security reasons). If you need to reuse a directory, remove it first: rm -rf /path/to/dir

Example workflow for repeated runs:

# Auto-generated temp dir (persists)
./rdhc.py --all
# Creates /tmp/rdhc-abc123/ (check logs or report for actual path)

# Auto-generated temp dir with automatic cleanup
./rdhc.py --all --cleanup
# Creates and removes temp dir after tests complete

# Custom directory (must not exist)
./rdhc.py --all -d /tmp/my-rdhc-run1/   # First run with this path
rm -rf /tmp/my-rdhc-run1/                # Clean up before next run
./rdhc.py --all -d /tmp/my-rdhc-run1/   # Can reuse after cleanup

# Note: --cleanup has no effect when -d is specified

---

The tool is designed to be easily extended with additional component tests by adding new test methods following the naming convention test_check_component_name().