TheRock/rocm-libraries/projects/rpp/utilities/test_suite/OPENCV/benchmarks

OpenCV vs RPP Host Benchmarking

This directory contains a comprehensive benchmarking suite that compares the performance of OpenCV operations against AMD's RPP (ROCm Performance Primitives) library on the host CPU.

Overview

The opencv_vs_rpp_host_benchmarking performs side-by-side performance comparisons of 50+ image processing operations including:

• Color adjustments (brightness, contrast, gamma, exposure, hue, saturation)
• Filtering operations (box, median, gaussian, sobel filters)
• Image transformations (resize, crop, flip, rotate)
• Color space conversions
• Advanced operations (warp affine, LUT, histogram equalization)

Each operation is run for a configurable number of iterations (default: 100) on a batch of 128 1080p images to provide statistically meaningful performance metrics.

Results are automatically exported to Excel files with a filename format:

• opencv_vs_rpp_benchmark_results_<N>threads.xlsx - Where <N> is the number of threads used

Examples:

• opencv_vs_rpp_benchmark_results_64threads.xlsx - Results using 64 threads
• opencv_vs_rpp_benchmark_results_32threads.xlsx - Results using 32 threads
• opencv_vs_rpp_benchmark_results_1threads.xlsx - Single-threaded results

Each file contains:

• Sheet 1: System information (RPP version, OpenCV version, CPU, memory, OS)
• Sheet 2: Grayscale image benchmarks (OpenCV vs RPP HOST comparison with speedup)
• Sheet 3: RGB image benchmarks (OpenCV vs RPP HOST comparison with speedup)

Directory Structure

OPENCV/
├── opencv_vs_rpp_host_benchmarking.cpp  # Main entry point and benchmark orchestration
├── opencv_benchmarks.cpp                # OpenCV benchmark implementations
├── rpp_benchmarks.cpp                   # RPP benchmark implementations
├── benchmarks_common.h                  # Shared declarations and data structures
├── benchmarks_utils.cpp                 # Utility functions (image loading, Excel export, system info)
├── dropout_helpers.cpp                  # Helper functions for dropout operations
├── CMakeLists.txt                       # CMake build configuration
├── generate_test_dataset.py             # Script to create synthetic test images
├── run_benchmarking.sh                  # One-command full setup and run
├── 1080p_128images_dataset/             # Test image directory (created by generate_test_dataset.py)
├── build_enabled_pthreads/              # Build output with parallel threads ON (created by run_benchmarking.sh)
└── build_disabled_pthreads/             # Build output with parallel threads OFF (created by run_benchmarking.sh)

Code Organization

The benchmark suite is organized into modular files for better maintainability:

Core Files:

• opencv_vs_rpp_host_benchmarking.cpp - Main program that orchestrates benchmark execution
• benchmarks_common.h - Shared header with data structures, constants, and function declarations

Implementation Files:

• opencv_benchmarks.cpp - All OpenCV benchmark function implementations (~70+ operations)
• rpp_benchmarks.cpp - All RPP benchmark function implementations (~70+ operations)
• benchmarks_utils.cpp - Common utilities (image loading, Excel export, system information)
• dropout_helpers.cpp - Helper functions for dropout augmentation operations

This modular design separates concerns and makes it easier to:

• Add new benchmark operations
• Maintain and debug existing implementations
• Understand the codebase structure
• Reuse utility functions

Prerequisites

System Requirements

• Linux system (tested on Ubuntu 20.04+)
• CMake 3.10 or higher
• GCC with C++17 support
• Python 3 with Pillow (PIL) library (for dataset generation)
• Sudo privileges (for installing OpenCV)
• Minimum 2GB RAM (4GB+ recommended for 128 images × 1080p dataset)
  • Grayscale dataset: ~250MB RAM
  • RGB dataset: ~700MB RAM
  • Total with both: ~1GB for images + overhead

Required Libraries

• OpenCV 4.x with components: core, imgproc, imgcodecs
• OpenMP for parallel processing
• RPP (ROCm Performance Primitives) - Automatically detected via CMake's find_package(rpp)
• ROCm/HIP - The RPP library requires HIP headers (platform is automatically defined in CMake)
• libxlsxwriter - For generating Excel output files

Environment Setup

The build system uses the following priority for locating ROCm/RPP:

1. ROCM_PATH environment variable (if set)
2. CMake cache variable ROCM_PATH (if set)
3. Default path: /opt/rocm

To use a custom ROCm installation:

export ROCM_PATH=/path/to/your/rocm
# or
cmake -DROCM_PATH=/path/to/your/rocm ..

Quick Start

Option 1: Complete Automated Setup and Run

The easiest way to get started is to use the complete setup script:

cd utilities/test_suite/OPENCV
./run_benchmarking.sh

This script will:

1. Generate synthetic test dataset (if not present)
2. Install OpenCV and dependencies (if needed)
3. Build the benchmark executable
4. Run the complete benchmark suite

Note: The script will prompt for sudo password if dependencies need to be installed.

Option 2: Step-by-Step Manual Setup

Step 1: Generate Test Dataset

Create 128 synthetic 1080p test images:

python3 generate_test_dataset.py

This generates various image patterns (gradients, checkerboards, circles, noise, stripes, etc.) in the 1080p_128images_dataset/ directory.

Step 2: Install Dependencies

Install OpenCV and build tools:

sudo apt-get update
sudo apt-get install -y libopencv-dev libgomp1 cmake build-essential libxlsxwriter-dev
pip3 install Pillow  # Required for generate_test_dataset.py

Verify OpenCV installation:

pkg-config --modversion opencv4  # or 'opencv' for older versions

Step 3: Build the Benchmark

mkdir -p build
cd build
cmake ..
make -j$(nproc)
cd ..

The executable will be created at build/opencv_vs_rpp_host_benchmarking

Step 4: Run the Benchmark

Basic usage (auto-detect threads):

./build/opencv_vs_rpp_host_benchmarking

Command-line options:

./build/opencv_vs_rpp_host_benchmarking [OPTIONS]

Options:
  -t, --threads <N>        Number of threads to use (default: auto-detect)
  -n, --num-runs <N>       Number of benchmark runs (default: 100)
  -g, --gray-path <PATH>   Path to grayscale images (default: 1080p_128images_dataset/)
  -r, --rgb-path <PATH>    Path to RGB images (default: 1080p_128images_dataset/)
  -h, --help               Display help message

Examples:
  ./build/opencv_vs_rpp_host_benchmarking                           # Auto-detect threads, 100 runs (default)
  ./build/opencv_vs_rpp_host_benchmarking --threads 64              # Use 64 threads
  ./build/opencv_vs_rpp_host_benchmarking -t 32 -n 50               # Use 32 threads with 50 runs
  ./build/opencv_vs_rpp_host_benchmarking -t 32 -g ./my_images/     # Use 32 threads with custom dataset

Thread Configuration:

• By default, the benchmark auto-detects the maximum available threads using OpenMP
• You can override this with the -t or --threads option
• Thread count only applies when built with ENABLE_PARALLEL_THREADS=ON
• When built with ENABLE_PARALLEL_THREADS=OFF, always uses 1 thread regardless of the argument

Understanding the Output

Console Output

The benchmark produces console output in the following format:

================================================
OpenCV vs RPP Benchmarking Script
================================================

System Information:
-------------------
CPU: AMD EPYC 7742 64-Core Processor
Kernel: Linux 6.17.0-22-generic
RPP Version: x.x.x
OpenCV Version: 4.x.x
Threads: 128
Batch Size: 128 images
Image Resolution: 1920x1080

Running Benchmarks (100 iterations each):
==========================================

benchmark_OpenCV_Brightness (Avg per run, 128 images, RGB): 45.2 ms
benchmark_RPP_Brightness    (Avg per run, 128 images, RGB): 32.1 ms

benchmark_OpenCV_GammaCorrection (Avg per run, 128 images, RGB): 52.3 ms
benchmark_RPP_GammaCorrection    (Avg per run, 128 images, RGB): 38.7 ms

...

Each operation shows:

• Operation name (OpenCV vs RPP)
• Number of images processed
• Color mode (RGB or Grayscale)
• Average execution time per run in milliseconds

Excel Output File

Results are automatically saved to an Excel file with three sheets. The filename indicates the number of threads used:

• opencv_vs_rpp_benchmark_results_<N>threads.xlsx - Where <N> is the thread count

For example:

• opencv_vs_rpp_benchmark_results_64threads.xlsx - Results using 64 threads
• opencv_vs_rpp_benchmark_results_1threads.xlsx - Single-threaded baseline

Sheet 1: System Information

Parameter	Value
RPP Version	3.1.2
OpenCV Version	4.6.0
Operating System	Linux 6.17.0-22-generic
CPU	AMD EPYC 7742 64-Core Processor
Memory	256.00 GB
Number of Threads	128
Number of Runs	100

Sheet 2: Grayscale Benchmarks

Operation	Parameters	OpenCV (ms)	RPP HOST (ms)	Speedup
Brightness	alpha=1.2, beta=20.0	45.20	32.10	1.41x
BoxFilter	kernel=3	26.02	26.04	1.00x
GammaCorrection	gamma=1.5	52.30	38.70	1.35x
...	...	...	...	...

Sheet 3: RGB Benchmarks

Operation	Parameters	OpenCV (ms)	RPP HOST (ms)	Speedup
Brightness	alpha=1.2, beta=20.0	135.60	96.30	1.41x
Hue	hueDelta=10.0	78.90	52.40	1.51x
BoxFilter	kernel=3	79.60	79.60	1.00x
...	...	...	...	...

The Excel file can be opened in Excel, LibreOffice Calc, or Google Sheets for further analysis and visualization.

Customizing the Benchmark

Enabling/Disabling Parallel Threading

The benchmark supports toggling parallel threading at build time using the ENABLE_PARALLEL_THREADS CMake option. This allows you to compare performance with and without OpenMP parallelization.

Build with parallel threads ENABLED (default):

cd build
cmake ..
# or explicitly:
cmake -DENABLE_PARALLEL_THREADS=ON ..
make -j$(nproc)

Build with parallel threads DISABLED:

cd build
cmake -DENABLE_PARALLEL_THREADS=OFF ..
make -j$(nproc)

When parallel threads are disabled, all OpenMP #pragma omp parallel for directives are excluded from compilation, allowing you to measure single-threaded performance. The output Excel filename will automatically indicate which mode was used.

Configuring Thread Count

The benchmark automatically detects the maximum available threads on your system. You can override this at runtime:

Runtime configuration (recommended):

# Use all available threads (auto-detect) with default 100 runs
./build/opencv_vs_rpp_host_benchmarking

# Use specific thread count
./build/opencv_vs_rpp_host_benchmarking --threads 64
./build/opencv_vs_rpp_host_benchmarking -t 32

# Use specific thread count with custom number of runs
./build/opencv_vs_rpp_host_benchmarking -t 32 -n 50

Note: Thread count configuration only applies when built with ENABLE_PARALLEL_THREADS=ON. When built with ENABLE_PARALLEL_THREADS=OFF, the benchmark always uses 1 thread regardless of command-line arguments.

Configuring Number of Benchmark Runs

You can configure the number of benchmark runs (iterations) at runtime using the -n or --num-runs option:

Runtime configuration (recommended):

# Default 100 runs
./build/opencv_vs_rpp_host_benchmarking

# Custom number of runs
./build/opencv_vs_rpp_host_benchmarking --num-runs 50
./build/opencv_vs_rpp_host_benchmarking -n 200

# Combine with thread count
./build/opencv_vs_rpp_host_benchmarking -t 32 -n 50

Alternative: Build-time configuration: You can also set a default value by editing benchmarks_common.h:

extern int NUM_RUNS;  // Default is 100, but can be overridden via command-line

The runtime command-line option will always override the default value.

Changing RPP Installation Path

The CMakeLists.txt automatically finds RPP using CMake's find_package(rpp). If RPP is installed in a non-standard location, you can:

Option 1: Set ROCM_PATH

export ROCM_PATH=/path/to/rocm
cmake ..

Option 2: Set RPP_DIR

cmake -DRPP_DIR=/path/to/rpp/lib/cmake/rpp ..

Option 3: Add to CMAKE_PREFIX_PATH

cmake -DCMAKE_PREFIX_PATH=/path/to/rpp ..

The build system will automatically locate the RPP headers and libraries.

Using Custom Image Dataset

Option 1: Command-line arguments (recommended):

# Use custom dataset path for both grayscale and RGB
./build/opencv_vs_rpp_host_benchmarking --gray-path ./my_images/ --rgb-path ./my_images/

# Use different paths for grayscale and RGB
./build/opencv_vs_rpp_host_benchmarking -g ./gray_imgs/ -r ./rgb_imgs/

Option 2: Replace default dataset: Replace the synthetic images in 1080p_128images_dataset/ with your own images.

Supported image formats:

• JPEG (.jpg, .jpeg)
• PNG (.png)
• BMP (.bmp)
• TIFF (.tiff)

The benchmark will automatically detect and process all valid images in the specified directory.

Troubleshooting

OpenCV Not Found

# Check if OpenCV is installed
pkg-config --modversion opencv4

# If not found, install it
sudo apt-get install libopencv-dev

RPP Library Not Found

If CMake cannot find RPP, you'll see an error like:

-- opencv_benchmarking_script requires RPP. Install RPP or set RPP_DIR to the RPP installation path

Solution 1: Set ROCM_PATH environment variable

export ROCM_PATH=/path/to/rocm

Solution 2: Set RPP_DIR when running cmake

cmake -DRPP_DIR=/path/to/rpp/lib/cmake/rpp ..

Verify RPP installation:

# Check if RPP is installed
ls -la $ROCM_PATH/lib/librpp.so
ls -la $ROCM_PATH/include/rpp/

# Or check the default location
ls -la /opt/rocm/lib/librpp.so
ls -la /opt/rocm/include/rpp/

libxlsxwriter Not Found

If CMake reports that libxlsxwriter is not found:

-- opencv_benchmarking_script requires libxlsxwriter for Excel output

Solution:

sudo apt-get install libxlsxwriter-dev

Then rebuild:

rm -rf build && mkdir build && cd build && cmake .. && make

Build Errors - HIP Platform Not Defined

If you see errors like:

error: #error ("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__");

This means the HIP platform macro is not defined. The CMakeLists.txt automatically defines __HIP_PLATFORM_AMD__ for AMD ROCm. If you still see this error, verify you have the latest CMakeLists.txt and rebuild:

rm -rf build
mkdir build
cd build
cmake ..
make -j$(nproc)

Other Build Errors

For general build issues, try a clean rebuild:

rm -rf build
mkdir build
cd build
cmake ..
make -j$(nproc)

Permission Denied

If you get permission errors when running scripts:

chmod +x run_benchmarking.sh
chmod +x generate_test_dataset.py

No Images Found Error

Make sure the test dataset exists:

ls -la 1080p_128images_dataset/

# If empty or missing, generate it:
python3 generate_test_dataset.py

Performance Notes

• Thread Configuration:

  • The benchmark auto-detects the maximum available threads by default
  • Override with --threads N command-line argument
  • Thread count only applies when built with ENABLE_PARALLEL_THREADS=ON
  • Uses OpenMP for parallel processing (can be disabled at build time with -DENABLE_PARALLEL_THREADS=OFF)

• Benchmark Iterations:

  • Each operation runs 100 times by default for stable average timings
  • Adjustable at runtime via -n or --num-runs command-line option (no rebuild required)
  • Example: ./build/opencv_vs_rpp_host_benchmarking -n 200 runs each benchmark 200 times

• Performance Variability:

  • Results vary based on CPU architecture, memory bandwidth, and system load
  • Close other resource-intensive applications for accurate comparisons
  • Test with different thread counts to find optimal performance for your system

• Parallel vs Single-threaded Comparison:

  • Build with ENABLE_PARALLEL_THREADS=ON and run with different --threads values
  • Build with ENABLE_PARALLEL_THREADS=OFF for single-threaded baseline
  • Compare the resulting Excel files to analyze scaling efficiency

Tested Operations

The benchmark includes the following operation categories:

Color Adjustments:

• Brightness, Contrast, Gamma Correction, Exposure
• Hue, Saturation, Color Jitter
• Color Temperature, Vignette

Filters:

• Box Filter, Median Filter, Gaussian Filter
• Sobel Filter, Non-Max Suppression
• Erode, Dilate

Transformations:

• Resize (Nearest Neighbor, Bilinear, Bicubic, Lanczos)
• Crop, Flip (Horizontal, Vertical, Both)
• Rotate (90°, 180°, 270°)
• Warp Affine

Color Conversions:

• RGB to Grayscale, HSV, BGR
• LUT (Lookup Table)
• Histogram Equalization
• Channel Extract

License

This benchmarking tool is part of the RPP (ROCm Performance Primitives) project.

Support

For issues related to:

• RPP library: Refer to the main RPP repository
• OpenCV: Consult OpenCV documentation at https://docs.opencv.org/
• This benchmark: Open an issue in the RPP repository