TheRock/rocm-libraries/projects/hipblaslt/clients/bench

hipblaslt-bench test

# Go to hipBLASLt build directory
cd hipBLASLt; cd build/release

# run hipblaslt-bench
./clients/hipblaslt-bench --help

./clients/hipblaslt-bench [ --data <path> | --yaml <path> ] <options> ...

--sizem |-m <value>        Specific matrix size: the number of rows or columns in matrix.                      (Default value is: 128)
--sizen |-n <value>        Specific matrix the number of rows or columns in matrix                             (Default value is: 128)
--sizek |-k <value>        Specific matrix size: the number of columns in A and rows in B.                     (Default value is: 128)
--lda <value>              Leading dimension of matrix A.
--ldb <value>              Leading dimension of matrix B.
--ldc <value>              Leading dimension of matrix C.
--ldd <value>              Leading dimension of matrix D.
--lde <value>              Leading dimension of matrix E.
--any_stride               Do not modify input strides based on leading dimensions
--stride_a <value>         Specific stride of strided_batched matrix A, second dimension * leading dimension.
--stride_b <value>         Specific stride of strided_batched matrix B, second dimension * leading dimension.
--stride_c <value>         Specific stride of strided_batched matrix C, second dimension * leading dimension.
--stride_d <value>         Specific stride of strided_batched matrix D, second dimension * leading dimension.
--stride_e <value>         Specific stride of strided_batched matrix E, second dimension * leading dimension.
--alpha <value>            specifies the scalar alpha                                                           (Default value is: 1)
--beta <value>             specifies the scalar beta                                                            (Default value is: 0)
--function |-f <value>     BLASLt function to test. Options: matmul                                             (Default value is: matmul)
--precision |-r <value>    Precision of matrix A,B,C,D Options: f32_r, f16_r, bf16_r, f64_r, i32_r, i8_r, f32_c, f64_c (Default value is: f16_r)
--a_type <value>           Precision of matrix A. Options: f32_r, f64_r, i32_r, f16_r, bf16_r, i8_r, f32_c, f64_c 
--b_type <value>           Precision of matrix B. Options: f32_r, f64_r, i32_r, f16_r, bf16_r, i8_r, f32_c, f64_c 
--c_type <value>           Precision of matrix C. Options: f32_r, f64_r, i32_r, f16_r, bf16_r, i8_r, f32_c, f64_c 
--d_type <value>           Precision of matrix D. Options: f32_r, f64_r, i32_r, f16_r, bf16_r, i8_r, f32_c, f64_c 
--compute_type <value>     Precision of computation. Options: s,f32_r,x,xf32_r,f64_r,i32_r          (Default value is: f32_r)
--compute_input_typeA <value>     Options: f32_r, f16_r, bf16_r, f8_r, bf8_r, f8_fnuz_r, bf8_fnuz_r, The default value indicates that the argument has no effect. (Default value is: INVALID)
--compute_input_typeB <value>     Options: f32_r, f16_r, bf16_r, f8_r, bf8_r, f8_fnuz_r, bf8_fnuz_r, The default value indicates that the argument has no effect. (Default value is: INVALID)
--scale_type <value>       Precision of scalar. Options: f16_r,bf16_r, f32_c, f64_c
--initialization <value>   Initialize matrix data.Options: rand_int, trig_float, hpl(floating), special, zero, norm_dist, uniform_01   (Default value is: hpl)
--transA <value>           N = no transpose, T = transpose, C = conjugate                                       (Default value is: N)
--transB <value>           N = no transpose, T = transpose, C = conjugate                                       (Default value is: N)
--batch_count <value>      Number of matrices. Only applicable to batched and strided_batched routines          (Default value is: 1)
--HMM                      Parameter requesting the use of HipManagedMemory
--verify |-v               Validate GPU results with CPU?
--iters |-i <value>        Iterations to run inside timing loop                                                 (Default value is: 10)
--cold_iters |-j <value>   Cold Iterations to run before entering the timing loop                               (Default value is: 2)
--adaptive                       Enable adaptive timing; not compatible with --iters / --cold_iters
--adaptive_warmup_time <value>   Warm up (ms) until this much wall-time has elapsed; requires --adaptive
--adaptive_sample_time <value>   Wall-time (ms) of each timed sample = back-to-back batch size (>= measure_time gives one batch); requires --adaptive
--adaptive_measure_time <value>  Minimum total measurement time (ms); requires --adaptive
--adaptive_max_measure_time <value> Measurement ceiling (ms, 0 = unbounded); requires --adaptive
--adaptive_min_iters <value>     Floor on total timed iterations; requires --adaptive
--adaptive_max_iters <value>     Ceiling on total timed iterations (0 = unbounded); requires --adaptive
--adaptive_noise_threshold <value> Convergence target: past the floor, keep running until the mean's relative standard error drops below this fraction (converged), or the robust spread plateaus (stable), or the ceiling is hit (noisy); 0 disables both (run to the ceiling); requires --adaptive
--adaptive_stability_threshold <value> Noise-plateau fallback: stop with status=stable once the recent rel_iqr readings vary by less than this fraction; 0 disables the fallback (a non-converging run then goes to the ceiling, status=noisy); requires --adaptive
--adaptive_stability_window <value> Number of recent rel_iqr readings the stability fallback tests for a plateau (>= 2); requires --adaptive
--adaptive_stability_interval <value> Record one rel_iqr reading for the stability fallback every N samples (>= 1); requires --adaptive
(adaptive defaults are shown by --help / -h)
--algo_method <value>      Use different algorithm search API. Options: heuristic, all, index.                  (Default value is: heuristic)
--solution_index <value>   Used with --algo_method 2.  Specify solution index to use in benchmark.              (Default value is: -1)
--requested_solution <value> Requested solution num. Set to -1 to get all solutions. Only valid when algo_method is set to heuristic.  (Default value is: 1)
--activation_type <value>  Options: None, gelu, relu, swish, clamp                                              (Default value is: none)
--activation_arg1 <value>  First extra argument for activation function if needed.                              (Default value is: 0)
--activation_arg2 <value>  Second extra argument for activation function if neeeded.                            (Default value is: inf)
--bias_type <value>        Precision of bias vector.Options: f16_r,bf16_r,f32_r,default(same with D type)
--bias_source <value>      Choose bias source: a, b, d                                                          (Default value is: d)
--bias_vector              Apply bias vector
--scaleA                   Apply scale for A buffer
--scaleB                   Apply scale for B buffer
--scaleAlpha_vector        Apply scaleAlpha vector
--amaxScaleA               Apple scale for A buffer by abs max of A buffer
--amaxScaleB               Apple scale for B buffer by abs max of B buffer
--use_e                    Apply AUX output/ gradient input
--gradient                 Enable gradient
--grouped_gemm             Use grouped_gemm.
--use_user_args            Use UserArguments located in device memory for grouped gemm.
--device <value>           Set default device to be used for subsequent program runs                            (Default value is: 0)
--c_equal_d                C and D are stored in same memory
--workspace <value>        Set fixed workspace memory size instead of using hipblaslt managed memory            (Default value is: 0)
--log_function_name        Function name precedes other items.
--function_filter <value>  Simple strstr filter on function name only without wildcards
--api_method <value>       Use extension API. c: C style API. mix: declaration with C hipblasLtMatmul Layout/Desc but set, initialize, and run the problem with C++ extension API. cpp: Using C++ extension API only. Options: c, mix, cpp.  (Default value is: c)
--print_kernel_info        Print solution, kernel name and solution index.
--rotating <value>         Use rotating memory blocks for each iteration, size in MB.                           (Default value is: 0)
--use_gpu_timer            Use hipEventElapsedTime to profile elapsed time.                                    (Default value is: false)
--splitk <value>           [Tuning parameter] Set split K for a solution, 0 is use solution's default value. (Only support GEMM + api_method mix or cpp)
--wgm <value>              [Tuning parameter] Set workgroup mapping for a solution, 0 is use solution's default value. (Only support GEMM + api_method mix or cpp)
--flush                    Flush icache
--sm_count_target <value>  Target compute-unit (CU) count for the matmul kernel selection and persistent-grid sizing. 0 (default) means use all CUs the device exposes. Negative values are rejected. (Default value is: 0)
--streamk_tile_scheduling <value>  Select the StreamK=5 tile scheduling sub-path via the HIPBLASLT_MATMUL_DESC_STREAMK_TILE_SCHEDULING_EXT extension attribute. Accepts off|0 (SK3 static; default when unset), on|1 (force SK4 dynamic), auto|2 (always run origami heuristic), case-insensitive. Omit to leave the attribute unset so the library default (off) applies; combine with `--sm_count_target` > 0 to engage the heuristic without setting auto|2.  (Default value is: unset)
--help |-h                 produces this help message
--version <value>          Prints the version number

demo

Run fp32 GEMM with validation

./clients/hipblaslt-bench --precision f32_r -v
[0]:transA,transB,grouped_gemm,batch_count,m,n,k,alpha,lda,stride_a,beta,ldb,stride_b,ldc,stride_c,ldd,stride_d,a_type,b_type,c_type,d_type,compute_type,scaleA,scaleB,scaleC,scaleD,amaxD,activation_type,bias_vector,bias_type,hipblaslt-Gflops,hipblaslt-GB/s,us,CPU-Gflops,CPU-us,norm_error,atol,rtol
    N,N,0,1,128,128,128,1,128,16384,0,128,16384,128,16384,128,16384,f32_r,f32_r,f32_r,f32_r,f32_r,0,0,0,0,0,none,0,non-supported type,403.298,17.6063,10.4,0.0960059,43688,2.44879e-07,1e-05,1e-05

Show the frequency with environment variable

HIPBLASLT_BENCH_FREQ=1 ./clients/hipblaslt-bench -m 16 -n 16 -k 4096 --transA T --transB N --a_type bf16_r --b_type bf16_r --c_type bf16_r --d_type bf16_r --activation_type none --compute_type f32_r
[0]:transA,transB,grouped_gemm,batch_count,m,n,k,alpha,lda,stride_a,beta,ldb,stride_b,ldc,stride_c,ldd,stride_d,a_type,b_type,c_type,d_type,compute_type,scaleA,scaleB,scaleC,scaleD,amaxD,swizzle_a,swizzle_b,activation_type,bias_vector,bias_type,aux_type,rotating_buffer,flush,use_gpu_timer,lowest_avg_freq,lowest_median_freq,avg_MCLK,median_MCLK,hipblaslt-Gflops,hipblaslt-GB/s,us
    T,N,0,1,16,16,4096,1,4096,65536,0,4096,65536,16,256,16,256,bf16_r,bf16_r,bf16_r,bf16_r,f32_r,0,0,0,0,0,0,0,none,0,bf16_r,bf16_r,0,0,0,191,191,2000,2000,275.941,32.1865,7.6

Show the multi-XCD frequencies with environment variable

HIPBLASLT_BENCH_FREQ_ALL=1 ./clients/hipblaslt-bench -m 16 -n 16 -k 4096 --transA T --transB N --a_type bf16_r --b_type bf16_r --c_type bf16_r --d_type bf16_r --activation_type none --compute_type f32_r
[0]:transA,transB,grouped_gemm,batch_count,m,n,k,alpha,lda,stride_a,beta,ldb,stride_b,ldc,stride_c,ldd,stride_d,a_type,b_type,c_type,d_type,compute_type,scaleA,scaleB,scaleC,scaleD,amaxD,swizzle_a,swizzle_b,activation_type,bias_vector,bias_type,aux_type,rotating_buffer,flush,use_gpu_timer,avg_freq0,avg_freq1,avg_freq2,avg_freq3,avg_freq4,avg_freq5,avg_freq6,avg_freq7,median_freq0,median_freq1,median_freq2,median_freq3,median_freq4,median_freq5,median_freq6,median_freq7,avg_MCLK,median_MCLK,hipblaslt-Gflops,hipblaslt-GB/s,us
    T,N,0,1,16,16,4096,1,4096,65536,0,4096,65536,16,256,16,256,bf16_r,bf16_r,bf16_r,bf16_r,f32_r,0,0,0,0,0,0,0,none,0,bf16_r,bf16_r,0,0,0,223,205,221,217,209,211,191,191,223,205,221,217,209,211,191,191,2000,2000,249.661,29.1211,8.4

Show the efficiency and other performance related args with environment variable

HIPBLASLT_BENCH_PERF=1 ./clients/hipblaslt-bench -m 4096 -n 4864 -k 32896 --transA N --transB N --a_type bf16_r --b_type bf16_r --c_type bf16_r --d_type bf16_r --compute_type f32_r --iters 416 --cold_iters 416 --use_gpu_timer
[0]:transA,transB,grouped_gemm,batch_count,m,n,k,alpha,lda,stride_a,beta,ldb,stride_b,ldc,stride_c,ldd,stride_d,a_type,b_type,c_type,d_type,compute_type,scaleA,scaleB,scaleC,scaleD,amaxD,swizzle_a,swizzle_b,activation_type,bias_vector,bias_type,aux_type,rotating_buffer,flush,use_gpu_timer,num_cu,tiles_per_cu,tile0_gran,tile1_gran,cu_gran,wave_gran,total_gran,mem_read_bytes,mem_write_bytes,lowest_avg_freq,lowest_median_freq,avg_MCLK,median_MCLK,efficiency,hipblaslt-Gflops,hipblaslt-GB/s,us
    N,N,0,1,4096,4864,32896,1,4096,134742016,0,32896,160006144,4096,19922944,4096,19922944,bf16_r,bf16_r,bf16_r,bf16_r,f32_r,0,0,0,0,0,0,0,none,0,bf16_r,bf16_r,0,0,1,256,0,0.984615,1,-nan,1,-nan,13312511180,119537664,1119,1205,2000,2000,92.3889,1.08405e+06,484.741,1209.14

Adaptive timing

By default the benchmark runs --cold_iters warmups followed by a single timed loop of --iters enqueues and reports the mean. Pass --adaptive to instead take a distribution-based measurement that is robust to noise and lets fast kernels accumulate more iterations automatically — without precomputing iteration counts per problem. The fixed-count default is unchanged, so existing commands behave exactly as before.

A sample is a batch of back-to-back iterations run with no synchronization between them, timed only at its start and end (so the sample's per-iteration time is that span divided by the batch size). The batch is sized from a quick warmup so each sample spans roughly --adaptive_sample_time ms.

The run lasts at least the floor — --adaptive_min_iters and --adaptive_measure_time — and at most the ceiling — --adaptive_max_measure_time and --adaptive_max_iters. Past the floor, it stops on the first of: the mean's relative standard error (stddev / mean / sqrt(n)) falling below --adaptive_noise_threshold (converged, status=converged); or, when that target cannot be met, the robust spread (IQR / median) flattening out so more samples will not change the result (stable, status=stable) — a fallback for heavy-tailed or unlocked-clock kernels; otherwise the ceiling (status=noisy). The stable fallback is tuned by --adaptive_stability_window / --adaptive_stability_interval and can be turned off with --adaptive_stability_threshold 0 (a non-converging run then runs to the ceiling and reports noisy). With --adaptive_noise_threshold 0 both the convergence check and the fallback are disabled, so the run goes to the ceiling and status is -. A ceiling is required — at least one of --adaptive_max_measure_time / --adaptive_max_iters must be > 0 (it is by default), otherwise the run is rejected — so a non-converging run is always bounded. (To run a fixed budget, set the floor and ceiling equal, e.g. --adaptive_measure_time 200 --adaptive_max_measure_time 200.)

The reported us/Gflops are the median; extra columns expose the distribution: batch, samples, hot_iters, mean_us, min_us, cv (stddev/mean), rel_iqr (interquartile range / median — a drift-robust dispersion measure), and status (converged / stable / noisy / -). Convergence requires at least 10 samples so the stddev is trustworthy.

--adaptive is the gate; the --adaptive_* options tune it and require --adaptive (passing one without it is an error). Run --help for their current default values.

Under --adaptive, warmup and per-sample batch size are determined adaptively, so --iters and --cold_iters have no effect and passing either with --adaptive is an error (use --adaptive_warmup_time for warmup and --adaptive_sample_time to influence the batch). Without --adaptive, timing is the fixed-count --cold_iters / --iters path and no distribution columns are printed.

Note on the statistic: the median is reported as the headline because it is robust to the occasional slow sample (OS/power jitter, clock dips) that inflates the mean on an unlocked GPU. The mean is kept as a column for comparison and matches the fixed-count number exactly (where median, mean and min coincide).

./clients/hipblaslt-bench -m 4096 -n 4096 -k 4096 --transA N --transB T \
  --a_type bf16_r --b_type bf16_r --c_type bf16_r --d_type bf16_r --compute_type f32_r \
  --rotating 512 --adaptive