TheRock/rocm-systems/projects/rocprofiler-sdk/CHANGELOG.md

Changelog for ROCprofiler-SDK

Full documentation for ROCprofiler-SDK is available at rocm.docs.amd.com/projects/rocprofiler-sdk

Unreleased

Added

API:

• Streaming Performance Monitor (SPM) counter collection support (beta):
  • New experimental API in rocprofiler-sdk/experimental/spm.h:
  • GPU-timestamped counter values alongside kernel dispatch information.
• Added spm_support along with reserved padding to rocprofiler_counter_info_v1_t

rocprofv3 (CLI):

• SPM counter collection support in rocprofv3 (beta):
  • --spm <counter> flag to specify counters for SPM collection.
  • --spm-sample-interval and --spm-sample-interval-unit parameters to configure sampling rate.
  • --spm-beta-enabled flag to opt in to the beta SPM feature.
  • --spm-config option in rocprofv3-avail to list available SPM configurations.
• JSON and rocpd output format support for SPM.
• OpenMP (OMPT) tracing via the new --ompt-trace flag:
  • Accepts a bare boolean or category list (all, thread, parallel, task, sync, mutex, target, device, error); also folded into --sys-trace/--runtime-trace.
  • rocpd-only trace: records go to the rocpd database (auto-added when another format is requested) and export via rocpd convert.

Documentation:

• SPM API reference guide (api-reference/spm.rst).
• SPM usage guide for rocprofv3 (how-to/using-spm.rst).
• --spm-config documentation to rocprofv3-avail usage guide.

Changed

• Bump rocpd schema to version 3.0.1 which supports NIC agent types.
• Bump rocpd schema to version 3.0.2 for HIP graph per-node attribution (graph_exec_id/graph_node_id columns on rocpd_kernel_dispatch/rocpd_memory_copy and the new rocpd_graph_launch table). The pre-graph-attribution 3.0.1 schema is now frozen under versions/3.0.1/ per the rocpd schema versioning scheme.
• Bump rocpd schema to version 3.0.3 for SPM support. The pre-spm-support 3.0.2 schema is now frozen under versions/3.0.2/ per the rocpd schema versioning scheme.

Removed

ROCprofiler-SDK 1.3.5

Added

• HipFile API tracing support

ROCprofiler-SDK 1.3.0 for ROCm release 7.2.4

Optimized

• Reduced ROCprofiler-sdk profiling overhead: Improved profiling stability for vLLM workloads traced with PyTorch torch.profiler using the rocprofiler-sdk backend.

ROCprofiler-SDK 1.3.0 for ROCm release 7.13

Added

API:

• Late-start profiling support: Added automatic late‑start profiling support, enabling profiling when rocprofiler-sdk is loaded after HSA/HIP runtimes have already initialized.

  • rocprofiler_force_configure() now automatically detects and profiles runtimes initialized before the SDK loads.
  • Integrates with rocprofiler-register to retrieve the registered API tables.
  • Supports all runtime types (HSA, HIP, ROCTX, RCCL, ROCDecode, ROCJpeg, and more) automatically.
  • No explicit late-start API calls required; works transparently.

• KFD (Kernel Fusion Driver) event tracing support:

  • Buffer service configurations for each KFD buffer tracing type.
  • New type tool_buffer_tracing_kfd_record_t using std::variant to wrap 8 different KFD buffer tracing types.
  • Each KFD event generates rocpd_info_pmc, rocpd_event, rocpd_region, and rocpd_pmc_event rows
  • Fixed handling for special SVM location in KFD prefetch location reporting
  • Fixed parsing for queue restore events to handle both correct format (character '0') and broken driver format (NULL character '\0')

• Per-graph-node attribution for HIP graph operations:

  • New callback tracing kind ROCPROFILER_CALLBACK_TRACING_HIP_GRAPH fires for hipGraphInstantiate*, hipGraphExecDestroy, and hipGraphLaunch{,_spt} lifecycle events. Payload carries a process-monotonic graph_exec_id and the raw hipGraphExec_t handle. Mirrors the HIP_STREAM design.
  • New buffer tracing kind ROCPROFILER_BUFFER_TRACING_HIP_GRAPH emits a summary record per successful hipGraphLaunch invocation with graph_exec_id, kernel_dispatch_count, agent, and launch timestamps.
  • Tools can associate kernel dispatches and memory copies with their producing graph node by maintaining a per-thread attribution stack (push on EXEC_LAUNCH ENTER, pop on EXIT) and capturing the top of stack into the external correlation id at request time. See the doc comment on rocprofiler_callback_tracing_hip_graph_data_t for the recipe.
  • Attribution accuracy across launches requires segmented scheduling (default), AMD_DIRECT_DISPATCH=1, and single-threaded launching of one hipGraphExec_t.

rocprofv3 (CLI):

• Multi-pass counter collection support: Support for multiple --pmc flags to define separate counter groups for different profiling passes.

  • Ability to combine command-line --pmc flags with input file counter groups.
  • Each pass generates output in a separate pass_n subdirectory.
  • Example: rocprofv3 --pmc SQ_WAVES --pmc GRBM_COUNT -- <app> creates two profiling passes.

• KFD (Kernel Fusion Driver) event tracing support:

  • KFD record dumping to rocpd with support for 8 main KFD event types.
  • Support for rocpd to Perfetto conversion for KFD events.
  • --kfd-trace flag to enable KFD event tracing.

• ROCTx Support for ATT: Added ROCtx support to device thread trace when using --att --selected-regions.

  • Allows roctxProfilerPause and roctxProfilerResume to explicitly control when ATT data collection starts and stops.
  • Enables more precise, region-focused ATT tracing with reduced overhead and noise.
  • Supports multiple resume/pause cycles, each producing separate trace output files.
  • Incompatible with --att-consecutive-kernels.

• PC Sampling Support for Dynamic Attach: Added PC Sampling support for Dynamic Attach feature, allowing users to attach to a running application and collect PC samples without restarting the workload.

  • Enables profiling long-running or production-style jobs at the point of interest.
  • Results integrate with the existing PC sampling analysis flow.

• HIP graph attribution fields and trace:

  • Kernel and memory-copy records gain graph_exec_id and graph_node_id fields in JSON and rocpd output. rocpd conversion exposes these fields in CSV, OTF2, and Perfetto output.
  • New --hip-graph-trace CLI flag emits per-launch summary records (one row per successful hipGraphLaunch) in JSON and rocpd output, including graph_exec_id and kernel_dispatch_count. Automatically enabled by --hip-trace / --hip-runtime-trace since HIP graphs are part of the HIP runtime.

• rocSHMEM API tracing support:

  • --rocshmem-trace flag to enable tracing of rocSHMEM host-stream APIs.
  • Included in the --sys-trace and --runtime-trace aggregate tracing options.
  • Emitted directly to the JSON and rocpd (default, .db) output formats; CSV, Perfetto (.pftrace), and OTF2 output are produced from the rocpd database via rocpd convert.

Documentation:

• Added marker-controlled thread tracing section to the thread trace how-to guide.
• Added cross-reference from ROCTx documentation to ATT with selected-regions.
• Added HIP graph attribution section to the rocprofv3 how-to guide covering the new output columns, the --hip-graph-trace flag, the determinism contract, and v1 limitations.

Changed

Implementation:

• Late-start architecture redesign: Removed direct runtime symbol access in favor of proper rocprofiler-register integration.
  • Replaced ~600 lines of dlopen/dlsym bypass logic with ~80 lines by using rocprofiler_register_invoke_all_registrations().
  • Late-start now works by requesting rocprofiler-register to repropagate stored API tables.
  • Extensible design. Automatically supports new runtimes without SDK code changes.
  • Provides a proper separation of concerns. rocprofiler-register manages the table storage while SDK manages the table wrapping.
• Counter dimension encoding changed from fixed-width to variable-width allocation per dimension type.
• Dimension selection and reduction logic now uses explicit dimension masks and single-index selection.
• HSA queue interception extended to handle AMD extended kernel dispatch packets.

Removed

• Counter collection support for plain text (.txt) input files has been deprecated due to lack of schema validation and input sanitization. Only structured file formats (JSON and YAML) with schema validation are supported.

Resolved issues

• Fixed rocpd OTF2 output to add ACCELERATOR_DEVICE as system tree node domain for AMD devices.
• Fixed rocprofv3 input file parsing where comment lines containing pmc: were incorrectly processed as valid counter collection directives, causing unintended profiling passes.

Internal APIs (non-public):

• Removed internal functions (were never in public headers):
  • rocprofiler_start_late_internal()
  • rocprofiler_is_late_start_internal()
  • rocprofiler_stop_late_internal()
• Replaced with single internal function: rocprofiler::late_start::invoke_register_propagation()

Note: Public API (rocprofiler_force_configure()) remains unchanged - no breaking changes for users

ROCprofiler-SDK for AFAR I

Added

• HSA API tracing
• Kernel dispatch tracing
• Kernel dispatch counter collection
  • Instances reported as single dimension
  • No serialization

ROCprofiler-SDK for AFAR II

Added

• HIP API tracing
• ROCTx tracing
• Tracing ROCProf Tool V3
• Documentation packaging
• ROCTx control (start and stop)
• Memory copy tracing

ROCprofiler-SDK for AFAR III

Added

• Kernel dispatch counter collection. This includes serialization and multidimensional instances.
• Kernel serialization.
• Serialization control (on and off).
• ROCprof tool plugin interface V3 for counters and dimensions.
• Support to list metrics.
• Correlation-Id retirement
• HIP and HSA trace distinction:
  • --hip-runtime-trace For collecting HIP Runtime API traces
  • --hip-compiler-trace For collecting HIP compiler-generated code traces
  • --hsa-core-trace For collecting HSA API traces (core API)
  • --hsa-amd-trace For collecting HSA API traces (AMD-extension API)
  • --hsa-image-trace For collecting HSA API traces (image-extension API)
  • --hsa-finalizer-trace For collecting HSA API traces (finalizer-extension API)

ROCprofiler-SDK for AFAR IV

Added

API:

• Page migration reporting
• Scratch memory reporting
• Kernel dispatch callback tracing
• External correlation Id request service
• Buffered counter collection record headers
• Option to remove HSA dependency from counter collection

Tool:

• rocprofv3 multi-GPU support in a single-process

ROCprofiler-SDK for AFAR V

Added

API:

• Agent or device counter collection
• PC sampling (beta)

Tool:

• Single JSON output format support
• Perfetto output format support (.pftrace)
• Input YAML support for counter collection
• Input JSON support for counter collection
• Application replay in counter collection
• rocprofv3 multi-GPU support:
  • Multiprocess (multiple files)

Changed

• rocprofv3 tool now requires mentioning -- before the application. For detailed use, see Using rocprofv3

Resolved issues

• Fixed SQ_ACCUM_PREV and SQ_ACCUM_PREV_HIRE overwriting issue

ROCprofiler-SDK 0.4.0 for ROCm release 6.2 (AFAR VI)

Added

• OTF2 tool support
• Kernel and range filtering
• Counter collection definitions in YAML
• Documentation updates (SQ block, counter collection, tracing, tool usage)
• rocprofv3 option --kernel-rename
• rocprofv3 options for Perfetto settings (buffer size and so on)
• CSV columns for kernel trace
  • Thread_Id
  • Dispatch_Id
• CSV column for counter collection

ROCprofiler-SDK 0.5.0 for ROCm release 6.3 (AFAR VII)

Added

• Start and end timestamp columns to the counter collection csv output
• Check to force tools to initialize context id with zero
• Support to specify hardware counters for collection using rocprofv3 as rocprofv3 --pmc [COUNTER [COUNTER ...]]
• Memory Allocation Tracing
• PC sampling tool support with CSV and JSON output formats
• List supported PC Sampling Configurations

Changed

• --marker-trace option for rocprofv3 now supports the legacy ROCTx library libroctx64.so when the application is linked against the new library librocprofiler-sdk-roctx.so.
• Replaced deprecated hipHostMalloc and hipHostFree functions with hipExtHostAlloc and hipFreeHost for ROCm versions starting 6.3.
• Updated rocprofv3 --help options.
• Changed naming of "agent profiling" to a more descriptive "device counting service". To convert existing tool or user code to the new name, use the following sed: find . -type f -exec sed -i 's/rocprofiler_agent_profile_callback_t/rocprofiler_device_counting_service_callback_t/g; s/rocprofiler_configure_agent_profile_counting_service/rocprofiler_configure_device_counting_service/g; s/agent_profile.h/device_counting_service.h/g; s/rocprofiler_sample_agent_profile_counting_service/rocprofiler_sample_device_counting_service/g' {} +
• Changed naming of "dispatch profiling service" to a more descriptive "dispatch counting service". To convert existing tool or user code to the new names, the following sed can be used: -type f -exec sed -i -e 's/dispatch_profile_counting_service/dispatch_counting_service/g' -e 's/dispatch_profile.h/dispatch_counting_service.h/g' -e 's/rocprofiler_profile_counting_dispatch_callback_t/rocprofiler_dispatch_counting_service_callback_t/g' -e 's/rocprofiler_profile_counting_dispatch_data_t/rocprofiler_dispatch_counting_service_data_t/g' -e 's/rocprofiler_profile_counting_dispatch_record_t/rocprofiler_dispatch_counting_service_record_t/g' {} +
• FETCH_SIZE metric on gfx94x now uses TCC_BUBBLE for 128B reads.
• PMC dispatch-based counter collection serialization is now per-device instead of being global across all devices.
• Added output return functionality to rocprofiler_sample_device_counting_service
• Added rocprofiler_load_counter_definition.

Resolved issues

• Create subdirectory when rocprofv3 --output-file includes a folder path
• Fixed misaligned stores (undefined behavior) for buffer records
• Fixed crash when only scratch reporting is enabled
• Fixed MeanOccupancy metrics
• Fixed aborted-application validation test to properly check for hipExtHostAlloc command
• Fixed implicit reduction of SQ and GRBM metrics
• Fixed support for derived counters in reduce operation
• Bug fixed in max-in-reduce operation
• Introduced fix to handle a range of values for select() dimension in expressions parser
• Conditional aql::set_profiler_active_on_queue only when counter collection is registered (resolves Navi3 kernel tracing issues)

Removed

• Removed gfx8 metric definitions
• Removed rocprofv3 installation to sbin directory

ROCprofiler-SDK 0.6.0 for ROCm release 6.4

Added

• Support for select() operation in counter expression.
• reduce() operation for counter expression with respect to dimension.
• --collection-period feature in rocprofv3 to enable filtering using time.
• --collection-period-unit feature in rocprofv3 to control time units used in collection period option.
• Deprecation notice for ROCProfiler and ROCProfilerV2.
• Support for rocDecode API Tracing
• Usage documentation for ROCTx
• Usage documentation for MPI applications
• SDK: rocprofiler_agent_v0_t support for agent UUIDs
• SDK: rocprofiler_agent_v0_t support for agent visibility based on gpu isolation environment variables such as ROCR_VISIBLE_DEVICES and so on.
• Accumulation VGPR support for rocprofv3.
• Host-trap based PC sampling support for rocprofv3.
• Support for OpenMP tool.

ROCprofiler-SDK 1.0.0 for ROCm release 7.0

Added

• Support for rocJPEG API Tracing.
• Support for AMD Instinct MI350X and MI355X accelerators.
• rocprofiler_create_counter to facilitate adding custom derived counters at runtime.
• Support in rocprofv3 for iteration based counter multiplexing.
• Perfetto support for counter collection.
• Support for negating rocprofv3 tracing options when using aggregate options such as --sys-trace --hsa-trace=no.
• --agent-index option in rocprofv3 to specify the agent naming convention in the output:
  • absolute == node_id
  • relative == logical_node_id
  • type-relative == logical_node_type_id
• MI300 and MI350 stochastic (hardware-based) PC sampling support in ROCProfiler-SDK and rocprofv3.
• Python bindings for rocprofiler-sdk-roctx
• SQLite3 output support for rocprofv3 using --output-format rocpd.
• rocprofiler-sdk-rocpd package:
  • Public API in include/rocprofiler-sdk-rocpd/rocpd.h.
  • Library implementation in librocprofiler-sdk-rocpd.so.
  • Support for find_package(rocprofiler-sdk-rocpd).
  • rocprofiler-sdk-rocpd DEB and RPM packages.
• --version option in rocprofv3.
• rocpd Python package.
• Thread trace as experimental API.
• ROCprof Trace Decoder as experimental API:
  • Requires ROCprof Trace Decoder plugin.
• Thread trace option in the rocprofv3 tool under the --att parameters:
  • See using thread trace with rocprofv3
  • Requires ROCprof Trace Decoder plugin.
• rocpd output format documentation:
  • Requires ROCprof Trace Decoder plugin.
• Perfetto support for scratch memory.
• Support in the rocprofv3 avail tool for command-line arguments.
• Documentation for rocprofv3 advanced options.
• Support for multi dispatch ATT file added

Changed

• SDK to NOT to create a background thread when every tool returns a nullptr from rocprofiler_configure.
• vaddr-to-file-offset mapping in disassembly.hpp to use the dedicated comgr API.
• rocprofiler_uuid_t ABI to hold 128 bit value.
• rocprofv3 shorthand argument for --collection-period to -P (upper-case) while -p (lower-case) is reserved for later use.
• Default output format for rocprofv3 to rocpd (SQLite3 database).
• rocprofv3 avail tool to be renamed from rocprofv3_avail to rocprofv3-avail tool.
• rocprofv3 tool to facilitate thread trace and PC sampling on the same agent.

Resolved issues

• Fixed missing callbacks around internal thread creation within counter collection service.
• Fixed potential data race in the ROCprofiler-SDK double buffering scheme.
• Fixed usage of std::regex in the core ROCprofiler-SDK library that caused segfaults or exceptions when used under dual ABI.
• Fixed Perfetto counter collection by introducing accumulation per dispatch.
• Fixed code object disassembly for missing function inlining information.
• Fixed queue preemption error and HSA_STATUS_ERROR_INVALID_PACKET_FORMAT error for stochastic PC-sampling in MI300X, leading to stabler runs.
• Fixed the system hang issue for host-trap PC-sampling on MI300X.
• Fixed rocpd counter collection issue when counter collection alone is enabled. rocpd_kernel_dispatch table is updated to be populated by counters data instead of kernel_dispatch data.
• Fixed rocprofiler_*_id_t structs for inconsistency related to a "null" handle:
  • The correct definition for a null handle is .handle = 0 while some definitions previously used UINT64_MAX.
• Fixed kernel trace csv output generated by rocpd.

Removed

• Support for compilation of gfx940 and gfx941 targets.

ROCprofiler-SDK 1.1.0 for ROCm release 7.1

Added

• Dynamic process attachment- ROCprofiler-sdk and rocprofv3 now facilitate dynamic profiling of a running GPU applications by attaching to its process ID (PID), rather than launching the application through the profiler itself.
• Scratch-memory trace information to the Perfetto output in rocprofv3.
• New capabilities to the thread trace support in rocprofv3, including real-time clock support for thread trace alignment on gfx9 architecture. This enables high-resolution clock computation and better synchronization across shader engines. Additionally, MultiKernelDispatch thread trace support is now available across all ASICs.
• Documentation for dynamic process attachment.
• Documentation for rocpd summaries.

Optimized

• Improved the stability and robustness of the rocpd output.

ROCprofiler-SDK 1.1.0 for ROCm release 7.2

Added

• Counter collection support for gfx1150 and gfx1151.
• HSA Extension API v8 support.
• hipStreamCopyAttributes API implementation.
• --profile-mpi-ranks option in rocprofv3 to selectively profile specific MPI ranks. Supports comma-separated ranges and individual ranks (e.g., --profile-mpi-ranks 0-3,8,10-15).

Optimized

• Improved process attachment and updated the corresponding documentation.
• Improved [Quick reference guide for rocprofv3] (https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/latest/quick_guide.html).
• Updated installation documentation with links to the latest repository (https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/latest/install/installation.html).

Resolved issues

• Fixed multi-GPU dimension mismatch.
• Fixed device lock issue for dispatch counters.
• Addressed OpenMP Tools task scheduling null pointer exception.
• Fixed stream ID errors arising during process attachment.
• Fixed issues arising during dynamic code object loading.