TheRock/rocm-systems/projects/rocprofiler-compute/docs/design/lld-profiler-hub-native-tool-integration.md

Write rocpd from the native tool via profiler-hub

Motivation

During a profiling run rocprofiler-compute produces data from two in-process libraries, both loaded via LD_PRELOAD into the target application (src/rocprof_compute_profile/profiler_rocprofiler_sdk.py):

1. The rocprofiler-sdk "SDK tool", run with ROCPROF_OUTPUT_FORMAT=rocpd, writes a rocpd SQLite database (kernel dispatches, agents, kernel symbols, an rocpd_info_pmc table, and an empty rocpd_pmc_event table). It is told NOT to collect counters (ROCPROF_COUNTER_COLLECTION=0).
2. The rocprofiler-compute "native tool" (src/lib/rocprofiler_compute_tool) collects the hardware counters and currently writes a CSV (<pid>_native_counter_collection.csv), which the Python layer parses and injects row by row into the SDK rocpd's rocpd_pmc_event table (src/utils/rocpd_data.py, src/utils/utils_profile.py).

This replaces the CSV round-trip: the native tool writes its own rocpd directly via profiler-hub, and the merge into the SDK rocpd moves to analyze mode as a single set-based SQL statement instead of bespoke CSV parsing in profile mode. Each tool keeps its own rocpd; they are never co-mingled into one schema.

This builds on separate, in-flight work that consolidates each tool's per-PID rocpds into a single rocpd per tool; that work must land first. This design takes the two consolidated rocpds (one per tool, per application-replay pass) as input and does not change how they are produced.

Scope is counters only. ISA / code-object collection stays as the existing JSON output: it backs PC sampling, which rocpd has no schema or profiler-hub writer support for yet, so there is nowhere for that data to land.

Data flow

flowchart TD
    App["target application<br/>(both libs via LD_PRELOAD)"]
    App --> SDK["rocprofiler-sdk 'SDK tool'<br/>ROCPROF_COUNTER_COLLECTION=0"]
    App --> NAT["rocprofiler-compute 'native tool'<br/>(HW counters)"]

    SDK -->|writes rocpd| SDKDB[("consolidated SDK rocpd (per pass)<br/>kernel_dispatch (pid, dispatch_id, event_id)<br/>info_pmc<br/>pmc_event [EMPTY]")]
    NAT -->|writes rocpd via profiler-hub| NATDB[("consolidated native rocpd (per pass)<br/>info_pmc (symbol, target_arch)<br/>event (pid, correlation_id = dispatch_id)<br/>pmc_event (value)")]

    SDKDB --> MERGE["analyze merge (per pass)<br/>ATTACH + INSERT...SELECT<br/>(pid, dispatch_id) -> SDK event_id<br/>symbol + target_arch -> SDK pmc_id"]
    NATDB --> MERGE

    MERGE --> OUT[("SDK pmc_event populated<br/>-> counters_collection view -> pmc_perf.csv")]

profiler-hub intro

profiler-hub is a standalone library for writing rocpd (SQLite) databases. Its public API is storage_t plus writer_t (register/insert calls for the rocpd tables).

Key behavior: writer_t mints a fresh rocpd_event per insert, so a pmc event and a dispatch cannot share an event_id through the API; dispatch linkage is carried on event_data_t.correlation_id, not a shared PK.

Nothing consumes profiler-hub yet and it is not part of the TheRock build; rocprofiler-compute will be its first consumer.

Native tool C++ changes

Add RocpdCountersWriter : public CountersWriter (the existing abstraction in counters_writer.h) as the default g_counters_writer, demoting the CSV writer, which stays for tests. The native tool always writes a _native_counter_collection.db rocpd (no env-var gating).

write_counters opens storage_t + writer_t, registers the FK-target info rows once (node, process, agent per distinct agent_id, pmc per distinct counter into rocpd_info_pmc), then per counter record calls insert_pmc_event_data with correlation_id = dispatch_id. It writes ONLY the counter tables; kernel_dispatch, kernel_symbol, and code_object come from the SDK rocpd at merge time.

Agent and counter metadata must be captured in the callbacks, which today keep only the agent handle and counter name. register_agent_info / register_pmc_info need more (agent type/name/node, counter target_arch): wrap rocprofiler_query_available_agents for the agent fields and retain the full rocprofiler_counter_info_v0_t already queried for counters. v0 has no event_code, so that column stays null as the SDK tool leaves it (confirmed null for every counter in the verified rocflop rocpd).

The native register_pmc_info must write the same target_arch the SDK tool uses so the merge join matches. In the verified rocflop rocpd that value is the literal string GPU for hardware counters, not a gfx arch, so the native writer sets target_arch = "GPU".

Analyze layer changes

Replace the profile-mode CSV read + row-by-row insert (update_rocpd_pmc_events in src/utils/rocpd_data.py) with a set-based merge that runs in analyze mode. For each application-replay pass, given the consolidated SDK and native rocpds:

ATTACH DATABASE :native_db AS native;

INSERT INTO "<sdk_pmc_event_table>" (guid, event_id, pmc_id, value)
SELECT
    sdk_kd.guid,
    sdk_kd.event_id,                      -- remapped to SDK event id space
    sdk_pmc.id,                           -- remapped to SDK info_pmc id space
    n_pmc.value
FROM native.rocpd_pmc_event  AS n_pmc
JOIN native.rocpd_event      AS n_ev
     ON n_ev.id = n_pmc.event_id
JOIN rocpd_kernel_dispatch   AS sdk_kd
     ON sdk_kd.dispatch_id = n_ev.correlation_id     -- dispatch_id carrier
     AND sdk_kd.pid = n_ev.pid                        -- pid scopes the dispatch
JOIN native.rocpd_info_pmc   AS n_info
     ON n_info.id = n_pmc.pmc_id
JOIN rocpd_info_pmc          AS sdk_pmc
     ON sdk_pmc.symbol = n_info.symbol
     AND sdk_pmc.target_arch IS n_info.target_arch;

DETACH DATABASE native;

The read side uses the per-db base-name views (rocpd_pmc_event, rocpd_event, rocpd_kernel_dispatch, rocpd_info_pmc) that the rocpd schema bakes into every db as SELECT * FROM <table>_<guid>. Both the SDK db and the native db carry them (profiler-hub writes the same rocpd schema), so the merge never computes a guid suffix to read. The two dbs stay distinct through the native. qualifier versus the main schema, which is required so the symbol to SDK pmc_id remap joins native rocpd_info_pmc against the SDK rocpd_info_pmc instead of collapsing them. This is also why the rocpd Python package is not used for the merge: its connect() unions same-base tables across all inputs, which would merge the two rocpd_info_pmc tables and destroy that cross-tool identity.

These base-name views are plain SELECT * views with no INSERT triggers, so they are not writable. The INSERT target therefore stays the physical SDK rocpd_pmc_event_<guid> table.

The remap is needed because event_id is assigned independently by each tool, while (pid, dispatch_id) is shared, so it is the stable key to recover the SDK event_id. The merge relies on two verified SDK-rocpd facts: rocpd_pmc_event and rocpd_kernel_dispatch both reference rocpd_event via event_id and rocpd_kernel_dispatch carries dispatch_id; and counter identity lives in rocpd_info_pmc (referenced by pmc_id), matched across tools on symbol + target_arch. The analysis counters view joins pmc to dispatch on the shared event_id, so the merge must land SDK event_id values.

After the merge, read the SDK counters_collection view and concatenate the result across all passes into pmc_perf.csv; the rest of the analyze pipeline is unchanged. Profile mode drops its native-counter injection and the now-dead CSV plumbing (src/utils/utils_profile_csv.py).

The symbol + target_arch join key is unique: in the verified rocflop rocpd all 15 rocpd_info_pmc rows have distinct symbol values.

Packaging changes

Build profiler-hub from source in-repo via ExternalProject_Add, linked statically into the single librocprofiler-compute-tool.so. add_subdirectory (one CMake configure) will not work: profiler-hub FetchContent's its own fmt, spdlog, nlohmann_json, and sqlite3, which collide with the fmt, json, and googletest under src/lib/external/ and duplicate those targets. A separate CMake invocation isolates its dependency resolution.

The sparse-checkout guard and ExternalProject_Add go in src/lib/CMakeLists.txt, not the top-level CMakeLists.txt, because two configure entry points build the native tool and only src/lib is common to both:

• Full / install build (TheRock CI): top-level CMakeLists.txt reaches it via add_subdirectory(src/lib).
• Runtime fallback build: the Python layer configures src/lib directly (src/utils/native_tool_finder.py) when no installed library is found, never running the top-level CMakeLists.txt.

src/lib/CMakeLists.txt is a standalone project(), so it configures either way, and the guard resolves the monorepo root via git rev-parse --show-toplevel regardless of entry point. The guard materializes profilers/profiler-hub for sparse checkouts (a no-op for full checkouts) before ExternalProject_Add builds it; requires cone-mode sparse checkout (git >= 2.27 default):

set(PROFILER_HUB_PATH profilers/profiler-hub)

execute_process(
  COMMAND ${GIT_EXECUTABLE} rev-parse --show-toplevel
  WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
  OUTPUT_VARIABLE REPO_ROOT OUTPUT_STRIP_TRAILING_WHITESPACE)

execute_process(
  COMMAND ${GIT_EXECUTABLE} config --get core.sparseCheckout
  WORKING_DIRECTORY ${REPO_ROOT}
  OUTPUT_VARIABLE SPARSE_ENABLED OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET)

if(SPARSE_ENABLED STREQUAL "true" AND NOT EXISTS "${REPO_ROOT}/${PROFILER_HUB_PATH}/CMakeLists.txt")
  message(STATUS "Adding profiler-hub to sparse-checkout...")
  execute_process(
    COMMAND ${GIT_EXECUTABLE} sparse-checkout add ${PROFILER_HUB_PATH}
    WORKING_DIRECTORY ${REPO_ROOT} RESULT_VARIABLE rc)
  if(NOT rc EQUAL 0)
    message(FATAL_ERROR "Failed to expand sparse-checkout for profiler-hub")
  endif()
endif()
# source now guaranteed on disk; ExternalProject_Add below builds it in isolation.

• In src/lib/CMakeLists.txt, add an ExternalProject_Add (or equivalent) that builds profiler-hub from profilers/profiler-hub with PROFILER_HUB_BUILD_TESTS=OFF, PROFILER_HUB_BUILD_BENCHMARKS=OFF, default bundled schema, producing libprofiler-hub.a.
• Link profiler-hub-static into rocprofiler-compute-tool (src/lib/rocprofiler_compute_tool/CMakeLists.txt).

Building and testing

• Clean-from-scratch build, confirming the ExternalProject wiring and dependency isolation hold.
• Run a small workload (e.g. tests/vcopy) with the native tool preloaded; confirm the _native_counter_collection.db has rocpd_info_pmc, rocpd_event (with correlation_id), and rocpd_pmc_event populated.
• Run the analyze-mode merge against real consolidated rocpds; confirm rocpd_pmc_event fills, the counters view joins cleanly, and pmc_perf.csv matches expected dispatch x counter rows.
• Cover the pure logic (merge SQL, per-pass native/SDK db pairing) with new unit tests.
• Run the full ctest suite and ensure it passes, to catch regressions in the existing CSV-era and analysis paths the merge replaces.