TheRock/rocm-systems/projects/rocprofiler-systems/docs/how-to/xgmi-pcie-sdma-sampling.rst

.. meta::
   :description: ROCm Systems Profiler XGMI, PCIe, and SDMA metrics sampling and
      monitoring
   :keywords: rocprof-sys, rocprofiler-systems, ROCm, tips, how to, profiler, tracking,
      XGMI, PCIe, SDMA, GPU connectivity, AMD

*************************************************************
XGMI, PCIe, and SDMA metrics sampling and monitoring
*************************************************************

`ROCm Systems Profiler`__ supports sampling of XGMI and PCIe interconnect metrics, as well as SDMA engine utilization, via AMD SMI.

It allows you to gather key performance metrics for GPU-to-GPU communication via XGMI
links, CPU-to-GPU communication via PCIe links, and asynchronous copy activity handled by
the SDMA engines. Use this information to optimize multi-GPU workloads, identify communication bottlenecks, and analyze data transfer efficiency in high-performance computing applications.

__ https://github.com/ROCm/rocm-systems/tree/develop/projects/rocprofiler-systems

Sampling support
=================

Sampling of XGMI, PCIe, and SDMA metrics is supported by leveraging `AMD SMI
<https://rocm.docs.amd.com/projects/amdsmi/en/latest/>`_ which provides the interface for
GPU metric collection. To enable sampling, follow these steps:

1. Set the ``ROCPROFSYS_USE_AMD_SMI`` environment variable to enable GPU metric collection:

.. code-block:: shell

  export ROCPROFSYS_USE_AMD_SMI=true

2. Update the ``ROCPROFSYS_AMD_SMI_METRICS`` variable to collect the XGMI, PCIe, and/or
   SDMA metrics. The default value is:

.. code-block:: shell

  ROCPROFSYS_AMD_SMI_METRICS=busy,temp,power,mem_usage

To include XGMI, PCIe, and SDMA usage metrics, update it to:

.. code-block:: shell

  ROCPROFSYS_AMD_SMI_METRICS=busy,temp,power,mem_usage,xgmi,pcie,sdma_usage

Alternatively, to collect all available GPU metrics, run:

.. code-block:: shell

  ROCPROFSYS_AMD_SMI_METRICS=all

XGMI metrics
------------

XGMI (AMD Infinity Fabric™ XGMI) provides high-bandwidth, low-latency GPU-to-GPU interconnects in multi-GPU systems. The following XGMI metrics are collected:

- **XGMI Link Width**: The number of active XGMI links between GPUs.
- **XGMI Link Speed**: The speed of XGMI links (in GT/s).
- **XGMI Read Data**: Accumulated data read through each XGMI link (in KB).
- **XGMI Write Data**: Accumulated data written through each XGMI link (in KB).

These metrics help identify GPU-to-GPU communication patterns and bandwidth utilization in multi-GPU workloads.

.. note::

   XGMI metrics are only available on systems with multiple GPUs connected via XGMI links.
   The availability depends on the system topology and GPU architecture. If unsupported or not
   available, the values will be reported as N/A in the output.

PCIe metrics
------------

PCIe (PCI Express) provides the connection between the CPU and GPU. The following PCIe metrics are collected:

- **PCIe Link Width**: The number of PCIe lanes currently active
- **PCIe Link Speed**: The current PCIe link generation and speed (e.g., Gen3, Gen4, Gen5)
- **PCIe Bandwidth Accumulated**: Total bandwidth accumulated over time (in MB)
- **PCIe Bandwidth Instantaneous**: Instantaneous bandwidth at the time of sampling (in MB/s)

These metrics help analyze CPU-to-GPU data transfer efficiency and identify PCIe bottlenecks.

SDMA metrics
------------

SDMA engines perform asynchronous memory copies and related DMA work on the GPU. When
the ``sdma_usage`` metric is enabled, ROCm Systems Profiler samples SDMA usage:
device-level SDMA utilization as a percentage (0-100), aggregated across processes on that
GPU.

These samples help correlate sustained ``hipMemcpy``-style traffic and other SDMA-backed
transfers with engine load in Perfetto or ROCpd output.

For a small workload that exercises H2D, D2D, and D2H copies for benchmarking and
profiling, see `sdma-test on GitHub`_ and the `sdma-test README on GitHub`_ (build steps,
flags, and ``rocprof-sys-run`` invocations).

.. _`sdma-test on GitHub`:
   https://github.com/ROCm/rocm-systems/tree/develop/projects/rocprofiler-systems/examples/sdma-test

.. _`sdma-test README on GitHub`:
   https://github.com/ROCm/rocm-systems/blob/develop/projects/rocprofiler-systems/examples/sdma-test/README.md

.. note::

   The ``sdma_usage`` metric requires AMD GPU Driver (amdgpu) 31.40.0 or later and an AMD Instinct-family GPU.

   If the driver or hardware does not expose SDMA usage, values may appear as ``N/A``.

Using TransferBench for testing
================================

For testing and benchmarking GPU connectivity, you can use `TransferBench
<https://rocm.docs.amd.com/projects/TransferBench/en/latest/index.html>`_.
TransferBench is a benchmarking utility designed to measure the performance of
simultaneous data transfers between user-specified devices, such as CPUs and GPUs.

For this example, TransferBench is used to profile XGMI and PCIe traffic for analysis. For
SDMA-focused runs, use the :ref:`sdma-test-example` workflow instead.

1. Source the ROCm Systems Profiler Environment using:

.. code-block:: shell

   source /opt/rocprofiler-systems/share/rocprofiler-systems/setup-env.sh

Alternatively, if you are using modules, use:

.. code-block:: shell

   module use /opt/rocprofiler-systems/share/modulefiles

2. Generate and configure the profiler config file.

.. code-block:: shell

   rocprof-sys-avail -G $HOME/.rocprofsys.cfg -F txt
   export ROCPROFSYS_CONFIG_FILE=$HOME/.rocprofsys.cfg

Edit ``.rocprofsys.cfg`` with the following settings:

.. code-block:: shell

  ROCPROFSYS_USE_AMD_SMI     = true
  ROCPROFSYS_AMD_SMI_METRICS = busy,temp,power,mem_usage,xgmi,pcie
  ROCPROFSYS_ROCM_DOMAINS    = hip_runtime_api,memory_copy,hsa_api

3. Profile the TransferBench application.

.. code-block:: shell

  rocprof-sys-sample -PTHD -- ./TransferBench a2a

.. note::

   Refer to these steps to `Install and build TransferBench <https://rocm.docs.amd.com/projects/TransferBench/en/latest/install/install.html#install-transferbench>`_.

At the end of the run, a similar message appears::

  [rocprofiler-systems][964294][perfetto]> Outputting '/home/demo/rocprofsys-transferBench-output/2025-04-25_15.52/perfetto-trace-964294.proto'
  (3124.52 KB / 3.12 MB / 0.00 GB)... Done


To view the generated ``.proto`` file in the browser, open the `Perfetto UI page
<https://ui.perfetto.dev/>`_.

Then, click on ``Open trace file`` and select the ``.proto`` file. In the browser, you can
visualize the XGMI and PCIe metrics.

.. image:: ../data/rocprof-sys-xgmi.png
   :alt: Visualization of a performance graph in Perfetto with XGMI tracks

.. image:: ../data/rocprof-sys-pcie.png
   :alt: Visualization of a performance graph in Perfetto with PCIe tracks

The visualization will show:

- **XGMI Read Data** and **XGMI Write Data** tracks showing data transfer through XGMI links over time
- **XGMI Link Width** and **XGMI Link Speed** tracks showing link configuration
- **PCIe Bandwidth** tracks showing CPU-to-GPU data transfer rates
- **PCIe Link Width** and **PCIe Link Speed** tracks showing PCIe link configuration


.. _sdma-test-example:

Using the sdma-test example for testing
=========================================

The `sdma-test on GitHub`_ tree contains the example that drives Host-to-Device,
Device-to-Device, and Device-to-Host async copies so you can observe SDMA utilization
alongside HIP memory-copy tracing. Build instructions, CLI flags (transfer size,
iterations, infinite mode), and additional environment options are documented in the
`sdma-test README on GitHub`_.

1. Source the ROCm Systems Profiler environment:

.. code-block:: shell

   source /opt/rocprofiler-systems/share/rocprofiler-systems/setup-env.sh

Alternatively, if you are using modules, use:

.. code-block:: shell

   module use /opt/rocprofiler-systems/share/modulefiles

2. Generate and point to a config file, then edit ``.rocprofsys.cfg`` with settings such
   as:

.. code-block:: shell

   rocprof-sys-avail -G $HOME/.rocprofsys.cfg -F txt
   export ROCPROFSYS_CONFIG_FILE=$HOME/.rocprofsys.cfg

.. code-block:: shell

  ROCPROFSYS_USE_AMD_SMI     = true
  ROCPROFSYS_AMD_SMI_METRICS = busy,temp,power,mem_usage,sdma_usage
  ROCPROFSYS_ROCM_DOMAINS    = hip_runtime_api,memory_copy

3. Build ``sdma-test`` (see the `sdma-test README on GitHub`_), then profile it, for example:

.. code-block:: shell

   rocprof-sys-run -- ./sdma-test -s 512 -n 5

Open the resulting Perfetto trace as described above; when ``sdma_usage`` is supported on
your system, look for **SDMA usage** / device SDMA utilization tracks in addition to HIP
memory-copy activity.

.. image:: ../data/rocprof-sys-sdma.png
   :alt: Visualization of a performance graph in Perfetto with SDMA tracks

Tips for effective profiling
=============================

1. Multi-GPU workloads: XGMI metrics are most useful when profiling applications that use multiple GPUs and transfer data between them.

2. Sampling frequency: Adjust the sampling frequency using ``ROCPROFSYS_PROCESS_SAMPLING_FREQ`` (default is 50Hz) to capture more or fewer samples based on your analysis needs.

3. Focus on specific metrics: If you only need XGMI, PCIe, or SDMA metrics, you can
   specify just those:

   .. code-block:: shell

     ROCPROFSYS_AMD_SMI_METRICS=xgmi  # Only XGMI metrics
     ROCPROFSYS_AMD_SMI_METRICS=pcie  # Only PCIe metrics
     ROCPROFSYS_AMD_SMI_METRICS=sdma_usage  # Only SDMA usage

4. Combine with API tracing: For detailed analysis, combine XGMI, PCIe, or SDMA
   metrics with HIP/HSA API tracing to correlate data transfers with application behavior:

   .. code-block:: shell

     ROCPROFSYS_ROCM_DOMAINS=hip_runtime_api,memory_copy,kernel_dispatch,hsa_api

Exploring available metrics
============================

To explore all supported metrics and domains, use the following commands:

.. code-block:: shell

  rocprof-sys-avail --all                    # Show all available options
  rocprof-sys-avail -bd -r AMD_SMI_METRICS   # Show AMD SMI metrics
  rocprof-sys-avail -bd -r ROCM_DOMAINS      # Show ROCm tracing domains

For more details on ROCm Systems Profiler configuration, refer to the `configuration guide <configuring-runtime-options.html>`_.