.. meta:: :description: A guide to troubleshooting the RCCL library of multi-GPU and multi-node collective communication primitives optimized for AMD GPUs :keywords: RCCL, ROCm, library, API, debug .. _troubleshooting-rccl: ********************* Troubleshooting RCCL ********************* This topic explains the steps to troubleshoot functional and performance issues with RCCL. While debugging, collect the output from the commands in this guide. This data can be used as supporting information when submitting an issue report to AMD. .. _debugging-system-info: Collecting system information ============================= Collect this information about the ROCm version, GPU/accelerator, platform, and configuration. * Verify the ROCm version. This might be a release version or a mainline or staging version. Use this command to display the version: .. code:: shell cat /opt/rocm/.info/version Run the following command and collect the output: .. code:: shell rocm_agent_enumerator Also, collect the name of the GPU or accelerator: .. code:: shell rocminfo * Run these ``amd-smi`` commands to display the system topology. .. code:: shell amd-smi amd-smi topology amd-smi static --driver amd-smi firmware amd-smi xgmi * Determine the values of the ``PATH`` and ``LD_LIBRARY_PATH`` environment variables. .. code:: shell echo $PATH echo $LD_LIBRARY_PATH * Collect the HIP configuration. .. code:: shell /opt/rocm/bin/hipconfig --full * Verify the network settings and setup. Use the ``ibv_devinfo`` command to display information about the available RDMA devices and determine whether they are installed and functioning properly. Run ``rdma link`` to print a summary of the network links. .. code:: shell ibv_devinfo rdma link Isolating the issue ------------------- The problem might be a general issue or specific to the architecture or system. To narrow down the issue, collect information about the GPU or accelerator and other details about the platform and system. Some issues to consider include: * Is ROCm running on: * A bare-metal setup * In a Docker container (determine the name of the Docker image) * In an SR-IOV virtualized * Some combination of these configurations * Is the problem only seen on a specific GPU architecture? * Is it only seen on a specific system type? * Is it happening on a single node or multinode setup? * Use the following troubleshooting techniques to attempt to isolate the issue. * Build or run the develop branch version of RCCL and see if the problem persists. * Try an earlier RCCL version (minor or major). * If you recently changed the ROCm runtime configuration, AMD Kernel-mode GPU Driver (KMD), or compiler, rerun the test with the previous configuration. .. _collecting-rccl-info: Collecting RCCL information ============================= Collect the following information about the RCCL installation and configuration. * Run the ``ldd`` command to list any dynamic dependencies for RCCL. .. code:: shell ldd * Determine the RCCL version. This might be the pre-packaged component in ``/opt/rocm/lib`` or a version that was built from source. To verify the RCCL version, enter the following command, then run either rccl-tests or an e2e application. .. code:: shell export NCCL_DEBUG=VERSION * Run rccl-tests and collect the results. For information on how to build and run rccl-tests, see the `rccl-tests GitHub `_. * Collect the RCCL logging information. Enable the debug logs, then run rccl-tests or any e2e workload to collect the logs. Use the following command to enable the logs. .. code:: shell export NCCL_DEBUG=INFO .. _use-rccl-replayer: Using the RCCL Replayer ------------------------ The RCCL Replayer is a debugging tool designed to analyze and replay the collective logs obtained from RCCL runs. It can be helpful when trying to reproduce problems, because it uses dummy data and doesn't have any dependencies on non-RCCL calls. For more information, see `RCCL Replayer GitHub documentation `_. You must build the RCCL Replayer before you can use it. To build it, run these commands. Ensure ``MPI_DIR`` is set to the path where MPI is installed. .. code:: shell cd rccl/tools/rccl_replayer MPI_DIR=/path/to/mpi make To use the RCCL Replayer, follow these steps: #. Collect the per-rank logs from the RCCL run by adding the following environment variables. This prevents any race conditions that might cause ranks to interrupt the output from other ranks. .. code:: shell NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=COLL NCCL_DEBUG_FILE=some_name_here.%h.%p.log #. Combine all the logs into a single file. This will become the input to the RCCL Replayer. .. code:: shell cat some_name_here_*.log > some_name_here.log #. Run the RCCL Replayer using the following command. Replace ```` with the number of MPI processes to run, ```` with the path to the collective log file generated during the RCCL runs, and ```` with the number of GPUs per MPI rank used in the application. .. code:: shell mpirun -np ./rcclReplayer In a multi-node application environment, you can replay the collective logs on multiple nodes using the following command: .. code:: shell mpirun --hostfile -np ./rcclReplayer .. note:: Depending on the MPI library you're using, you might need to modify the ``mpirun`` command. .. _analyze-performance-info: Analyzing performance issues ============================= If the issues involve performance issues in an e2e workload, try the following microbenchmarks and collect the results. Follow the instructions in the subsequent sections to run these benchmarks and provide the results to the support team. * TransferBench * RCCL Unit Tests * rccl-tests Collect the TransferBench data --------------------------------- TransferBench allows you to benchmark simultaneous copies between user-specified devices. For more information, see the :doc:`TransferBench documentation `. To collect the TransferBench data, follow these steps: #. Clone the TransferBench Git repository. .. code:: shell git clone https://github.com/ROCm/TransferBench.git #. Change to the new directory and build the component. .. code:: shell cd TransferBench make #. Run the TransferBench utility with the following parameters and save the results. .. code:: shell USE_FINE_GRAIN=1 GFX_UNROLL=2 ./TransferBench a2a 64M 8 Collect the RCCL microbenchmark data ------------------------------------- To use the RCCL tests to collect the RCCL benchmark data, follow these steps: #. Disable NUMA auto-balancing using the following command: .. code:: shell sudo sysctl kernel.numa_balancing=0 Run the following command to verify the setting. The expected output is ``0``. .. code:: shell cat /proc/sys/kernel/numa_balancing #. Build MPI, RCCL, and rccl-tests. To download and install MPI, see either `OpenMPI `_ or `MPICH `_. To learn how to build and run rccl-tests, see the `rccl-tests GitHub `_. #. Run rccl-tests with MPI and collect the performance numbers. RCCL and NCCL comparisons ============================= If you are also using NVIDIA hardware or NCCL and notice a performance gap between the two systems, collect the system and performance data on the NVIDIA platform. Provide both sets of data to the support team. .. _heterogeneous-nic-counts: Ranks with different NIC counts =============================== On multi-NIC systems, a job can end up with an inconsistent number of network (NET) devices per rank, for example when ranks are launched with different ``NCCL_SOCKET_IFNAME`` or ``NCCL_IB_HCA`` values, or when nodes in the job have a different number of NICs. Historically this only became visible much later during transport setup as an obscure connection or timeout failure. During communicator initialization RCCL now gathers each rank's local NET device count and compares them. When they differ, rank 0 logs the offending ranks and the min/max counts, for example: .. code:: shell NCCL INFO Local Net device counts across ranks: min 1 max 2 NCCL INFO Rank 1 has 1 local Net devices (max 2). The reaction is controlled by ``NCCL_IGNORE_NET_MISMATCH`` (and the CollNet equivalent ``NCCL_IGNORE_COLLNET_MISMATCH``): * ``NCCL_IGNORE_NET_MISMATCH=1`` (the default) continues, logging the mismatch at ``INFO`` level:: NCCL INFO Detected mixed local Net device counts across ranks (min 1, max 2). Ignoring due to NCCL_IGNORE_NET_MISMATCH. * ``NCCL_IGNORE_NET_MISMATCH=0`` fails initialization with ``ncclSystemError`` so the misconfiguration is caught immediately:: NCCL WARN Detected mixed local Net device counts across ranks (min 1, max 2). Set NCCL_IGNORE_NET_MISMATCH=1 to continue. If you hit this warning, verify that every rank selects the same set of NICs (consistent ``NCCL_SOCKET_IFNAME`` / ``NCCL_IB_HCA``) and that all nodes expose the same NIC count. Set ``NCCL_IGNORE_NET_MISMATCH=0`` in CI or bring-up to turn a heterogeneous-NIC misconfiguration into an immediate, explicit failure.