derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
164 lines
4.4 KiB
ReStructuredText
164 lines
4.4 KiB
ReStructuredText
.. meta::
|
|
:description: RCCL is a stand-alone library that provides multi-GPU and multi-node collective communication primitives optimized for AMD GPUs
|
|
:keywords: RCCL, ROCm, library, API
|
|
|
|
.. _library-specification:
|
|
|
|
============================
|
|
RCCL library specification
|
|
============================
|
|
|
|
This document provides details of the API library.
|
|
|
|
Communicator functions
|
|
----------------------
|
|
|
|
.. doxygenfunction:: ncclGetUniqueId
|
|
|
|
.. doxygenfunction:: ncclCommInitRank
|
|
|
|
.. doxygenfunction:: ncclCommInitAll
|
|
|
|
.. doxygenfunction:: ncclCommDestroy
|
|
|
|
.. doxygenfunction:: ncclCommAbort
|
|
|
|
.. doxygenfunction:: ncclCommCount
|
|
|
|
.. doxygenfunction:: ncclCommCuDevice
|
|
|
|
.. doxygenfunction:: ncclCommUserRank
|
|
|
|
Fault tolerance and communicator management
|
|
-------------------------------------------
|
|
|
|
These functions let applications detect failures and recover by resizing or
|
|
rebuilding communicators. See :ref:`fault-tolerance` for usage guidance.
|
|
|
|
.. doxygenfunction:: ncclCommGetAsyncError
|
|
|
|
.. doxygenfunction:: ncclCommFinalize
|
|
|
|
.. doxygenfunction:: ncclCommSplit
|
|
|
|
.. doxygenfunction:: ncclCommShrink
|
|
|
|
.. doxygenfunction:: ncclCommGrow
|
|
|
|
.. doxygenfunction:: ncclCommGetUniqueId
|
|
|
|
.. doxygenfunction:: ncclCommRevoke
|
|
|
|
.. _communicator-suspend-resume:
|
|
|
|
Communicator suspend and resume
|
|
-------------------------------
|
|
|
|
These functions release and reacquire the resources held by a communicator
|
|
that is temporarily idle, and report per-communicator memory statistics. They
|
|
are useful when an application wants to free GPU memory held by an inactive
|
|
communicator, and then later resume collective operations on the same communicator.
|
|
|
|
Releasing the physical backing of a suspended communicator requires virtual
|
|
memory management (VMM) support enabled through ``NCCL_CUMEM_ENABLE``. Without
|
|
it, ``ncclCommSuspend`` and ``ncclCommResume`` succeed but don't release GPU
|
|
memory. See :ref:`suspend-resume` for the full list of prerequisites.
|
|
|
|
Use ``ncclGroupStart`` and ``ncclGroupEnd`` when suspending or resuming multiple
|
|
communicators from one thread. Requests run at ``ncclGroupEnd`` after all ranks
|
|
of each affected communicator synchronize.
|
|
|
|
.. doxygendefine:: NCCL_SUSPEND_MEM
|
|
|
|
.. doxygenfunction:: ncclCommSuspend
|
|
|
|
.. doxygenfunction:: ncclCommResume
|
|
|
|
.. doxygenenum:: ncclCommMemStat_t
|
|
|
|
.. doxygenfunction:: ncclCommMemStats
|
|
|
|
Collective communication operations
|
|
-----------------------------------
|
|
|
|
Collective communication operations must be called separately for each communicator in a communicator clique.
|
|
|
|
They return when operations have been enqueued on the hipstream.
|
|
|
|
Since they may perform inter-CPU synchronization, each call has to be done from a different thread or process, or need to use Group Semantics (see below).
|
|
|
|
.. doxygenfunction:: ncclReduce
|
|
|
|
.. doxygenfunction:: ncclBcast
|
|
|
|
.. doxygenfunction:: ncclBroadcast
|
|
|
|
.. doxygenfunction:: ncclAllReduce
|
|
|
|
.. doxygenfunction:: ncclReduceScatter
|
|
|
|
.. doxygenfunction:: ncclAllGather
|
|
|
|
.. doxygenfunction:: ncclSend
|
|
|
|
.. doxygenfunction:: ncclRecv
|
|
|
|
.. doxygenfunction:: ncclGather
|
|
|
|
.. doxygenfunction:: ncclScatter
|
|
|
|
.. doxygenfunction:: ncclAllToAll
|
|
|
|
Group semantics
|
|
---------------
|
|
When managing multiple GPUs from a single thread, and since NCCL collective
|
|
calls may perform inter-CPU synchronization, we need to "group" calls for
|
|
different ranks/devices into a single call.
|
|
|
|
Grouping NCCL calls as being part of the same collective operation is done
|
|
using ncclGroupStart and ncclGroupEnd. ncclGroupStart will enqueue all
|
|
collective calls until the ncclGroupEnd call, which will wait for all calls
|
|
to be complete. Note that for collective communication, ncclGroupEnd only
|
|
guarantees that the operations are enqueued on the streams, not that
|
|
the operation is effectively done.
|
|
|
|
Both collective communication and ncclCommInitRank can be used in conjunction
|
|
of ncclGroupStart/ncclGroupEnd.
|
|
|
|
.. doxygenfunction:: ncclGroupStart
|
|
|
|
.. doxygenfunction:: ncclGroupEnd
|
|
|
|
Library functions
|
|
-----------------
|
|
|
|
.. doxygenfunction:: ncclGetVersion
|
|
|
|
.. doxygenfunction:: ncclGetErrorString
|
|
|
|
Types
|
|
-----
|
|
|
|
There are few data structures that are internal to the library. The pointer types to these
|
|
structures are given below. The user would need to use these types to create handles and pass them
|
|
between different library functions.
|
|
|
|
.. doxygentypedef:: ncclComm_t
|
|
|
|
.. doxygenstruct:: ncclUniqueId
|
|
|
|
|
|
|
|
Enumerations
|
|
------------
|
|
|
|
This section provides all the enumerations used.
|
|
|
|
.. doxygenenum:: ncclResult_t
|
|
|
|
.. doxygenenum:: ncclRedOp_t
|
|
|
|
.. _rccl-supported-data-types:
|
|
|
|
.. doxygenenum:: ncclDataType_t
|