TheRock/rocm-systems/projects/rocshmem/CHANGELOG.md

Changelog for rocSHMEM

Unreleased - rocSHMEM 3.6.0 for ROCm x.x.x

rocSHMEM 3.5.0 for ROCm 7.14

Added

• Added new APIs:
  • rocshmem_align
  • rocshmem_calloc
  • rocshmem_buffer_unregister_all
  • rocshmem_buffer_register/unregister for GDA backend
  • rocshmem_buffer_register_symmetric for IPC and GDA backends
  • rocshmem_buffer_unregister_symmetric for IPC and GDA backends
  • rocshmem_reduce_on_stream
  • rocshmem_team_split_2D
• Added tile-granular RMA operations for the IPC backend
• Added host-initiated RMA operations in the IPC backend for the non-MPI bootstrapping path: put, get, fence, quiet, arithmetic AMOs, and P2P sync ops
• Added team creation using non-contiguous parent teams in the IPC backend
• Added Python bindings of memory-management APIs
• Added Python bindings coverage for team APIs
• Added support for GPU initiated operations using the SDMA engines
• Added ASAN build support

Changed

• Changed default ROCSHMEM_DEBUG_LEVEL from WARN to ERROR
• Performance optimizations:
  • Separated put/get memcpy primitives to apply correct cache coherence semantics and fences
  • Use constmem for backend variables and provider muxing
  • Updated O(1) IPC availability check using pattern detection

rocSHMEM 3.4.0 for ROCm 7.13

Added

• Added new APIs:
  • rocshmem_quiet_on_stream
  • rocshmem_sync_all_on_stream
  • rocshmem_TYPENAME_alltoall_wg
  • rocshmem_TYPENAME_alltoallv_wg
  • rocshmem_team_my_pe
  • rocshmem_team_n_pes
  • rocshmem_barrier
  • rocshmem_barrier_wave
  • rocshmem_barrier_wg
  • rocshmem_buffer_register
  • rocshmem_buffer_unregister
  • rocshmem_info_get_version
  • rocshmem_info_get_name
  • rocshmem_vendor_get_version_info
• Added library constants: ROCSHMEM_MAJOR_VERSION, ROCSHMEM_MINOR_VERSION, ROCSHMEM_MAX_NAME_LEN, ROCSHMEM_VENDOR_STRING, ROCSHMEM_VERSION, ROCSHMEM_VENDOR_MAJOR_VERSION, ROCSHMEM_VENDOR_MINOR_VERSION, ROCSHMEM_VENDOR_PATCH_VERSION
• Added vendor string and backend metadata to the rocshmem_info output
• Added ROCSHMEM_TEAM_WORLD for device code
• Added ROCSHMEM_TEAM_SHARED predefined team for PEs sharing a common memory domain (same node)
• Added new environment variables:
  • OVERRIDE_NIC_FIRMWARE_CHECK
  • ROCSHMEM_GDA_NUM_QPS_PER_PE_DEFAULT_CTX
  • ROCSHMEM_GDA_NUM_QPS_PER_PE_USR_CTX
  • ROCSHMEM_MAX_SYMM_REGIONS
• Added VMM POSIX memory allocator (USE_HEAP_DEVICE_VMM_POSIX)
  • Uses HIP Virtual Memory Management (VMM) APIs for fine-grained memory control
  • Requires ROCm 7.0+ and Linux kernel 5.6+
  • Not compatible with MPI-based initialization (use ROCSHMEM_INIT_WITH_UNIQUEID instead)

Changed

• Use CQ collapsing for the Mellanox MLX5 GDA conduit

rocSHMEM 3.2.1 for ROCm 7.2.1

Added

• Warn if large BAR is not available

Resolved Issues

• GDA Backend will disable itself when no GDA compatible NICs are available rather than crashing
• Fix memory coherency issues on gfx1201

Known issues

• Only 64bit rocSHMEM atomic APIs are implemented for the GDA conduit

rocSHMEM 3.2.0 for ROCm 7.2.0

Added

• Added the GDA conduit for AMD Pensando IONIC

Changed

• Dependency libraries are loaded dynamically
• The following APIs now have an implementation for the GDA conduit
  • rocshmem_p
  • fetching atomics rochsmem_<TYPE>_fetch_<op>
  • collective APIs
• The following APIs now have an implementation for the IPC conduit
  • rocshmem_<TYPE>_atomic_{and,or,xor,swap}
  • rocshmem_<TYPE>_atomic_fetch_{and,or,xor,swap}

Known issues

• Only 64bit rocSHMEM atomic APIs are implemented for the GDA conduit

rocSHMEM 3.1.0 for ROCm 7.1.1

Added

• Allow for IPC, RO, GDA backends to be selected at runtime
• Added the GDA conduit for different NIC vendors
  • Broadcom BNXT_RE (Thor 2)
  • Mellanox MLX5 (IB and RoCE ConnectX-7)
• Added new APIs:
  • rocshmem_get_device_ctx
  • rocshmem_ctx_pe_quiet
  • rocshmem_pe_quiet

Changed

• The following APIs have been deprecated:
  • rocshmem_wg_init
  • rocshmem_wg_finalize
  • rocshmem_wg_init_thread
• rocshmem_ptr can now return non-null pointer to a shared memory region when the IPC transport is available to reach that region. Previously, it would return a null pointer.
• ROCSHMEM_RO_DISABLE_IPC was renamed to ROCSHMEM_DISABLE_MIXED_IPC. This environment variable was not documented for prior releases. It is now documented to inform users who were using this undocumented feature.

Removed

• rocSHMEM no-longer requires rocPRIM and rocThrust as dependencies
• Removed MPI compile-time dependency

Known issues

• Only a subset of rocSHMEM APIs are implemented for the GDA conduit

rocSHMEM 3.0.0 for ROCm 7.0.0

Added

• Added the Reverse Offload conduit
• Added new APIs:
  • rocshmem_ctx_barrier
  • rocshmem_ctx_barrier_wave
  • rocshmem_ctx_barrier_wg
  • rocshmem_barrier_all
  • rocshmem_barrier_all_wave
  • rocshmem_barrier_all_wg
  • rocshmem_ctx_sync
  • rocshmem_ctx_sync_wave
  • rocshmem_ctx_sync_wg
  • rocshmem_sync_all
  • rocshmem_sync_all_wave
  • rocshmem_sync_all_wg
  • rocshmem_init_attr
  • rocshmem_get_uniqueid
  • rocshmem_set_attr_uniqueid_args
• Added dlmalloc based allocator
• Added XNACK support
• Added support for initialization with MPI communicators other than MPI_COMM_WORLD

Changed

• Changed collective APIs to use _wg suffix rather than _wg_ infix

Resolved Issues

• Resolved segfault in rocshmem_wg_ctx_create, now provides nullptr if ctx cannot be created

rocSHMEM 2.0.1 for ROCm 6.4.2

Resolved Issues

• Resolved incorrect output for rocshmem_ctx_my_pe and rocshmem_ctx_n_pes
• Resolved multi-team errors by providing team specific buffers in rocshmem_ctx_wg_team_sync
• Resolved missing implementation of rocshmem_g for IPC conduit