# Changelog for rocThrust

Documentation for rocThrust available at
[https://rocm.docs.amd.com/projects/rocThrust/en/latest/](https://rocm.docs.amd.com/projects/rocThrust/en/latest/).

## rocThrust 4.6.0 for ROCm 10.0
 
### Added
 
* True global support in hipstdpar
* Return value checks for assorted locations in the codebase that were previously missing.
 
### Fixed
 
* Fixed over-read and NULL deref in __hipstdpar_realloc

### Upcoming changes

* CCCL 2.8.x compatibility is deprecated. hipCUB and rocThrust will be brought forward to CCCL 3.0.x compatibility in an upcoming version.

## rocThrust 4.5.0 for ROCm 7.14

### Added

* gfx1250 support
* For hipstdpar algorithms running on GPUs that support xnack, and __HIPSTDPAR_INTERPOSE_ALLOC__ or __HIPSTDPAR_INTERPOSE_ALLOC_V1__ are not enabled, emit a runtime warning once if xnack is off.

### Upcoming changes

* CCCL 2.8.x compatibility is deprecated. hipCUB and rocThrust will be brought forward to CCCL 3.0.x compatibility in an upcoming version.

## Since last release ROCm 7.12

### Resolved issues

* Fixed memory leak in unit test.
* Fixed unit test compatibility with ASAN.

## rocThrust 4.3.0 for ROCm 7.12

### Added

* If you are using rocThrust on the host-side only, you can now build using g++ or non-HIP-aware clang++. To configure rocThrust in this-way, set the new CMake option `ROCTHRUST_DEVICE_SYSTEM` to `CPP` (other options include `HIP`, `CUDA`, `OpenMP`, and `TBB`), and set `CXX` to g++ or clang++. Then install rocThrust via `make install`. When you compile your application, don't forget to include the rocThrust include directory (`-I /opt/rocm/include`), since this won't happen automatically like it does when building with hipcc. Note that currently, rocThrust tests and benchmarks cannot be built when configuring rocThrust for host-side-only use.

* Added `generate_resource_spec.cpp` to the test directory and built as a new target by CMake. It generates the resource spec file required by CTest when running tests in parallel.

### Changed

* Updated the documentation on how to run rocThrust tests on multiple GPUs in parallel.
* Renamed CMake option `ROCTHRUST_DEVICE_SYSTEM` to `LINK_HIP_DEVICE_LIBS` and changed it to a boolean (`ON/OFF`). Added new CMake options `THRUST_DEVICE_SYSTEM` and `THRUST_HOST_SYSTEM` to set the device and host backends, respectively. Device system options include `OMP`, `TBB`, `CPP`, and `HIP`. Host system options include `CPP`, `OMP`, and `TBB`.

### Removed

* Removed the `GenerateResourceSpec.cmake` script - it is replaced by the added `generate_resource_spec.cpp` code above.

## rocThrust 4.2.0 for ROCm 7.2

### Added

* Added `thrust::unique_ptr` - a smart pointer for managing device memory with automatic cleanup.
* Added a new cmake option, `BUILD_OFFLOAD_COMPRESS`. When rocThrust is build with this option enabled, the `--offload-compress` switch is passed to the compiler. This causes the compiler to compress the binary that it generates. Compression can be useful in cases where you are compiling for a large number of targets, since this often results in a large binary. Without compression, in some cases, the generated binary may become so large symbols are placed out of range, resulting in linking errors. The new `BUILD_OFFLOAD_COMPRESS` option is set to `ON` by default.

## rocThrust 4.1.0 for ROCm 7.1

### Added

* Added a new CMake option `-DSQLITE_USE_SYSTEM_PACKAGE` to allow SQLite to be provided by the system.
* Introduced `libhipcxx` as a soft depedency. When `liphipcxx` can be included, rocthrust, may use structs and methods defined in `libhipcxx`. This allows for a more complete behaviour parity with CCCL and mirrors CCCL's thrust own depedency on `libcudacxx`.
* Added a new CMake option `-DUSE_SYSTEM_LIB` to allow tests to be built from `ROCm` libraries provided by the system.

### Resolved issues

* Fixed an issue where the test `test_scan_by_key.inclusive.hip` failed when performing an "in-place" inclusive scan by reusing "keys" as output, by adding a buffer to store the last keys of each block (excluding the last block). Changes were made in rocprim. This fix only affects the specific case of reusing "keys" as output in an inclusive scan, and does not affect other cases.

### Known Issues

* `event` test is failing on CI and local runs on MI300, MI250 and MI210.

* rocThrust, as well as its dependencies rocPRIM and rocRAND have been moved into the new rocm-libraries "monorepo" repository (https://github.com/ROCm/rocm-libraries). This repository contains a number of ROCm libraries that are frequently used together.
  * The repository migration requires a few changes to the way that rocThrust's ROCm library dependencies are fetched.
  * There are new cmake options for obtaining rocPRIM and (optionally, if BUILD_BENCHMARKS is enabled) rocRAND.
  * cmake build options `ROCPRIM_FETCH_METHOD` and `ROCRAND_FETCH_METHOD` may be set to one of the following:
    * `PACKAGE` - (default) searches for a preinstalled packaged version of the dependency. If it is not found, the build will fall back using option `DOWNLOAD`, described below.
    * `DOWNLOAD` - downloads the dependency from the rocm-libraries repository. If git >= 2.25 is present, this option uses a sparse checkout that avoids downloading more than it needs to. If not, the whole monorepo is downloaded (this may take some time).
    * `MONOREPO` - this options is intended to be used if you are building rocThrust from within a copy of the rocm-libraries repository that you have cloned (and therefore already contains the dependencies rocPRIM and rocRAND). When selected, the build will try find the dependency in the local repository tree. If it cannot be found, the build will attempt to add it to the local tree using a sparse-checkout. If that also fails, it will fall back to using the `DOWNLOAD` option.

### Changed

* The previously hidden cmake build option `FORCE_DEPENDENCIES_DOWNLOAD` has been unhidden and renamed `EXTERNAL_DEPS_FORCE_DOWNLOAD` to differentiate it from the new rocPRIM and rocRAND dependency options described above. It's behaviour remains the same - it forces non-ROCm dependencies (Google Benchmark, Google Test, and SQLite) to be downloaded instead of searching for existing installed packages. This option defaults to `OFF`.

### Removed

* The previous dependency-related build options `DOWNLOAD_ROCPRIM` and `DOWNLOAD_ROCRAND` have been removed. Please use `ROCPRIM_FETCH_METHOD=DOWNLOAD` and `ROCRAND_FETCH_METHOD=DOWNLOAD` instead.

## rocThrust 4.0.0 for ROCm 7.0

### Changed

* Updated the required version of Google Benchmark from 1.8.0 to 1.9.0.
* Drop `c++14` support for rocthrust.
* Renamed `cpp14_required.h` to `cpp_version_check.h`
* Refactored `test_header.hpp` into separte modules `test_param_fixtures.hpp`, `test_real_assertions.hpp`, `test_imag_assertions.hpp`, and `test_utils.hpp`.
  * This is done to prevent unit tests from having access to modules that they're not testing. This will improve the accuracy of code coverage reports.

### Added
* Additional unit tests for:
  * binary_search
  * complex
  * c99math
  * catrig
  * ccosh
  * cexp
  * clog
  * csin
  * csqrt
  * ctan
* Added `test_param_fixtures.hpp` to store all the parameters for typed test suites.
* Added `test_real_assertions.hpp` to handle unit test assertions for real numbers.
* Added `test_imag_assertions.hpp` to handle unit test assertions for imaginary numbers.
* `clang++` is now used to compile google benchmarks on Windows.
* Added gfx950 support.
* Merged changes from upstream CCCL/thrust 2.6.0

### Removed

* `device_malloc_allocator.h` has been removed. This header file was unused and should not impact users.
* Removed C++14 support, only C++17 is supported.
* `test_header.hpp` has been removed. The `HIP_CHECK` function, as well as the `test` and `inter_run_bwr` namespaces, have been moved to `test_utils.hpp`.
* `test_assertions.hpp` has been split into `test_real_assertions.hpp` and `test_imag_assertions.hpp`.

### Upcoming changes

* `thrust::device_malloc_allocator` is deprecated as of this version. It will be removed in an upcoming version.

### Resolved issues

* Fixed an issue with internal calls to unqualified `distance()` which would be ambigious due to also visibile implementation through ADL.

### Known Issues
* The order of the values being compared by thrust::exclusive_scan_by_key and thrust::inclusive_scan_by_key can change between runs when integers are being compared. This can cause incorrect output when a non-commutative operator such as division is being used.

## rocThrust 3.3.0 for ROCm 6.4

### Added

* Added a section to install Thread Building Block (TBB) inside `cmake/Dependencies.cmake` if TBB is not already available.
* Made Thread Building Block (TBB) an optional dependency with the new `BUILD_HIPSTDPAR_TEST_WITH_TBB` flag, default is `OFF`. When the flag is `OFF` and TBB is not already on the machine it will compile without TBB. Otherwise is will compile it with TBB.
* Added extended tests to `rtest.py`. These tests are extra tests that did not fit the criteria of smoke and regression tests. These tests will take much longer to run relative to smoke and regression tests. Use `python rtest.py [--emulation|-e|--test|-t]=extended` to run these tests.
* Added regression tests to `rtest.py`. These tests recreate scenarios that have caused hardware problems in past emulation environments. Use `python rtest.py [--emulation|-e|--test|-t]=regression` to run these tests.
* Added smoke test options, which runs a subset of the unit tests and ensures that less than 2gb of VRAM will be used. Use `python rtest.py [--emulation|-e|--test|-t]=smoke` to run these tests.
* Added `--emulation` option for `rtest.py`
* Merged changes from upstream CCCL/thrust 2.4.0
* Merged changes from upstream CCCL/thrust 2.5.0
* Added `find_first_of` to HIPSTDPAR
* Added `search` and `find_end` to HIPSTDPAR
* Added `search_n` to HIPSTDPAR
* Updated HIPSTDPAR's `adjacent_find` to use rocPRIM's implementation

### Changed

* Changed the C++ version from 14 to 17. C++14 will be deprecated in the next major release.
* `--test|-t` is no longer a required flag for `rtest.py`. Instead, the user can use either `--emulation|-e` or `--test|-t`, but not both.
* Split the contents of HIPSTDPAR's forwarding header into several implementation headers.
* Fixed `copy_if` to work with large data types (512 bytes)

### Known Issues
*  `thrust::inclusive_scan_by_key` might produce incorrect results when it's used with -O2 or -O3 optimization.
  - The error is caused by a recent compiler change. There is a fix available that will be released at a later date.

## rocThrust 3.2.0 for ROCm 6.3

### Added

* Merged changes from upstream CCCL/thrust 2.3.2
  * Only the NVIDIA backend uses `tuple` and `pair` types from libcu++, other backends continue to
    use the original Thrust implementations and hence do not require libcu++ (CCCL) as a dependency.
* Added the `thrust::hip::par_det` execution policy to enable bitwise reproducibility on algorithms that are not bitwise reproducible by default.

### Changed

* Updated the default value for the `-a` argument from `rmake.py` to `gfx906:xnack-,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201`.
* Enabled the upstream (thrust) test suite for execution by default. It can still be disabled by CMake option `-DENABLE_UPSTREAM_TESTS=OFF`.

### Resolved issues

* Fixed an issue in `rmake.py` where the list storing cmake options would contain individual characters instead of a full string of options.
* Fixed the HIP backend not passing `TestCopyIfNonTrivial` from the upstream (thrust) test suite.
* Fixed tests failing when compiled with `-D_GLIBCXX_ASSERTIONS=ON`.

## rocThrust 3.1.0 for ROCm 6.2

### Additions

* Merged changes from upstream CCCL/thrust 2.2.0
  * Updated the contents of `system/hip` and `test` with the upstream changes to `system/cuda` and `testing`
* Added HIPSTDPAR library as part of rocThrust.

### Changes

* Updated internal calls to `rocprim::detail::invoke_result` to use the public API `rocprim::invoke_result`.
* Use `rocprim::device_adjacent_difference` for `adjacent_difference` API call.
* Updated internal use of custom iterator in `thrust::detail::unique_by_key` to use rocPRIM's `rocprim::unique_by_key`.
* Updated `adjecent_difference` to make use of `rocprim:adjecent_difference` when iterators are comparable and not equal otherwise use `rocprim:adjacent_difference_inplace`.

### Fixes
* Fixed incorrect implementation of `thrust::optional<T&>::emplace()`.

### Known issues
* `thrust::reduce_by_key` outputs are not bit-wise reproducible, as run-to-run results for pseudo-associative reduction operators (e.g. floating-point arithmetic operators) are not deterministic on the same device.
* Note that currently, rocThrust memory allocation is performed in such a way that most algorithmic API functions cannot be called from within hipGraphs.

## rocThrust 3.0.0 for ROCm 6.0

### Additions

* Updated to match upstream Thrust 2.0.1
* NV_IF_TARGET macro from libcu++ for NVIDIA backend and HIP implementation for HIP backend.

### Changes

* The cmake build system now additionally accepts `GPU_TARGETS` in addition to `AMDGPU_TARGETS` for
  setting the targeted gpu architectures. `GPU_TARGETS=all` will compile for all supported architectures.
  `AMDGPU_TARGETS` is only provided for backwards compatibility, `GPU_TARGETS` should be preferred.
* Removed cub symlink from the root of the repository.
* Removed support for deprecated macros (THRUST_DEVICE_BACKEND and THRUST_HOST_BACKEND).

### Fixes
* Fixed a segmentation fault when binary search / upper bound / lower bound / equal range was invoked with `hip_rocprim::execute_on_stream_base` policy.

### Known issues

* The `THRUST_HAS_CUDART` macro, which is no longer used in Thrust (it's provided only for legacy
  support) is replaced with `NV_IF_TARGET` and `THRUST_RDC_ENABLED` in the NVIDIA backend. The
  HIP backend doesn't have a `THRUST_RDC_ENABLED` macro, so some branches in Thrust code may
  be unreachable in the HIP backend.

## rocThrust 2.18.0 for ROCm 5.7

### Fixes
* `lower_bound`, `upper_bound`, and `binary_search` failed to compile for certain types.
* Fixed issue where `transform_iterator` would not compile with `__device__`-only operators.

### Changes
* Updated `docs` directory structure to match the standard of [rocm-docs-core](https://github.com/RadeonOpenCompute/rocm-docs-core).
* Removed references to and workarounds for deprecated hcc

## rocThrust 2.17.0 for ROCm 5.5

### Additions

* Updates to match upstream Thrust 1.17.2

### Changes

* `partition_copy` now uses `rocprim::partition_two_way` for increased performance

### Fixes

* `set_difference` and `set_intersection` no longer hang if the number of items is above `UINT_MAX`
  (the unit tests for `set_difference` and `set_intersection` used to fail the
  `TestSetDifferenceWithBigIndexes`)

## rocThrust 2.16.0 for ROCm 5.3

### Additions

* Updates to match upstream Thrust 1.16.0

### Changes

* rocThrust functionality dependent on device malloc is functional (ROCm 5.2 reenabled device malloc); you can now use device launched `thrust::sort` and `thrust::sort_by_key`

## rocThrust 2.15.0 for ROCm 5.2

### Additions

* Packages for tests and benchmark executables on all supported operating systems using CPack

### Known issues

* `async_copy`, `partition`, and `stable_sort_by_key` unit tests are failing for HIP on Windows

## rocThrust 2.14.0 for ROCm 5.1

### Additions

* Updates to match upstream Thrust 1.15.0

### Known issues

* `async_copy`, `partition`, and `stable_sort_by_key` unit tests are failing for HIP on Windows

## rocThrust 2.13.0 for ROCm 5.0

### Changes

* Updates to match upstream Thrust 1.13.0
* Updates to match upstream Thrust 1.14.0
* Added async scan
* Scan algorithms: `inclusive_scan` now uses the `input-type` as `accumulator-type`; `exclusive_scan`
  uses `initial-value-type`
  * This changes the behavior of small-size input types with large-size output types (e.g. `short` input,
    `int` output) and low-res input with high-res output (e.g. `float` input, `double` output)

## rocThrust-2.11.2 for ROCm 4.5.0

### Additions

* Initial HIP on Windows support

### Changes

* Packaging has changed to a development package (called `rocthrust-dev` for `.deb` packages and
  `rocthrust-devel` for `.rpm` packages). Because rocThrust is a header-only library, there is no runtime package. To aid in the transition, the development package sets the `provides` field to `rocthrust`, so that existing packages that are dependent on rocThrust can continue to work. This `provides` feature is introduced as a deprecated feature because it will be removed in a future ROCm release.

### Known issues

* `async_copy`, `partition`, and `stable_sort_by_key` unit tests are failing for HIP on Windows
* Mixed-type exclusive scan algorithm is not using the initial value type for the results type

## [rocThrust-2.11.1 for ROCm 4.4.0]

### Additions

* gfx1030 support
* AddressSanitizer build option

### Fixes

* async_transform unit test failure

## [rocThrust-2.11.0 for ROCm 4.3.0]

### Additions

* Updates to match upstream Thrust 1.11
* gfx90a support
* gfx803 support re-enabled

## [rocThrust-2.10.9 for ROCm 4.2.0]

### Additions

* Updates to match upstream Thrust 1.10

### Changes

* rocThrust now requires CMake version 3.10.2 or greater

### Fixes

* Size zero inputs are now properly handled with newer ROCm builds, which no longer allow zero-size
  kernel grid/block dimensions
* Warning of unused results

## [rocThrust-2.10.8 for ROCm 4.1.0]

* There are no changes with this release

## [rocThrust-2.10.7 for ROCm 4.0.0]

### Additions

* Updated to upstream Thrust 1.10.0
* Implemented runtime error for unsupported algorithms and disabled respective tests
* Updated CMake to use downloaded rocPRIM

## [rocThrust-2.10.6 for ROCm 3.10]

### Additions

* `copy_if` on device test case

### Known issues

* We've disabled ROCm support for device malloc. As a result, rocThrust functionality dependent on
  device malloc does not work--avoid using device launched `thrust::sort` and `thrust::sort_by_key`. Note
  that Host launched functionality is not impacted.
  * A partial enablement of device malloc is possible by setting `HIP_ENABLE_DEVICE_MALLOC` to 1.
  * `thrust::sort` and `thrust::sort_by_key` may work on certain input sizes but we don't recommended
    this for production code.

## [rocThrust-2.10.5 for ROCm 3.9.0]

### Additions

* Updated to upstream Thrust 1.9.8
* New test cases for device-side algorithms

### Fixes

* Bug for binary search
* Implemented workarounds for `hipStreamDefault` hang

### Known issues

* We've disabled ROCm support for device malloc. As a result, rocThrust functionality dependent on
  device malloc does not work--avoid using device launched `thrust::sort` and `thrust::sort_by_key`. Note
  that Host launched functionality is not impacted.
  * A partial enablement of device malloc is possible by setting `HIP_ENABLE_DEVICE_MALLOC` to 1.
  * `thrust::sort` and `thrust::sort_by_key` may work on certain input sizes but we don't recommended
    this for production code.

## [rocThrust-2.10.4 for ROCm 3.8.0]

### Known issues

* We've disabled ROCm support for device malloc. As a result, rocThrust functionality dependent on
  device malloc does not work--avoid using device launched `thrust::sort` and `thrust::sort_by_key`. Note
  that Host launched functionality is not impacted.
  * A partial enablement of device malloc is possible by setting `HIP_ENABLE_DEVICE_MALLOC` to 1.
  * `thrust::sort` and `thrust::sort_by_key` may work on certain input sizes but we don't recommended
    this for production code.

## [rocThrust-2.10.3 for ROCm 3.7.0]

### Additions

* Updated to upstream Thrust 1.9.4

### Changes

* Package dependency has changed to rocPRIM only

### Known issues

* We've disabled ROCm support for device malloc. As a result, rocThrust functionality dependent on
  device malloc does not work--avoid using device launched `thrust::sort` and `thrust::sort_by_key`. Note
  that Host launched functionality is not impacted.
  * A partial enablement of device malloc is possible by setting `HIP_ENABLE_DEVICE_MALLOC` to 1.
  * `thrust::sort` and `thrust::sort_by_key` may work on certain input sizes but we don't recommended
    this for production code.

## [rocThrust-2.10.2 for ROCm 3.6.0]

### Known issues

* We've disabled ROCm support for device malloc. As a result, rocThrust functionality dependent on
  device malloc does not work--avoid using device launched `thrust::sort` and `thrust::sort_by_key`. Note
  that Host launched functionality is not impacted.
  * A partial enablement of device malloc is possible by setting `HIP_ENABLE_DEVICE_MALLOC` to 1.
  * `thrust::sort` and `thrust::sort_by_key` may work on certain input sizes but we don't recommended
    this for production code.

## [rocThrust-2.10.1 for ROCm 3.5.0]

### Additions

* Improved tests with fixed and random seeds for test data

### Changes

* CMake searches for rocThrust locally first; if it isn't found, CMake downloads it from GitHub

### Deprecations

* HCC build has been deprecated