derek.holloway
7502bf64a4
2.4 KiB
Hacking and Debugging RocSHMEM
This documentation is mostly for core RocSHMEM developers and contributors. Power users may still find it useful.
How to debug parallel programs
When using Open MPI as the launch mechanism, you can use OMPI_MCA_mpi_abort_delay=-1 to keep parallel processes active after a crash. You can then use ssh -t $nodename rocgdb -p $pid to connect a rocgdb to the failed process. Sometimes errors are caught by UCX, UCX_HANDLE_ERRORS=freeze will have the same effect in such cases.
Look into scripts/functional_tests/GDB_README about an alternative technique that deploys multiple xterm to gdb into parallel processes.
How to use the address sanitizer (ASAN)
Refer to General documentation for ASAN on AMD GPUs.
Compiling with ASAN
If this is a fresh build directory, simply add -DASAN=ON to the cmake invocation.
cmake . <...> -DASAN=ON
If you are enabling ASAN in a previously used build directory, use ccmake to alter the CMake Cache
ccmake .
In the ccmake interface:
- find and toggle
ASANON - find and delete
COMPILING_TARGETS(keybindd)
Do not forget to delete COMPILING_TARGETS again when disabling ASAN (otherwise xnack will remain required, causing failure in production runs).
Running with ASAN
You need to set environment variable HSA_XNACK=1. Do not forget to unset this variable when not using ASAN (it will impact performance).
You may need to add the path to libclang_rt.asan-x86_64.so into LD_LIBRARY_PATH by hand. Depending on the ROCm version, it may be in an unusual place, e.g., $ROCM_ROOT/lib/llvm/lib/clang/19/lib/linux/libclang_rt.asan-x86_64.so; find /opt/rocm -name libclang_rt.asan-x86_64.so may be required to find it.
ASAN may crash when using Open MPI. If that happens, set environment variable OMPI_MCA_memory=^patcher. Do not forget to unset this variable when not using ASAN (it will impact performance).
When running the program, the behavior of ASAN can be controlled with the ASAN_OPTIONS environment variable.
you can redude the amount of spurious leak reports by setting environment variable LSAN_OPTIONS=suppressions=$ROCSHEM_SRC/scripts/lsan-suppressions.txt.