derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
118 lines
6.8 KiB
Markdown
118 lines
6.8 KiB
Markdown
Container for rocSHMEM development with MPI support
|
|
===================================================
|
|
|
|
* Source: https://github.com/ROCm/rocm-systems/projects/rocshmem
|
|
* Documentation: https://rocm.docs.amd.com/projects/rocSHMEM/en/latest/docker
|
|
* Contact: rocshmem-maintainers@amd.com
|
|
|
|
Building the container
|
|
----------------------
|
|
|
|
```bash
|
|
# Run from the projects/rocshmem directory:
|
|
docker build -f docker/Dockerfile.ubuntu --tag $USER/rocshmem docker/
|
|
```
|
|
|
|
Build parameters:
|
|
|
|
* Build a PR branch AND devel in the container: `docker build -f docker/Dockerfile.ubuntu --build-arg PR_NUM=4574 --tag $USER/rocshmem-pr4574 docker/`
|
|
* Use a different SSH port: `docker build -f docker/Dockerfile.ubuntu --build-arg SSH_PORT=2223 --tag $USER/rocshmem docker/`
|
|
|
|
Moving the image to compute nodes:
|
|
|
|
`for node in node01 node02; do docker save $USER/rocshmem | ssh $node docker load; done`
|
|
|
|
Adding DeepEP/VLLM
|
|
------------------
|
|
|
|
This dockerfile does not contain, but can easily be adapted to build DeepEP/VLLM.
|
|
However, we recommend following up-to-date instructions from https://github.com/ROCm/DeepEP?tab=readme-ov-file#development.
|
|
|
|
If you still want to adapt this dockerfile:
|
|
1. Change the FROM to use a rocm/pytorch base, and uncomment OPTIONAL dependencies.
|
|
2. ADD rocm/deepep and RUN setup.py to build it in the docker.
|
|
|
|
How to run examples
|
|
-------------------
|
|
|
|
1. Sanity check: print IB/RoCE info, notably firmware version
|
|
|
|
```bash
|
|
docker run -it --rm --shm-size 64G --network host --device /dev/dri --device /dev/kfd --device /dev/infiniband --ipc host --group-add video --cap-add SYS_PTRACE --security-opt seccomp=unconfined --privileged \
|
|
# Broadcom Thor2 specific options to import the host bnxt driver libraries (note: Thor2 bnxt GDA requires specific firmware version)
|
|
-v /usr/local/lib/libbnxt_re-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libbnxt_re-rdmav34.so:ro -v /usr/local/lib/libbnxt_re.so:/usr/local/lib/libbnxt_re.so:ro \
|
|
# Pensando AMD Pollara specific options to import host ionic driver libraries (note: Pollara ionic GDA requires specific firmware version)
|
|
-v /sys/class/infiniband:/sys/class/infiniband:ro -v /usr/lib/x86_64-linux-gnu/libionic.so.1:/usr/lib/x86_64-linux-gnu/libionic.so.1:ro -v /usr/lib/x86_64-linux-gnu/libionic.so:/usr/lib/x86_64-linux-gnu/libionic.so:ro \
|
|
-v /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:ro \
|
|
-v /etc/libibverbs.d/ionic.driver:/etc/libibverbs.d/ionic.driver:ro \
|
|
$USER/rocshmem ibv_devinfo
|
|
```
|
|
|
|
2. Sanity checks, print hardware as seen by rocSHMEM
|
|
|
|
```bash
|
|
docker run -it --rm --shm-size 64G --network host --device /dev/dri --device /dev/kfd --device /dev/infiniband --ipc host --group-add video --cap-add SYS_PTRACE --security-opt seccomp=unconfined --privileged \
|
|
# Broadcom Thor2 options to import host driver libraries (note: Thor2 bnxt GDA requires specific firmware version)
|
|
-v /usr/local/lib/libbnxt_re-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libbnxt_re-rdmav34.so:ro -v /usr/local/lib/libbnxt_re.so:/usr/local/lib/libbnxt_re.so:ro \
|
|
# Pensando AMD Pollara options to import host driver libraries (note: Pollara ionic GDA requires specific firmware version)
|
|
-v /sys/class/infiniband:/sys/class/infiniband:ro -v /usr/lib/x86_64-linux-gnu/libionic.so.1:/usr/lib/x86_64-linux-gnu/libionic.so.1:ro -v /usr/lib/x86_64-linux-gnu/libionic.so:/usr/lib/x86_64-linux-gnu/libionic.so:ro \
|
|
-v /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:ro \
|
|
-v /etc/libibverbs.d/ionic.driver:/etc/libibverbs.d/ionic.driver:ro \
|
|
$USER/rocshmem tools/rocshmem_info
|
|
```
|
|
|
|
3. Launch persistent containers (you can use srun, pdsh, ssh, kubernetes, etc to deploy the containers, we show an example with srun here)
|
|
|
|
```bash
|
|
srun -N2 --ntasks-per-node=1 --gpus=8 \
|
|
docker run -d --rm \
|
|
--shm-size 64G --network host --device /dev/dri --device /dev/kfd --device /dev/infiniband --ipc host --group-add video --cap-add SYS_PTRACE --security-opt seccomp=unconfined --privileged
|
|
# Broadcom Thor2 options to import host driver libraries (note: Thor2 bnxt GDA requires specific firmware version)
|
|
-v /usr/local/lib/libbnxt_re-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libbnxt_re-rdmav34.so:ro -v /usr/local/lib/libbnxt_re.so:/usr/local/lib/libbnxt_re.so:ro \
|
|
# Pensando AMD Pollara options to import host driver libraries (note: Pollara ionic GDA requires specific firmware version)
|
|
-v /sys/class/infiniband:/sys/class/infiniband:ro -v /usr/lib/x86_64-linux-gnu/libionic.so.1:/usr/lib/x86_64-linux-gnu/libionic.so.1:ro -v /usr/lib/x86_64-linux-gnu/libionic.so:/usr/lib/x86_64-linux-gnu/libionic.so:ro \
|
|
-v /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:ro \
|
|
-v /etc/libibverbs.d/ionic.driver:/etc/libibverbs.d/ionic.driver:ro \
|
|
--name $USER-rocshmem $USER/rocshmem \
|
|
# Default command will run rocshmem_info, then launch SSHD, lets print the outcome.
|
|
&& sleep 5 && srun -N 2 --ntasks-per-node=1 -l docker logs $USER-rocshmem
|
|
```
|
|
|
|
4. Run a simple rocshmem_waveput_nbi test with 16 Workgroups, 256 threads per WG in the persistent containers
|
|
|
|
`srun -n1 docker exec -it -e ROCSHMEM_DEBUG_LEVEL=ENV $USER-rocshmem mpiexec --report-bindings -n 2 tests/functional_tests/rocshmem_functional_tests -a 31 -w 16 -z 256`
|
|
|
|
5. Run the full rocSHMEM testsuite in the persistent containers
|
|
|
|
`srun -n1 docker exec -it $USER-rocshmem bash tests/rocshmem_functional_driver.sh /app/build/tests/functional_tests/rocshmem_functional_tests gda logs`
|
|
|
|
Performance comparison
|
|
----------------------
|
|
|
|
Build image with a PR branch and develop, then run comparison with one command:
|
|
|
|
```bash
|
|
docker build --build-arg PR_NUM=4574 --tag $USER/rocshmem-pr4574 \
|
|
projects/rocshmem/docker/
|
|
|
|
mkdir -p pr4574-results
|
|
docker run --rm \
|
|
--shm-size 64G --network host --device /dev/dri --device /dev/kfd \
|
|
--ipc host --group-add video --cap-add SYS_PTRACE \
|
|
--security-opt seccomp=unconfined --privileged \
|
|
-v "$(pwd)/pr4574-results:/results" \
|
|
$USER/rocshmem-pr4574 perf-compare --suite heatmap --iterations 5
|
|
# Plots: ./pr4574-results/heatmap_summary.png
|
|
```
|
|
|
|
See scripts/functional_tests/README-perf_compare.md for all options.
|
|
|
|
Important directories
|
|
---------------------
|
|
|
|
1. `/app/build1234`: if `--build-arg PR_NUM=1234` is provided, this is a build of the selected PR branch. This is the default working directory.
|
|
2. `/app/build`: this is a build of the _develop_ branch.
|
|
3. `/app/rocm-systems/projects/rocshmem`: source directory, it is a git repo, you may fetch, checkout and created new build directories as needed.
|
|
4. `/root/rocshmem`: the default install directory, if you build DeepEP it will pull rocshmem from that location.
|
|
5. `/app/ompi`: build of Open MPI with UCX support as required to run RO backend.
|