TheRock/rocm-libraries/projects/hipdnn/dockerfiles/README.md

hipDNN Docker Environments

This directory contains the Dockerfile for building the hipDNN development environment Docker image.

📋 Prerequisites

• Docker with Buildx installed (Version 27.5.1+ recommended)
  • Why? This Dockerfile relies on BuildKit (Buildx) to selectively build only the necessary stages. Legacy builders or older versions may attempt to build all stages—including those with missing arguments—causing build failures.
• ROCm-compatible GPU
  • Required for running applications with GPU support.
• Sufficient disk space
  • Required for storing development Docker images.

🐳 Ubuntu 24.04

• File: Dockerfile.ubuntu24
• Base Image: Ubuntu 24.04 LTS
• Purpose: Primary development environment for hipDNN
• Includes:
  • ROCm development tools
  • CMake build system
  • Google Test framework
  • Ninja build tool

🔨 Building the Docker Image

The Dockerfile supports two build types: prebuilt (using nightly tarballs) and fullbuild (building from source).

Build Arguments

🔧Select Build Type

Argument	Default	Description	Valid Values
BUILD_TYPE	prebuilt	Selects build method	prebuilt, fullbuild

📦 Prebuilt-Only Arguments

Note

Prebuilt mode downloads pre-compiled binaries from TheRock nightly builds using install_rocm_from_artifacts.py (much faster than building from source)

Note

There is currently an issue with using the prebuilt binaries with the gfx90X ASIC family. Refer to this GitHub issue for more details: https://github.com/ROCm/TheRock/issues/2179

Argument	Default	Description
THEROCK_RELEASE	latest	Release version to install. Use latest to automatically fetch the newest nightly build, or specify a version like 7.12.0a20260202. Available versions can be found at TheRock nightly tarballs.
THEROCK_ASIC	gfx94X	GPU architecture family prefix. Combined with THEROCK_ASIC_VARIANT to form the artifact group.
THEROCK_ASIC_VARIANT	dcgpu	GPU variant suffix (e.g., dcgpu, all, dgpu). Combined with THEROCK_ASIC to form the artifact group.
THEROCK_ARTIFACT_GROUP	$THEROCK_ASIC-$THEROCK_ASIC_VARIANT	Full artifact group override. Available groups: gfx90X-dcgpu, gfx94X-dcgpu, gfx950-dcgpu, gfx110X-all, gfx110X-dgpu, gfx120X-all, etc. See TheRock releases for the full list.

Version Logging

The prebuilt stage writes the installed TheRock version to /opt/rocm/THEROCK_VERSION inside the image. For pinned releases, this contains the exact version string. For latest builds, it captures version information from the install output.

⚠️ Deprecated Prebuilt Arguments

Caution

The following build args are deprecated and will be removed in a future release. When any of these args are set, the build uses the original wget/tar download method instead of install_rocm_from_artifacts.py. A deprecation warning is printed during the build.

Deprecated Argument	Replacement	Behavior
THEROCK_TARBALL	THEROCK_ARTIFACT_GROUP + THEROCK_RELEASE	Used as-is as the tarball filename for wget download
THEROCK_PREBUILT_ID	THEROCK_ARTIFACT_GROUP + THEROCK_RELEASE	Prefixed with therock-dist-linux- and suffixed with .tar.gz for wget download
THEROCK_GIT_TAG	THEROCK_RELEASE	Combined with THEROCK_ARTIFACT_GROUP to construct the tarball filename

Priority when multiple legacy args are set: THEROCK_TARBALL > THEROCK_PREBUILT_ID > THEROCK_GIT_TAG.

Migration examples:

# Old: --build-arg THEROCK_GIT_TAG=7.0.0rc20250909
# New:
--build-arg THEROCK_RELEASE=7.0.0rc20250909

# Old: --build-arg THEROCK_PREBUILT_ID=gfx94X-dcgpu-7.0.0rc20250909
# New:
--build-arg THEROCK_ARTIFACT_GROUP=gfx94X-dcgpu --build-arg THEROCK_RELEASE=7.0.0rc20250909

# Old: --build-arg THEROCK_TARBALL=therock-dist-linux-gfx94X-dcgpu-7.0.0rc20250909.tar.gz
# New:
--build-arg THEROCK_ARTIFACT_GROUP=gfx94X-dcgpu --build-arg THEROCK_RELEASE=7.0.0rc20250909

🏗️ Fullbuild-Only Arguments

Note

Fullbuild mode clones and compiles TheRock from source (will take several hours to complete but more flexible)

Argument	Default	Description
THEROCK_GIT_HASH	default	Specific git commit hash to checkout (uses default branch if not specified).
ROCM_LIBRARIES_REF	default	Specific git commit hash for rocm-libraries submodule to checkout (uses default branch if not specified).
THEROCK_ASIC	gfx94X	GPU architecture target. The values for THEROCK_ASIC for fullbuild mode can be found in the LLVM Target column of the Supported GPU table here.
THEROCK_BUILD_MODE	Release	Build mode: Preset (uses TheRock presets), Debug, or Release (uses CMake build types).
THEROCK_BUILD_PRESET	linux-release-package	Specify which build preset to use when THEROCK_BUILD_MODE=Preset.
BUILD_JOBS	0	Number of parallel build jobs. 0 uses all available CPU cores.

Build Examples

📦 Prebuilt Mode (Recommended)

Default prebuilt (latest nightly for gfx94X-dcgpu):

docker build -f Dockerfile.ubuntu24 -t hipdnn:prebuilt .

This will automatically download and install the latest nightly build for gfx94X-dcgpu.

Specific release version:

docker build -f Dockerfile.ubuntu24 \
    --build-arg THEROCK_RELEASE=7.12.0a20260202 \
    -t hipdnn:prebuilt .

Different ASIC family (MI100/MI200 series):

docker build -f Dockerfile.ubuntu24 \
    --build-arg THEROCK_ASIC=gfx90X \
    --build-arg THEROCK_ASIC_VARIANT=dcgpu \
    -t hipdnn:prebuilt_gfx90X .

Full artifact group override (RDNA3):

docker build -f Dockerfile.ubuntu24 \
    --build-arg THEROCK_ARTIFACT_GROUP=gfx110X-all \
    -t hipdnn:prebuilt_rdna3 .

🏗️ Fullbuild Mode

Note that the full build can take several hours to complete.

Default fullbuild (gfx94X, default branch):

docker build -f Dockerfile.ubuntu24 --build-arg BUILD_TYPE=fullbuild -t hipdnn:fullbuild .

Debug build (gfx94X, debug mode):

docker build -f Dockerfile.ubuntu24 --build-arg BUILD_TYPE=fullbuild --build-arg THEROCK_BUILD_MODE=Debug -t hipdnn:debug .

Release build with limited cores (gfx94X, release mode, 4 cores):

docker build -f Dockerfile.ubuntu24 --build-arg BUILD_TYPE=fullbuild --build-arg THEROCK_BUILD_MODE=Release --build-arg BUILD_JOBS=4 -t hipdnn:release .

Custom preset with all cores (gfx94X, custom preset):

docker build -f Dockerfile.ubuntu24 --build-arg BUILD_TYPE=fullbuild --build-arg THEROCK_BUILD_MODE=Preset --build-arg THEROCK_BUILD_PRESET=linux-debug-package -t hipdnn:custom-preset .

Custom ASIC and Git Hash (gfx950, hash abcd1234):

docker build -f Dockerfile.ubuntu24 --build-arg BUILD_TYPE=fullbuild --build-arg THEROCK_ASIC=gfx950 --build-arg THEROCK_GIT_HASH=abcd1234 -t hipdnn:custom_source_build .

🚀 Running the Containers

Basic Run Command

To run the hipDNN docker environment with full GPU support:

docker run -it \
  -v /path/to/hipdnn/repo:/workspace/hipdnn \
  --privileged \
  --rm \
  --device=/dev/kfd \
  --device /dev/dri:/dev/dri:rw \
  --volume /dev/dri:/dev/dri:rw \
  -v /var/lib/docker/:/var/lib/docker \
  --group-add video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  <image-name>

Command Options Explained

Option	Purpose
-it	Interactive terminal
-v /path/to/hipdnn/repo:/workspace/hipdnn	Mount hipDNN source code
--privileged	Required for GPU access
--rm	Remove container after exit
--device=/dev/kfd	AMD GPU compute device
--device /dev/dri:/dev/dri:rw	Direct Rendering Infrastructure for GPU
--group-add video	Add container to video group for GPU access
--cap-add=SYS_PTRACE	Enable debugging capabilities
--security-opt seccomp=unconfined	Required for some debugging tools
<image-name>	The tag added to the image with the -t option in the build command

Example Run Commands

1. Switch to the rocm-libraries/projects/hipdnn folder.
2. Using image tagged hipdnn:prebuilt:

docker run -it \
  -v $(pwd):/workspace/hipdnn \
  --privileged \
  --rm \
  --device=/dev/kfd \
  --device /dev/dri:/dev/dri:rw \
  --group-add video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  hipdnn:prebuilt

Building hipDNN

Follow the quick start steps in the build guide to build hipDNN.

💡 Tips and Best Practices

1. Volume Mounting: Mount your hipDNN source code directory to persist changes:

   -v /absolute/path/to/hipdnn:/workspace/hipdnn
   

2. GPU Verification: Once inside the container, verify GPU access:

   amd-smi
   

   or

   rocminfo
   

3. Development Workflow: The containers include all necessary development tools, so you can edit code on your host machine and build/test inside the container.

⚠️ Troubleshooting

GPU Not Detected

• Ensure ROCm is properly installed on the host system
• Verify that /dev/kfd and /dev/dri exist on the host
• Check that your user is in the video and render groups

Permission Issues

• Make sure to use the --privileged flag
• Verify that the mounted directories have appropriate permissions

Build Failures

• Ensure all submodules are initialized: git submodule update --init --recursive
• Check that the ROCm version in the container matches your GPU requirements

Fullbuild not updating to latest version of TheRock

• Since the docker build doesn't change when the cloned source is updated, you will need to either provide a new hash for the docker build to rebuild --build-arg THEROCK_GIT_HASH=abcd1234, or provide --no-cache option when building to force rebuild

Build issues

Build fails attempting to build all stages / Argument mismatches:

• Symptom: The build attempts to execute stages you didn't select (e.g., compiling from source in fullbuild when you requested prebuilt), often failing due to default arguments or configuration conflicts in those unselected stages.
• Cause: You are likely using the legacy Docker builder instead of BuildKit (Buildx). The legacy builder attempts to process all stages, whereas BuildKit only builds what is necessary for the target.
• Solution:
  • Ensure Docker Buildx is installed and enabled (docker buildx version).
  • Upgrade Docker to the recommended version (27.5.1+).
  • If using an older version, try explicitly enabling BuildKit: DOCKER_BUILDKIT=1 docker build ...