TheRock/docs/development/ccache_troubleshooting.md

CCache Troubleshooting

This page covers how to diagnose and fix ccache hit rate issues in TheRock CI.

Introduction

ccache is a compiler cache that speeds up recompilation by caching previous compilation results and detecting when the same compilation is being done again. TheRock uses ccache for both local developer builds and CI builds, with a shared remote cache server so that cache entries can be reused across CI runners.

sccache is an alternative compiler cache with similar goals. We are evaluating both ccache and sccache for use across ROCm and downstream frameworks like PyTorch (see build_tools/setup_sccache_rocm.py). The goal is to standardize on a cache setup that works well across our full ecosystem of projects using shared infrastructure (e.g., cache servers). This page focuses on ccache, which is the current default for TheRock CI.

Key files

File	Description
build_tools/setup_ccache.py	Generates ccache config for local and CI builds
build_tools/posix_ccache_compiler_check.py	Custom compiler fingerprinting for POSIX
build_tools/hack/ccache/	Sanity tests and analysis scripts

CI infrastructure

Remote cache servers

CI uses bazel-remote as a shared remote cache, accessed via ccache's remote_storage option with layout=bazel:

Preset	Server	Used by
github-oss-dev	bazelremote-svc.bazelremote-ns.svc.cluster.local:8080	PR builds, postsubmit
github-oss-release	bazelremote-svc-rel.bazelremote-ns.svc.cluster.local:8080	Release builds

Both servers are on the Kubernetes cluster, accessible without authentication from any pod in the cluster.

Namespace version

CCACHE_NAMESPACE_VERSION in setup_ccache.py controls the cache namespace. Bump it when making hash-affecting config changes (sloppiness, compiler_check, etc.) to isolate from stale entries. All entries in the old namespace become unreachable — the cache starts cold, so the first CI run after a bump will be all misses.

Currently a single namespace is shared across all repos, platforms, and build configurations. ccache's own hashing (compiler binary, flags, source content) partitions entries naturally, but if cross-repo or cross-platform pollution becomes an issue, per-repo or per-platform namespaces could be introduced.

Platform differences

	Linux	Windows
ccache version	Pinned in manylinux Dockerfile (4.11.2 as of 2026-05)	Installed via chocolatey in the Windows workflow (4.13.6 as of 2026-05)
compiler_check	Custom script (hashes binary + shared libs via sha256sum)	content (hashes binary content)
Build container	Same manylinux Docker image for all jobs	Ephemeral VMs with unique workspace GUIDs
Workspace path	/__w/TheRock/TheRock/ (consistent)	C:\home\runner\_work\TheRock\TheRock\ (consistent)
Build directory	/__w/TheRock/TheRock/build/	B:\build (volume mount)

Windows drive mounts

Windows CI runners use Azure VMs where B:\ is a temporary disk mounted as a volume mount point to C:\{GUID}\. The GUID is unique per VM, so B:\build resolves to different absolute paths on different runners.

The -resource-dir flag is explicitly passed to clang in the toolchain setup (cmake/therock_subproject.cmake) to prevent clang from auto-detecting its resource directory through the resolved mount path.

Cross-repo cache sharing

Multiple ROCm repos (TheRock, SPIRV-LLVM-Translator, rocm-libraries, rocm-systems, etc.) use the same setup_ccache.py and share the same bazelremote cache servers. Each repo gets a different workspace path (e.g., /_work/{repo}/{repo} on Linux, C:\home\runner\_work\{repo}\{repo} on Windows). The namespace isolates entries to some extent, but be aware of cross-repo pollution — entries from one repo may share manifests with another if they compile the same source files with compatible flags.

Release builds use the local cache only (no remote)

Prerelease (release-candidate) and nightly-bkc builds use only the local cache. The --release-type prerelease and --release-type nightly-bkc options automatically select the local config preset. Users can select the same local-only behavior explicitly with --config-preset=local.

Stable releases are a repackage of the prerelease artifacts and reading only from the local cache guards against cache poisoning attacks. The local cache is kept because some builds (notably Windows) need at least a local cache for performance and runner stability.

Expect zero remote hits in prerelease and nightly-bkc "Report" steps; local hits are expected and fine. Nightly, dev-bkc, dev, and ci builds are unaffected and continue to use a shared remote cache.

Tip

To confirm which cache a run actually used, look for the Cache mode line in the "Setup ccache" step. It is read back from the generated ccache.conf, so it reports what was written rather than only the requested options:

[setup_ccache] Cache mode: release_type=prerelease preset=local remote=disabled local=/.../ccache
[setup_ccache] Cache mode: release_type=nightly-bkc preset=local remote=disabled local=/.../ccache
[setup_ccache] Cache mode: release_type=nightly preset=github-oss-release remote=http://bazelremote-svc-rel...:8080|layout=bazel|connect-timeout=50 local=/.../ccache

Downloading and inspecting CI logs

Finding a workflow run

The helper scripts in build_tools/ can find runs and artifacts automatically:

# Find the latest successful run for an artifact group
python build_tools/find_latest_artifacts.py --artifact-group gfx110X-all -v

# Find artifacts for a specific commit
python build_tools/find_artifacts_for_commit.py --commit abc123 \
  --artifact-group gfx110X-all

For more manual exploration using the GitHub API directly:

# Find recent multi-arch CI runs
gh api "repos/ROCm/TheRock/actions/workflows/210763103/runs?per_page=10" \
  -q '.workflow_runs[] | "\(.id) \(.created_at[:16]) \(.conclusion) \(.head_sha[:10])"'

# Find math-libs jobs in a specific run
gh api "repos/ROCm/TheRock/actions/runs/{RUN_ID}/jobs?per_page=100" --paginate \
  -q '.jobs[] | select(.name | contains("math-libs") and contains("gfx1151")) | "\(.id) \(.name) \(.conclusion)"'

Checking ccache stats from a job

The ccache stats are printed in the "Report" step of each build job:

gh api repos/ROCm/TheRock/actions/jobs/{JOB_ID}/logs 2>/dev/null \
  | grep -A 25 "Cacheable calls"

Downloading ccache logs from S3

Build logs are uploaded to S3 as artifacts. The ccache log archive contains ccache.log (detailed per-compilation log) and ccache_stats.log (per-file statistics).

# URL pattern
https://therock-ci-artifacts.s3.amazonaws.com/{RUN_ID}-{PLATFORM}/logs/{STAGE}/{GFX_FAMILY}/ccache_logs.tar.zst

# Example
https://therock-ci-artifacts.s3.amazonaws.com/25465494022-windows/logs/math-libs/gfx1151/ccache_logs.tar.zst

To download and extract:

curl -sL -o ccache_logs.tar.zst \
  "https://therock-ci-artifacts.s3.amazonaws.com/{RUN_ID}-windows/logs/math-libs/gfx1151/ccache_logs.tar.zst"

# Extract (requires zstandard package)
import tarfile, io, zstandard

dctx = zstandard.ZstdDecompressor()
with open("ccache_logs.tar.zst", "rb") as f:
    reader = dctx.stream_reader(f)
    with tarfile.open(fileobj=reader, mode="r|") as tf:
        tf.extractall("output_dir", filter="tar")

Using the analysis scripts

# Analyze a CI run's ccache logs (downloads from S3 automatically)
python build_tools/hack/ccache/analyze_ccache_logs.py \
  --run-id 25465494022 --stage math-libs --gfx gfx1151

# Analyze a local log file
python build_tools/hack/ccache/analyze_ccache_logs.py \
  --log-file /path/to/ccache.log

# Compare hit rates per subproject between two logs (e.g., Linux vs Windows)
python build_tools/hack/ccache/compare_ccache_by_project.py \
  /path/to/linux/ccache.log /path/to/windows/ccache.log

Other log files

The S3 index page lists all available logs for a run:

https://therock-ci-artifacts.s3.amazonaws.com/{RUN_ID}-{PLATFORM}/logs/{STAGE}/{GFX_FAMILY}/index.html

This includes per-subproject build/configure/install logs, ninja logs, and the ccache logs archive.

Symptoms of poor cache hit rates

Symptom: Low overall hit rate

Check the "Report" step output. A healthy warm-cache run should show ~95%+ hit rates for cacheable calls.

What to check:

1. Is the cache warm? The first run after a namespace bump or config change will be all misses. Check the second run.

2. Which compiler is missing? Use analyze_ccache_logs.py to break down hits/misses by compiler binary. cl.exe and clang++ may have different hit rates.

Symptom: "can't be read" entries in ccache log

C:\{GUID}\build\...\__stdarg_va_copy.h is mentioned in a manifest
entry but can't be read (No such file or directory)

This means manifest entries reference absolute paths from a different runner that no longer exist on the current runner.

Likely cause: Path instability — the build directory resolves to a different absolute path on each runner (e.g., GUID-based volume mounts on Windows).

Fix: Ensure all tools see and use stable paths. Passing an explicit -resource-dir to clang (in therock_subproject.cmake) can help.

Symptom: Manifest found but no entries match

ccache finds the manifest (same source + command line + compiler hash) but all result entries fail verification.

Likely causes:

• Unstable include file paths: Header paths change between runs (see "can't be read" above)
• Generated headers with volatile content: Version headers that embed git hashes or timestamps change on every build. See THEROCK_FLAG_STAMP_LIBRARY_GIT_VERSIONS in FLAGS.cmake — this should be OFF for non-release builds to avoid embedding git hashes in hip_version.h.
• Re-staged headers with fresh mtime: The build system may rewrite headers during the stage/install step even when content is unchanged. See issue #5009.

Symptom: Different manifest keys across runs for the same file

The direct hash (manifest key) differs between runs even for unchanged source files.

Likely causes:

• GUID in compiler flags: Check if -D defines or other flags contain runner-specific paths (e.g., -DHIP_COMPILER_FLAGS embedding the resolved C:\{GUID}\... path)
• Different compiler binary: If the compiler is rebuilt from source and compiler_check = content is used, verify the binary is byte-identical across runs. Use the analysis scripts or compare SHA256 hashes of the compiler binary from the CI artifacts.
• Different ccache version: Different ccache versions use different hash formats. Entries from one version can't be used by another.

Validating fixes

Verifying compiler binary reproducibility

Download the compiler binary from two different CI runs and compare:

# Download from S3 (streams ~600MB, extracts just clang++)
python -c "
import tarfile, io, urllib.request, zstandard, hashlib
url = 'https://therock-ci-artifacts.s3.amazonaws.com/{RUN_ID}-windows/amd-llvm_run_generic.tar.zst'
with urllib.request.urlopen(url) as resp:
    data = resp.read()
dctx = zstandard.ZstdDecompressor()
reader = dctx.stream_reader(io.BytesIO(data))
with tarfile.open(fileobj=reader, mode='r|') as tf:
    for member in tf:
        if 'clang++.exe' in member.name and member.isfile():
            f = tf.extractfile(member)
            print(f'{hashlib.sha256(f.read()).hexdigest()}  {member.name}')
            break
"

Both runs should produce the same SHA256. If they don't, check that /Brepro is being applied (see therock_subproject.cmake).

Verifying path stability

Check the ccache log for GUID or runner-specific paths:

# Count "can't be read" entries (should be 0)
grep -c "can't be read" ccache.log

# Check for GUID patterns in the log
grep -oP "[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}" ccache.log | sort -u

# Check the setup_ccache.py log output for resolved paths
grep "setup_ccache" job_log.txt

Verifying flag stability

Compare command lines for the same source file across two runs using the ccache logs. Key things to check:

• -I and -isystem paths should be consistent
• -D defines should not contain runner-specific values
• -resource-dir, --hip-path, and --hip-device-lib-path should use stable paths