TheRock/rocm-systems/projects/rccl/tools/scripts/test_runner/db

Test Runner -> results dashboard: results emission & scrape contract

This documents how test_runner.py emits results and how the results dashboard consumes them.

Two writers, one schema

Both paths write the same schema (schema.sql) and are idempotent on runs.run_id, so a run may be delivered by either or both without duplication:

1. test_runner.py --db-push — best-effort direct push from the test host.
2. Dashboard-side ingest job — reads scraped *.tar.gz and upserts.

run_id is generated once per invocation as YYYYMMDDThhmmssZ-<8hex>.

Runs may carry tags (--tag/--tags) for dashboard filtering. Tags are set at emit time; on re-ingest they are left untouched, so once a run is in the DB its tags are mutable only via the dashboard (admin-gated).

Enabling emission

# Local files only (durable; feeds the hourly scrape):
python3 test_runner.py -c configs/rccl_perf_tests.json --emit-results

# Local files + best-effort direct DB push:
export RCCL_RESULTS_DSN='postgresql://rccl_writer:***@dash-host:5432/rccl'
python3 test_runner.py -c configs/rccl_perf_tests.json --db-push

• --db-push implies --emit-results.
• If the DB push fails/times out, the run still succeeds and the local tarball is retained for the scrape. The DSN is read only from RCCL_RESULTS_DSN; it is never hardcoded or written into the emitted files.
• Enabling emission also turns on per-test log capture (via tee), which is what lets the emitter parse busbw/algbw from rccl-tests output. Default behaviour (no capture) is unchanged when emission is off.

Output layout (--results-dir, default <workspace>/results)

results/
  <run_id>/
    run.json         # run manifest (sha, host, config, env, summary, coverage, tags)
    tests.jsonl      # one line per test result
    perf.jsonl       # one line per (size, place) perf row, with collective/avg
    coverage.json    # llvm-cov TOTAL percentages (if --coverage-report)
    logs/            # captured per-test logs (+ coverage rawfiles)
    report/          # llvm-cov HTML + function_coverage_report.txt (if generated)
  <run_id>.tar.gz    # self-contained snapshot of <run_id>/
  latest.tar.gz      # copy of the most recent snapshot (stable scrape target)
  latest.json        # most recent run manifest
  index.jsonl        # append-only: one summary line per run

Scrape contract (hourly cron + scp)

The dashboard host pulls new snapshots on a schedule and ingests them:

# On the dashboard host. Pulls any new run tarballs, then ingests.
0 * * * * rsync -az --ignore-existing \
    testhost:/path/to/results/*.tar.gz /var/lib/rccl-dashboard/incoming/ \
    && /var/lib/rccl-dashboard/ingest.py /var/lib/rccl-dashboard/incoming

Notes:

• rsync --ignore-existing (or scp of only-new files) makes the pull idempotent; the ingest job is also idempotent on run_id.
• Scrapers that only want the newest run can grab latest.tar.gz / latest.json.
• index.jsonl lets a scraper enumerate historical runs without untarring.

Applying the schema

psql "$RCCL_RESULTS_DSN" -f schema.sql