TheRock/rocm-libraries/dnn-providers/integration-tests/migration-scripts/README.md

# Migration Scripts — C++ Graph Tests to Bundle Format

These scripts convert C++ integration graph tests into the compressed
template+sweep bundle format (ALMIOPEN-2221, ALMIOPEN-2279).

## Why

C++ graph tests build their graphs programmatically — each test is a
`buildGraph()` function plus `INSTANTIATE_TEST_SUITE_P` with hardcoded
parameter lists. This makes them hard to extend, audit, and reuse.

The bundle format stores graphs as JSON, decoupling the test data (graph
topology, tensor shapes, dtypes) from the test harness (build, execute,
verify). A single template+sweep pair can replace dozens of C++ test
registrations that only differ in shapes/dtypes/layouts.

## Pipeline

```
C++ graph test
    |
    |  Hop A: --capture-bundles (run the binary, serialize each graph)
    v
standalone bundle (one JSON per test case)
    |
    |  Hop B: place_bundles.py (group by structure, templatize, compress)
    v
template + sweep (one template per topology, one sweep with all cases)
    |
    |  Hop C: verify_migration.py (reconcile counts + byte-diff graphs/metadata)
    v
VERIFIED at byte level — the migrated graph is identical to the captured one
    |
    |  Hop D: diff_coverage.py (run both suites, compare pass sets)
    v
VERIFIED at behavior level — every C++ PASS has a bundle that also PASSes
```

**Hop C proves the bytes survived; Hop D proves the behavior survived.** A
graph can round-trip perfectly on disk yet fail to run — Hop D is what makes
"turn off the C++ tests, lose nothing" a checked fact rather than a hope.

## How to Run

### Prerequisites

Build the integration test binary (no GPU required for Layers 1-3):

```bash
cmake --build build --target integration_tests
```

### Automated: Full pipeline

```bash
migration-scripts/run_capture_pipeline.sh build/bin/integration_tests
```

This runs Hops A–D plus the supporting checks in sequence. Pass
`--skip-hopd` to skip the differential coverage check (which requires a GPU).

### Manual: Step by step

#### Step 1: Census — see what exists

```bash
python3 migration-scripts/census.py build/bin/integration_tests
```

Runs `--gtest_list_tests` and classifies every test case as `graph`
(C++ graph test to migrate), `bundle` (already a bundle), or `other`.

#### Step 2: Capture (Hop A) — serialize C++ graphs as JSON

```bash
./build/bin/integration_tests --capture-bundles captured_bundles \
    --gtest_filter='*IntegrationGpu*'
```

The `--gtest_filter` restricts capture to `IntegrationGpu*` graph tests
— the tests that build a graph and verify it across GPU plugins. These
are the tests whose graphs can be bundled and re-executed with any
plugin. Non-graph tests (e.g. perf benchmarks, serialization tests)
use different naming prefixes and are excluded by construction.

Each C++ graph test serializes its graph JSON and metadata into:
```
captured_bundles/{SuiteName}/{CaseName}/{CaseName}.json
captured_bundles/{SuiteName}/{CaseName}/{CaseName}.meta.json
```

#### Step 3: Place (Hop B) — compress into template+sweep

```bash
python3 migration-scripts/place_bundles.py \
    --capture-dir captured_bundles \
    --output-dir dnn-providers/integration-tests/integration_test_bundles
```

Groups captured graphs by **structure** (node types + wiring + tensor
set). Graphs sharing the same topology collapse into one template+sweep.
Use `find_case.py` to query cases by any parameter (see below).

#### Step 4: Verify (Hop C) — reconcile everything

```bash
python3 migration-scripts/verify_migration.py \
    --census census.json \
    --capture-dir captured_bundles \
    --bundle-dir integration_test_bundles
```

Three-way reconciliation: census count == captured count == placed count,
plus per-case byte-exact comparison of graph AND metadata (seed, inputs).

#### Step 5: Import individual graphs (incremental)

```bash
python3 migration-scripts/import_graph.py \
    --graph path/to/graph.json \
    --bundle-dir integration_test_bundles/ \
    --meta reference_source="c++ integration suite: Suite.Case"
```

Dedup-aware placement. Default: skip exact duplicates. `--strict` exits
non-zero on dup (CI mode). `--force` appends regardless.

## Searching and Running Bundles

### find_case.py — query cases by any parameter

`find_case.py` reads `sweep.json` directly (single source of truth).
No separate manifest to maintain.

```bash
# List all batchnorm cases
python3 migration-scripts/find_case.py --op Batchnorm

# Find fp16 nhwc cases
python3 migration-scripts/find_case.py --dtype fp16 --layout nhwc

# Find cases that have an epsilon input (any range)
python3 migration-scripts/find_case.py --input epsilon

# Find cases where epsilon is in [-1,1]
python3 migration-scripts/find_case.py --input epsilon:-1,1

# Find by shape
python3 migration-scripts/find_case.py --shape 1x16x3x3

# Combine filters
python3 migration-scripts/find_case.py --op Batchnorm --dtype bfp16 --input scale:-2,2

# Full detail for a case (includes the exact --gtest_filter command)
python3 migration-scripts/find_case.py --id f446b9 --detail
```

### gtest_filter examples

Bundle suites register under gtest as `{tier}_{Op}_{Topology}` (e.g.
`quick_Batchnorm_Default`). Each case within a suite is named by its
case id:

```bash
# Run all quick-tier bundles
--gtest_filter='quick_*'

# Run all batchnorm cases (any tier)
--gtest_filter='*Batchnorm*'

# Run all bfp16 nhwc cases across all ops
--gtest_filter='*bfp16_nhwc*'

# Run one exact case (by hash suffix)
--gtest_filter='*f446b9*'

# Combine: quick batchnorm, only fp32
--gtest_filter='quick_Batchnorm_*/*fp32*'
```

When two cases share the same shape/dtype/layout but differ in input
ranges or seeds, a 6-char content hash is appended to disambiguate:

```
1_3_14_bfp16_ncl          ← unique readable prefix, no hash needed
1_3_14_bfp16_ncl_abc123   ← same prefix, different input ranges
1_3_14_bfp16_ncl_def456   ← same prefix, different input ranges
```

Use `find_case.py --id abc123 --detail` to see what a hashed case contains.

### Adding new test cases

New tests should be added directly as bundle cases — no C++ needed.

**Using import_graph.py (recommended):**

```bash
python3 migration-scripts/import_graph.py \
    --graph new_conv.json \
    --bundle-dir integration_test_bundles/
```

What happens:

1. The script computes the graph's skeleton hash and finds matching
   topologies in the bundle tree.
2. **Duplicate?** If an identical case already exists (same graph +
   seed + inputs), it reports `DUPLICATE` and skips. No manual check
   needed.
3. **New case for existing topology?** Appends to that sweep. The case
   id is auto-generated: `{shape}_{dtype}_{layout}_{attrs}[_{hash6}]`.
4. **New topology?** Creates a new template+sweep directory.
5. The auto-generated id is printed to stderr so you see what the test
   will be called in gtest output:

```
  appended case '1_16_3_3_bfp16_nhwc_dil1x1_prepad1x1_postpad1x1' to
    integration_test_bundles/quick/ConvolutionFwd/Default/sweep.json
```

That case id is the gtest name — it appears in CI logs, `--gtest_filter`,
and `find_case.py` queries. You never need to invent or assign it.

**Manually editing sweep.json:**

Add a new entry to the `"cases"` array with `"values"` and `"metadata"`.
Set `"id"` to a descriptive name following the pattern
`{shape}_{dtype}_{layout}[_{attrs}]`, or run `import_graph.py` to have
it assigned automatically.

The migration pipeline (`run_capture_pipeline.sh`) is a one-time
conversion tool. Going forward, the bundle tree is the source of truth.

## Verification

Two hops prove the migration; two supporting checks guard the tooling:

| Check | What | Needs GPU? |
|---|---|---|
| **Hop C — byte round-trip** | Census ↔ capture ↔ sweep count reconciliation, per-case byte-diff of graph + seed + inputs | No |
| Supporting — loader smoke | Real C++ binary loads every placed bundle via the production template expander | Depends on test |
| Supporting — idempotency | Re-run pipeline; `git diff --exit-code` must be clean | No |
| **Hop D — diff coverage** | `pass_set_bundle ⊇ pass_set_cpp` — no C++ PASS becomes a bundle SKIP | **Yes** |

Hop D is the acceptance proof: when it's green, turning off C++
integration tests provably loses no coverage for graph+GPU-plugin tests.

```bash
python3 migration-scripts/diff_coverage.py \
    --cpp /tmp/cpp.json --bundle /tmp/bundle.json \
    --bundle-dir integration_test_bundles
```

## Field Mapping — What Gets Bundled and Where

| Field | C++ Origin | Bundle Destination | Compression |
|---|---|---|---|
| Node types + wiring | `buildGraph()` | `graph.template.json` nodes[] | Invariant (skeleton) |
| Tensor dims | `TensorAttributes` | sweep `values.tensors[uid].dims` | Per-case |
| Tensor strides | `generateStrides()` | sweep `values.tensors[uid].strides` | Per-case |
| data_type | `getDataTypeEnumFromType<T>()` | sweep `values.tensors[uid].data_type` | Per-case |
| Node attrs | `ConvFpropAttributes` etc. | sweep `values.attributes.<key>` | Per-case, only if varies |
| Seed | `synthesis().setGlobalSeed()` | sweep `case.metadata.seed` | Per-case metadata |
| Distribution/range | `SynthesisConfig.fills()` | sweep `case.metadata.inputs.{uid}` | Per-case metadata |
| Per-op defaults | `SynthesizeInputs.cpp` | **NOT stored** | Re-derived from topology |
| Provenance | (new) | sweep `case.metadata.reference_source` | Per-case metadata |

## Scripts

| Script | Purpose |
|---|---|
| `bundle_utils.py` | Shared utilities (skeleton hash, canonicalization, expansion, case-id) |
| `census.py` | List and classify all C++ test cases (the migration denominator) |
| `place_bundles.py` | Convert captured bundles into template+sweep format (Hop B) |
| `verify_migration.py` | Reconcile census ↔ capture ↔ sweep, byte-diff graphs + metadata (Hop C) |
| `import_graph.py` | Import a single graph with duplicate detection |
| `find_case.py` | Query cases by op, dtype, layout, shape, input range, or id |
| `diff_coverage.py` | Differential coverage: assert `pass_set_bundle ⊇ pass_set_cpp` (Hop D) |
| `run_capture_pipeline.sh` | Orchestrate all hops + verification layers |
| `test_migration.py` | Self-test on synthetic fixture (no binary needed) |

## How It Works

### Structure-hash grouping

Every graph gets a canonical skeleton fingerprint: node types in order,
tensor UID wiring (canonically renumbered), and the tensor set. Graphs
with the same fingerprint share a topology.

### Derive, don't classify

Instead of maintaining a per-op allowlist:
- **Structural** (fixed in template): `node.type`, `*_tensor_uid`,
  `tensor.uid`, `tensor.virtual`
- **Knob** (templatized if varies): dims, strides, dtypes, node attrs

### Case ids — handle, not description

A sweep case can differ along many axes: tensor dims, strides, dtype, node
attributes (padding/stride/dilation), and per-tensor input ranges/seeds. That
is too high-dimensional to encode in a readable gtest name. So the id is a
**handle**, following the content-addressable pattern (`docker`, `git`, `nix`):

```
{shape}_{dtype}_{layout}_{≤3 salient attrs}[_{hash6}]
```

- The readable tokens (shape, dtype, layout, top attrs) are the **filter
  surface** (RFC 0011 §4.1) — they only encode axes that actually vary in the
  sweep, so ids stay short.
- The `hash6` suffix is a short content hash over the case's **full identity**
  (`values` + `metadata.inputs` + `seed`), appended only when the readable base
  is not already unique. It guarantees uniqueness — including for two cases that
  differ *only* in an input range (e.g. a bias filled from `[-0.5, 0.5]` vs
  `[-1, 1]`) — and is stable across additions/reordering, unlike an `_N` counter.

Use `find_case.py` to query the full per-case truth the id cannot carry.

### Verify gate

Every expanded case is compared byte-for-byte against the original.
Any mismatch falls back to a standalone bundle. Nothing is silently dropped.