# Hotswap failure reducer

`utils/hotswap/hotswap_reduce.py` turns a failing hotswap corpus bundle into a
small, standalone reproducer. It only accepts a transformation when a
caller-provided interestingness command continues to return the configured exit
code. The original inputs are never modified and the output directory is
published atomically.

The reducer works from the outside in:

1. Delta-debug the list of code objects.
2. Delta-debug the top-level `kernels`, `cases`, and `arguments` metadata
   lists, followed by each retained case's `arguments` list.
3. Optionally remove explicitly allowed, non-allocated ELF sections with
   `llvm-objcopy`.
4. Repeat the hierarchy until a complete pass makes no change. This gives each
   retained dimension another reduction opportunity after lower-level changes.

The section pass gets section names and flags from `llvm-readobj` JSON. It does
not parse ELF itself. Allocated sections and the default runtime-critical
section set are always protected. Section removal is disabled unless at least
one `--allow-remove-section` glob is provided. If multiple sections have the
same name, one allocated or protected instance protects every instance because
`llvm-objcopy` removes sections by name. Every transformed output is parsed
again to verify that it is still AMDGPU ELF, every requested section is gone,
and no unrequested section disappeared.

## Interestingness command

The command is an argv vector, not a shell string. Pass the executable with
`--predicate` and repeat `--predicate-arg` for its arguments. These literal
placeholders are expanded for each candidate:

- `{bundle}`: candidate `bundle.json`
- `{input}`: the sole candidate code object (only valid with one object)
- `{metadata}`: candidate `metadata.json`
- `{workspace}`: candidate directory

Exit zero means interesting by default. Use `--interesting-exit-code` when the
reproducer intentionally reports a failure with a nonzero exit. A timeout,
launch error, or exit-code disagreement between `--predicate-runs` repetitions
rejects a candidate. Predicate output retained in the log is bounded to avoid
unbounded memory use. Timed-out predicate process trees are terminated before
their temporary workspace is removed.

Stable outcomes are cached by the complete candidate content and predicate
configuration; `--cache-file` makes that cache persistent. The resolved
predicate executable and regular-file argv items are identified by path,
content, and file mode. Repeat `--cache-dependency` for dynamically loaded
libraries, reference data, or other file inputs, and repeat `--cache-tag` for
relevant environment or configuration identity. A dependency that changes
during a run aborts the reduction instead of consulting a stale cache entry.
Predicates must be deterministic and must not rely on side effects outside
their candidate workspace because a cache hit replaces the process execution.

For example:

```console
python3 utils/hotswap/hotswap_reduce.py \
  --bundle failing-bundle.json \
  --output reduced \
  --predicate /usr/bin/python3 \
  --predicate-arg=/absolute/path/is_interesting.py \
  --predicate-arg='{bundle}' \
  --predicate-runs 3 \
  --timeout 20 \
  --allow-remove-section='.debug_*' \
  --allow-remove-section='.comment'
```

No argument is evaluated by a shell. Supplying an absolute predicate and
absolute script path makes the recorded reproduction argv unambiguous on the
same filesystem. Existing regular-file argv items are resolved to absolute
paths before the first predicate run.

## PR #3646 offline differential workflow

An A0 runtime failure is not required. The predicate can be any deterministic
offline check, which makes the reducer useful while validating the hotswap
implementation in PR #3646. For a discrepant hipBLASLt or hipSOLVER case,
first put the involved code objects and launch records in a regular bundle.
Then use one of these predicate shapes:

- A wrapper rewrites the candidate with the PR #3646 build, emits its
  structural manifest, compares it with a fixed PR #3598 reference manifest,
  and exits with the interesting code while a difference remains.
- A wrapper runs `hotswap-audit` over every retained object and exits with the
  interesting code while the audit reports an invariant violation.
- A wrapper runs `hotswap-semcheck` for the retained kernels and exits with the
  interesting code while it returns a counterexample.

For example, an offline manifest differential can be driven as:

```console
python3 utils/hotswap/hotswap_reduce.py \
  --bundle hipblaslt-discrepancy.json \
  --output reduced-3646-difference \
  --predicate /absolute/path/manifest-diff-predicate \
  --predicate-arg=/absolute/path/pr3598-reference.json \
  --predicate-arg='{bundle}' \
  --predicate-runs 2
```

The wrapper owns the meaning of the comparison and the selected build. The
reducer only sees an argv vector and an exit code, so no PR-specific opcode,
kernel name, or library test name is encoded in its reduction logic. Keep the
reference manifest outside the candidate bundle so ddmin cannot reduce the
oracle itself.

## Input and output bundle

Input bundles use this versioned JSON form. Object paths and a string-valued
`metadata` path are relative to the bundle file and cannot escape its directory,
including through a symlink. Use direct `--code-object` or `--worklist` input
when the object files intentionally live elsewhere.

```json
{
  "format": "comgr-hotswap-reducer-bundle",
  "version": 1,
  "code_objects": [
    {"id": "hipblaslt-case-17", "path": "objects/case 17.co"}
  ],
  "metadata": {
    "kernels": [{"name": "kernel"}],
    "cases": [{"name": "case", "arguments": [1, 2, 3]}],
    "arguments": []
  }
}
```

`--code-object` can be repeated instead of providing a bundle, with optional
metadata from `--metadata`.

An inventory or test-selection pipeline can provide a NUL-delimited worklist:

```console
python3 utils/hotswap/hotswap_reduce.py \
  --worklist unique-code-objects.list \
  --metadata selection.json \
  --output reduced \
  --predicate /absolute/path/offline-predicate \
  --predicate-arg='{bundle}'
```

Every worklist entry is treated as a path, never shell text. Absolute paths
from the inventory utility work directly; relative paths are resolved from the
worklist's directory. Empty, unterminated, or duplicate entries are rejected.
A selector document can be passed directly as metadata. In addition to
`kernels`, `cases`, and `arguments`, a top-level `selected_tests` list is
delta-debugged generically; all unknown selector fields are preserved.

The output contains:

- `bundle.json`, `metadata.json`, and the retained objects under `objects/`
- `reduction-log.json`, a versioned deterministic record of every accepted or
  rejected transformation
- a final reproduction argv and interesting exit code in that log

The log also records original and final content digests. It intentionally
contains no timestamps, elapsed times, or cache-hit state so identical inputs
and stable predicates produce identical logs.

## Safety limits

- Existing output directories are never overwritten.
- Cache files cannot overwrite an input (including external bundle metadata),
  predicate identity, dependency, or live inside the atomically published
  output directory.
- Malformed bundles and metadata fail before a predicate runs.
- Section pruning requires AMDGPU ELF input and available LLVM tools.
- Protected or allocated sections cannot be opted into removal.
- Temporary candidates are removed after interruption, timeout, or failure.
- `llvm-reduce` is not run on linked ELF code objects; it is an IR/MIR reducer.
  Use it before linking when a retained bundle also has LLVM IR or MIR source.
- Persistent cache writes are atomic but are not merged across concurrent
  reducer processes; give each concurrent process a distinct cache file.