TheRock/external-builds/pytorch/skip_tests

Skipping pytorch tests

Introduction

This tooling allows to narrow down pytorch tests by skipping explicitely tests. Either in general, or additioanlly filtered by amdgpu_family and/or pytorch_version.

By default, we are trying to follow the recommended PyTorch test suite. Below you find the extract of it (1st Oct, 2025).

PYTORCH_TEST_WITH_ROCM=1 python3 test/run_test.py --verbose \
--include test_nn test_torch test_cuda test_ops \
test_unary_ufuncs test_binary_ufuncs test_autograd

Note: 1st Oct 2025: Missing is test/test_ops.py and test/inductor/test_torchinductor.py due to issues in Triton.

However, the filtering using PYTORCH_TEST_WITH_ROCM=1 is not fully reflecting our failures. As such, this tooling will provide a more fine-grained filtering mechanism in our control to skip additional tests.

Independent of this tooling, it is always welcome to get those changes upstream! :)

How to run

../run_pytorch_tests.py steers the pytest and is used by the CI, while ./create_skip_tests.py creates the list of tests to be included or excluded.

Structure

The files including the tests to be skipped are:

• generic.py is always included to skip tests
• pytorch_<version>.py is only included for the given PyTorch version to skip tests

Within those files, the following structure is used

skip_tests = {
    "common": {
        <PyTorch test module> : [ <Tests> ],
    },
    "<amdgpu family short form>": {,
        <PyTorch test module> : [ <Tests> ],
    },
}

Amdgpu family short form is the minimum entry needed to match the right architecture. E.g.

• gfx94 to match gf94X-dcgpu and its arch gfx942
• gfx120 to match gfx120X-all and its archs gfx1200 and gfx1201
• gfx1150 to match gfx1150

The PyTorch test modules are nn, cuda, unary_ufuncs etc. This ordering is mainly added for easier debugging as otherwise it is difficult to determine which test module contains a given tests like test_host_memory_stats belongs to.

When adding new tests to be skipped, consider adding a small comment why it was added, and best if there is any condition/resolution waiting when it can be taken off again.

An example how it could look like is given below:

skip_tests = {
    "common": {
        "cuda": [
            "test_device_count_not_cached_pre_init",
            "test_host_memory_stats",
        ]
    },
    "gfx94": {
        "autograd": [
            "test_multi_grad_all_hooks",
            "test_side_stream_backward_overlap"
            ],
        "cuda": [
            "test_cpp_memory_snapshot_pickle",
            "test_memory_compile_regions",
        ],
        "nn": [
            "test_side_stream_backward_overlap"
        ],
        "torch": [
            "test_terminate_handler_on_crash",  # hangs forever
        ]
    },
}

How to: Upstream skipped tests to PyTorch

For example: Error message

FAILED [0.2901s] external-builds/pytorch/pytorch/test/test_cuda.py::TestCudaMallocAsync::test_memory_compile_regions - TypeError: 'CustomDecompTable' object is not a mapping

1. Go to GitHub PyTorch
2. Search for the class name TestCudaMallocAsync
3. Find the function test_memory_compile_regions
4. Decide further steps, e.g add @skipIfRocm

Function description of @skip<..> can be found in torch/testing/_internal/common_utils.py. They include

@skipIfRocm
@skipIfRocmArch(MI300_ARCH)
@unittest.skipIf(not has_triton(), "test needs triton")
...