derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
280 lines
12 KiB
Markdown
280 lines
12 KiB
Markdown
|
||
README for the OpenMP Tooling Interface Testing Library (ompTest)
|
||
=================================================================
|
||
|
||
# Introduction
|
||
OpenMP Tooling Interface Testing Library (ompTest)
|
||
ompTest is a unit testing framework for testing OpenMP implementations.
|
||
It offers a simple-to-use framework that allows a tester to check for OMPT
|
||
events in addition to regular unit testing code, supported by linking against
|
||
GoogleTest by default. It also facilitates writing concise tests while bridging
|
||
the semantic gap between the unit under test and the OMPT-event testing.
|
||
|
||
# Testing macros
|
||
|
||
Corresponding macro definitions are located in: `./include/AssertMacros.h`
|
||
|
||
## OMPT_GENERATE_EVENTS(NumberOfCopies, EventMacro)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_EVENT(Name, Group, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET(EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_GROUPED(Group, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_NAMED(Name, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_EVENT_NOT(Name, Group, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_NOT(EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_GROUPED_NOT(Group, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_NAMED_NOT(Name, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_EVENT(Name, Group, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE(EventTy, ...)
|
||
This macro checks for the occurrence of the provided event, which also
|
||
entails the exact sequence of events. When only using this assertion macro one
|
||
has to provide every single event in the exact order of occurrence.
|
||
|
||
## OMPT_ASSERT_SEQUENCE_GROUPED(Group, EventTy, ...)
|
||
This macro acts like `OMPT_ASSERT_SEQUENCE` with the addition of grouping.
|
||
|
||
## OMPT_ASSERT_SEQUENCE_NAMED(Name, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_EVENT_NOT(Name, Group, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_NOT(EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_GROUPED_NOT(Group, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_NAMED_NOT(Name, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_SUSPEND()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_ONLY(EventTy, ...)
|
||
This macro acts like `OMPT_ASSERT_SEQUENCE`, while actually being preceded
|
||
-AND- succeeded by commands to suspend sequenced assertion until the next match.
|
||
As a result, one may omit all other "unnecessary" events from the sequence.
|
||
|
||
## OMPT_ASSERT_SEQUENCE_GROUPED_ONLY(Group, EventTy, ...)
|
||
This macro acts like `OMPT_ASSERT_SEQUENCE_ONLY`, plus grouping.
|
||
|
||
## OMPT_ASSERT_SEQUENCE_NAMED_ONLY(Name, EventTy, ...)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_MODE_STRICT(Asserter)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_MODE_RELAXED(Asserter)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_MODE_STRICT()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_MODE_RELAXED()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_MODE_STRICT()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_MODE_RELAXED()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_DISABLE(Asserter)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_ENABLE(Asserter)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_DISABLE()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SET_ENABLE()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_DISABLE()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SEQUENCE_ENABLE()
|
||
`TODO`
|
||
|
||
## OMPT_REPORT_EVENT_DISABLE()
|
||
`TODO`
|
||
|
||
## OMPT_REPORT_EVENT_ENABLE()
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_PERMIT_EVENT(Asserter, EventTy)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_SUPPRESS_EVENT(Asserter, EventTy)
|
||
`TODO`
|
||
|
||
## OMPT_PERMIT_EVENT(EventTy)
|
||
`TODO`
|
||
|
||
## OMPT_SUPPRESS_EVENT(EventTy)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_LOG_LEVEL(Asserter, LogLevel)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERTER_LOG_FORMATTED(Asserter, FormatLog)
|
||
`TODO`
|
||
|
||
## OMPT_ASSERT_SYNC_POINT(SyncPointName)
|
||
`TODO`
|
||
|
||
### Grouping Asserts
|
||
|
||
This allows to generate and verify data during runtime of a test.
|
||
Currently, we only use target region information which manifests into groups.
|
||
This allows to correlate multiple events to a certain target region without
|
||
manual interaction just by specifying a groupname for these events.
|
||
|
||
When a target region is encountered and we are about to enter it, we gather the
|
||
`target_id` (non-EMI) -OR- `target_data->value` (EMI). This value is stored
|
||
along the groupname for future reference. Upon target region end, the
|
||
corresponding group is erased. (Note: The groupname is available again.)
|
||
|
||
Other asserted callbacks which may occur within target regions query their
|
||
groupname: retrieving and comparing the value of the group against the observed
|
||
event's value.
|
||
|
||
### Suspending Sequenced Asserts
|
||
|
||
When a sequence of events is not of interest while testing, these additional
|
||
events may be ignored by suspending the assertion until the next match. This
|
||
can be done by using `OMPT_ASSERT_SEQUENCE_SUSPEND` manually or the `_ONLY`
|
||
macro variants, like `OMPT_ASSERT_GROUPED_SEQUENCE_ONLY`.
|
||
|
||
The former adds a special event to the queue of expected events and signal
|
||
that any non-matching event should be ignored rather than failing the test.
|
||
`_ONLY` macros embed their corresponding macro between two calls to
|
||
`OMPT_ASSERT_SEQUENCE_SUSPEND`. As a consequence, we enter passive assertion
|
||
until a match occurs, then enter passive assertion again. This enables us to
|
||
"only" assert a certain, single event in arbitrary circumstances.
|
||
|
||
### Asserter Modes
|
||
`TODO`
|
||
|
||
## Aliases (shorthands)
|
||
To allow for easier writing of tests and enhanced readability, the following set
|
||
of aliases is introduced. The left hand side represents the original value,
|
||
while the right hand side depicts the shorthand version.
|
||
|
||
| Type | Enum Value | Shorthand |
|
||
|---------------------------|---------------------------------------------|---------------------------|
|
||
| **ompt_scope_endpoint_t** | | |
|
||
| | ompt_scope_begin | BEGIN |
|
||
| | ompt_scope_end | END |
|
||
| | ompt_scope_beginend | BEGINEND |
|
||
| **ompt_target_t** | | |
|
||
| | ompt_target | TARGET |
|
||
| | ompt_target_enter_data | ENTER_DATA |
|
||
| | ompt_target_exit_data | EXIT_DATA |
|
||
| | ompt_target_update | UPDATE |
|
||
| | ompt_target_nowait | TARGET_NOWAIT |
|
||
| | ompt_target_enter_data_nowait | ENTER_DATA_NOWAIT |
|
||
| | ompt_target_exit_data_nowait | EXIT_DATA_NOWAIT |
|
||
| | ompt_target_update_nowait | UPDATE_NOWAIT |
|
||
| **ompt_target_data_op_t** | | |
|
||
| | ompt_target_data_alloc | ALLOC |
|
||
| | ompt_target_data_transfer_to_device | H2D |
|
||
| | ompt_target_data_transfer_from_device | D2H |
|
||
| | ompt_target_data_delete | DELETE |
|
||
| | ompt_target_data_associate | ASSOCIATE |
|
||
| | ompt_target_data_disassociate | DISASSOCIATE |
|
||
| | ompt_target_data_alloc_async | ALLOC_ASYNC |
|
||
| | ompt_target_data_transfer_to_device_async | H2D_ASYNC |
|
||
| | ompt_target_data_transfer_from_device_async | D2H_ASYNC |
|
||
| | ompt_target_data_delete_async | DELETE_ASYNC |
|
||
| **ompt_callbacks_t** | | |
|
||
| | ompt_callback_target | CB_TARGET |
|
||
| | ompt_callback_target_data_op | CB_DATAOP |
|
||
| | ompt_callback_target_submit | CB_KERNEL |
|
||
| **ompt_work_t** | | |
|
||
| | ompt_work_loop | WORK_LOOP |
|
||
| | ompt_work_sections | WORK_SECT |
|
||
| | ompt_work_single_executor | WORK_EXEC |
|
||
| | ompt_work_single_other | WORK_SINGLE |
|
||
| | ompt_work_workshare | WORK_SHARE |
|
||
| | ompt_work_distribute | WORK_DIST |
|
||
| | ompt_work_taskloop | WORK_TASK |
|
||
| | ompt_work_scope | WORK_SCOPE |
|
||
| | ompt_work_loop_static | WORK_LOOP_STA |
|
||
| | ompt_work_loop_dynamic | WORK_LOOP_DYN |
|
||
| | ompt_work_loop_guided | WORK_LOOP_GUI |
|
||
| | ompt_work_loop_other | WORK_LOOP_OTH |
|
||
| **ompt_sync_region_t** | | |
|
||
| | ompt_sync_region_barrier | SR_BARRIER |
|
||
| | ompt_sync_region_barrier_implicit | SR_BARRIER_IMPL |
|
||
| | ompt_sync_region_barrier_explicit | SR_BARRIER_EXPL |
|
||
| | ompt_sync_region_barrier_implementation | SR_BARRIER_IMPLEMENTATION |
|
||
| | ompt_sync_region_taskwait | SR_TASKWAIT |
|
||
| | ompt_sync_region_taskgroup | SR_TASKGROUP |
|
||
| | ompt_sync_region_reduction | SR_REDUCTION |
|
||
| | ompt_sync_region_barrier_implicit_workshare | SR_BARRIER_IMPL_WORKSHARE |
|
||
| | ompt_sync_region_barrier_implicit_parallel | SR_BARRIER_IMPL_PARALLEL |
|
||
| | ompt_sync_region_barrier_teams | SR_BARRIER_TEAMS |
|
||
|
||
|
||
Limitations
|
||
===========
|
||
Currently, there are some peculiarities which have to be kept in mind when using
|
||
this library:
|
||
|
||
## Callbacks
|
||
* It is not possible to e.g. test non-EMI -AND- EMI callbacks within the same
|
||
test file. Reason: all testsuites share the initialization and therefore the
|
||
registered callbacks.
|
||
* It is not possible to check for device initialization and/or load callbacks
|
||
more than once per test file. The first testcase being run, triggers these
|
||
callbacks and is therefore the only testcase that is able to check for them.
|
||
This is because, after that, the device remains initialized.
|
||
* It is not possible to check for device finalization callbacks, as libomptest
|
||
is un-loaded before this callback occurs. Same holds true for the final
|
||
ThreadEnd event(s).
|
||
|
||
Miscellaneous
|
||
=============
|
||
|
||
## Default values
|
||
|
||
To allow for easier writing of tests, many OMPT events may be created using less
|
||
parameters than actually requested by the spec -- by using default values. These
|
||
defaults are currently set to the corresponding data type's minimum as follows,
|
||
for example integers use: `std::numeric_limits<int>::min()`.
|
||
|
||
When an expected / user-specified event has certain values set to the
|
||
corresponding default, these values are ignored. That is, when compared to an
|
||
observed event, this property is considered as 'equal' regardless of their
|
||
actual equality relation.
|
||
|
||
References
|
||
==========
|
||
[0]: ompTest – Unit Testing with OMPT
|
||
https://doi.org/10.1109/SCW63240.2024.00031
|
||
|
||
[1]: OMPTBench – OpenMP Tool Interface Conformance Testing
|
||
https://doi.org/10.1109/SCW63240.2024.00036
|